Patent Description:
Forklifts are widely used to transport goods by inserting forks into holes in pallets. Automated guided forklifts (AGFs) have been developed to provide this type of transport unmanned.

If the pallet is placed in a predetermined position and orientation (rotational angle in plan view), the AGF can insert the fork into holes of the pallet according to an approach path. However, although the actual pallet is placed so as to fit in a predetermined area (slot), the position and orientation in the slot varies with each placement. Therefore, it is possible that the AGF which only inserts the fork according to the approach path cannot insert the fork into holes of the pallet.

To solve this problem, a method for detecting the position and orientation of the pallet has been proposed. For example, it is known that 3D-LiDAR (LiDAR: Light Detection And Ranging), 2D-LiDAR, or 3D cameras are used to acquire three-dimensional data of the pallet to detect the position and orientation of the pallet.

In the method using 2D-LiDAR, it is necessary to measure the pallet while moving the laser up and down in order to acquire three-dimensional data of the pallet. For example, Patent Document <NUM> discloses the acquisition of three-dimensional distance images while moving the laser sensor in the vertical direction to acquire three-dimensional data of the pallet.

<CIT> discloses a forklift capable of carrying a load placed on a pallet having two openings into which a fork of the forklift is inserted. The forklift includes a sensor configured to irradiate laser light toward a predetermined space forward of the fork, and measure a distance from the sensor to an object located in the predetermined space based on reflected light of the laser light reflected by the object; and a processor configured to identify positions of sidewalls of the two openings of the pallet that is to be lifted based on distance data measured by the sensor, the processor being further configured to identify a center of a front surface of the pallet based on the positions of the sidewalls of the two openings.

<NPL> discloses an autonomous manipulation of a priori unknown palletized cargo with a forklift.

<CIT> discloses a method which may include receiving, from a sensor on a vehicle, an initial plurality of sensor data points representing a position of a face of a pallet. The vehicle may include tines configured to engage the pallet. A baseline geometric representation of the face of the pallet may be determined based on the initial plurality of sensor data points. The vehicle may be caused to reposition the tines relative to the pallet. A subsequent plurality of sensor data points representing the position of the face of the pallet after repositioning the tines may be received from the sensor. An updated geometric representation of the face of the pallet may be determined based on the subsequent sensor data points. It may be determined that the updated geometric representation deviates from the baseline geometric representation by more than a threshold value and, in response, motion of the vehicle may be adjusted.

Since 3D-LiDAR is very expensive, there is a cost problem in mounting it on a forklift. In addition, 3D cameras are not very convenient because their applicability is limited due to the degradation of accuracy by sunlight and the limitations of the distance and viewing angle that can be measured. Therefore, it is desirable to use 2D-LiDAR to detect the position and orientation of the pallet.

In Patent Document <NUM>, by acquiring three-dimensional data of the pallet while moving the laser sensor in the vertical direction, a straight line corresponding to the front surface of the pallet is extracted based on a group of observation points on the same plane obtained by reflected light from the front surface of the pallet. However, this method requires the laser sensor to be moved in the vertical direction, which makes the measurement time consuming.

In view of the above, an object of the present disclosure is to provide a pallet detection device that can accurately extract a straight line corresponding to the front surface of a pallet without moving a two-dimensional distance measurement device in the vertical direction and detect the position and orientation of the pallet in a short time.

A pallet detection device according to the present disclosure comprises: a point cloud acquisition unit configured to acquire point cloud data indicating a point cloud measured by a two-dimensional distance measurement device on a depth map; a straight line detection unit configured to detect a straight line corresponding to a front surface of a pallet based on the point cloud in a region presumed to include the front surface of the pallet in the point cloud data; a line segment detection unit configured to detect a line segment indicating the front surface of the pallet based on the straight line; and a position/orientation acquisition unit configured to acquire position and orientation of the pallet based on the line segment. The straight line detection unit is configured to: acquire one or more straight line candidates as candidates for the straight line, assign, for each of the one or more straight line candidates, a score to the point cloud so that points in front of the straight line candidate at a specified distance or more are assigned with a score with a constant whose priority in selection is lower than the rest of points, select the straight line from the one or more straight line candidates based on a score accumulated value obtained by accumulating the score for the point cloud in the region presumed to include the front surface of the pallet, and in the calculation of the score accumulated value, assign a higher priority score to points within the specified distance from the straight line candidate than to points in front of the straight line candidate at the specified distance or more, and the closer to the straight line candidate, the higher the priority of the score.

A forklift according to the present disclosure comprises: the above-described pallet detection device; a two-dimensional distance measurement device for acquiring a point cloud; and a driving unit configured to perform transport of a pallet according to a detection result of the pallet detection device.

A pallet detection method carried out by a computer according to the present disclosure comprises: acquiring point cloud data indicating a point cloud measured by a two-dimensional distance measurement device on a depth map; detecting a straight line corresponding to a front surface of a pallet based on the point cloud in a region presumed to include the front surface of the pallet in the point cloud data; detecting a line segment indicating the front surface of the pallet based on the straight line; and acquiring position and orientation of the pallet based on the line segment. The detecting the straight line includes acquiring one or more straight line candidates as candidates for the straight line, assigning, for each of the one or more straight line candidates, a score to the point cloud so that points in front of the straight line candidate at a specified distance or more are assigned with a score with a constant whose priority in selection is lower than the rest of points, selecting the straight line from the one or more straight line candidates based on a score accumulated value obtained by accumulating the score for the point cloud in the region presumed to include the front surface of the pallet, and assigning, in the calculation of the score accumulated value, a higher priority score to points within the specified distance from the straight line candidate than to points in front of the straight line candidate at the specified distance or more, and the closer to the straight line candidate, the higher the priority of the score.

A program according to the present disclosure is configured to cause a computer to execute: a process of acquiring point cloud data indicating a point cloud measured by a two-dimensional distance measurement device on a depth map; a process of detecting a straight line corresponding to a front surface of a pallet based on the point cloud in a region presumed to include the front surface of the pallet in the point cloud data; a process of detecting a line segment indicating the front surface of the pallet based on the straight line; and a process of acquiring position and orientation of the pallet based on the line segment. The process of detecting the straight line includes acquiring one or more straight line candidates as candidates for the straight line, assigning, for each of the one or more straight line candidates, a score to the point cloud so that points in front of the straight line candidate at a specified distance or more are assigned with a score with a constant whose priority in selection is lower than the rest of points, selecting the straight line from the one or more straight line candidates based on a score accumulated value obtained by accumulating the score for the point cloud in the region presumed to include the front surface of the pallet, and assigning, in the calculation of the score accumulated value, a higher priority score to points within the specified distance from the straight line candidate than to points in front of the straight line candidate at the specified distance or more, and the closer to the straight line candidate, the higher the priority of the score.

The present disclosure provides a pallet detection device that can accurately extract a straight line corresponding to the front surface of a pallet without moving a two-dimensional distance measurement device in the vertical direction and detect the position and orientation of the pallet in a short time.

Embodiments will now be described in detail with reference to the accompanying drawings.

First, the usage environment of a forklift <NUM> equipped with a pallet detection device <NUM> according to an embodiment will be described. A pallet <NUM> is placed to fit into a predetermined area (slot <NUM>).

As described later, the pallet detection device <NUM> can acquire information about the position of the slot <NUM>, the dimension and shape of the pallet <NUM>, the position of the forklift <NUM>, and the position of a two-dimensional distance measurement device <NUM> (see <FIG>). The two-dimensional distance measurement device <NUM> is, for example, a two-dimensional laser scanner, such as 2D-LiDAR, which measures the distance and direction to the measurement object in a certain measurement space region by scanning the measurement light horizontally and receiving the reflected light. In the following description, a set of measurement points acquired by the two-dimensional distance measurement device <NUM> will be referred to as a point cloud. The pallet <NUM> is placed so that, in plan view, the translational displacement and rotational angle relative to the center of the slot <NUM> are within certain limits. For example, the limits are a misalignment of <NUM> or less and an angular misalignment of <NUM>° or less. The pallet <NUM> is placed on the floor, ground, etc., for example.

Here, an arrangement example of pallets <NUM> will be described, and various problems to be solved by the pallet detection device <NUM> will be described. <FIG> is a plan view showing an example where pallets <NUM> (10A, 10B, 10C) are placed at equal intervals. <FIG> is a plan view showing an example where pallets <NUM> (10A, 10B) are in close proximity to each other. <FIG> is a plan view showing an example where pallets <NUM> (10A, 10B) are placed at a spacing that is equal to the width D of holes <NUM> in the pallets <NUM>. <FIG> is a plan view showing an example where a pallet <NUM> (10B) is rotated relative to the slot <NUM>. <FIG> is a plan view showing an example where a pallet <NUM> (10B) is placed farther back than adjacent pallets <NUM> (10A, 10C).

In these examples, three pallets <NUM> and slots <NUM> are adjacent to each other. The number of pallets <NUM> and the number of slots <NUM> are not limited to the illustrated examples. As shown in <FIG>, each pallet <NUM> has two holes <NUM> into which a fork <NUM> of the forklift <NUM> is inserted in the width direction of the pallet <NUM>. Each pallet <NUM> may have a plurality of sets of two holes <NUM> in the height direction of the pallet <NUM>. As shown by the arrows in the figures, the two-dimensional distance measurement device <NUM> irradiates a measurement light (for example, a laser beam) toward the pallet <NUM> and scans in the horizontal direction. The front surface of the pallet <NUM> is a lower end surface in the figures.

In the example shown in <FIG>, three pallets <NUM> are placed at equal intervals in the centers of the respective slots <NUM>, and each rotation angle with respect to the slot <NUM> is zero. This is the ideal arrangement. If such an arrangement can be realized each time the pallet is placed, the fork <NUM> can be inserted from the front only based on the positional relationship between the slot <NUM> and the forklift <NUM> without detecting the pallet. However, in practice, it is difficult to insert the fork <NUM> without detecting the pallet because the position and orientation of the pallet <NUM> in the slot <NUM> can vary within certain limits for example as shown in <FIG>.

In the example shown in <FIG>, the middle pallet <NUM> (10B) and the right pallet <NUM> (10A) are close to each other, with almost no gap therebetween. In this case, even if the two-dimensional distance measurement device <NUM> simply acquires a point cloud P1, which is a set of measurement points, based on the reflected light from the pallet <NUM>, it is difficult to identify the point cloud P1 for each pallet <NUM>. For example, the middle pallet <NUM> (10B) and the right pallet <NUM> (10A) may be identified as one, or the boundary between the two may not be identified.

In the example shown in <FIG>, the spacing between the middle pallet <NUM> (10B) and the right pallet <NUM> (10A) are the same as the width D of the hole <NUM>. In this case, even if the width of the region where the point cloud P1 acquired by the two-dimensional distance measurement device <NUM> does not exist is simply focused, the gap between the pallets <NUM> may not be distinguished from the hole <NUM>.

In the example shown in <FIG>, the middle pallet <NUM> (10B) is rotated by <NUM>° with respect to the slot <NUM>. In this case, unless the orientation of the middle pallet <NUM> (10B) is detected, the fork <NUM> may not be inserted into the hole <NUM> due to angular deviation from the front direction.

In the example shown in <FIG>, the pallet <NUM> (10B) is placed farther back than flanking pallets <NUM> (10A, 10C). In this case, when detecting the front surface of the middle pallet <NUM> (10B), one of the front surfaces of the flanking pallets <NUM> (10A, 10C) located forward of the front surface of the middle pallet <NUM> (10B) may be erroneously detected as the front surface of the middle pallet <NUM> (10B).

<FIG> is a schematic side view of the forklift <NUM> according to an embodiment. <FIG> is a schematic plan view of the forklift <NUM> according to an embodiment. For reference, these figures also show the pallet <NUM> on which a load W is placed.

As shown in <FIG>, the forklift <NUM> includes a pallet detection device <NUM>, three two-dimensional distance measurement devices <NUM> for acquiring a point cloud P1, and a driving unit (not shown) configured to perform transport of a pallet <NUM> according to a detection result of the pallet detection device <NUM>. Generally, the AGF is equipped with a safety sensor (2D-LiDAR) having an obstacle detection function and an emergency stop function according to a safety standard. The two-dimensional distance measurement device <NUM> may be the safety sensor (2D-LiDAR).

The two-dimensional distance measurement device <NUM> as the safety sensor are disposed on the forklift <NUM>, for example, two on the left and right sides on the side of the fork <NUM>, and one in the center on the side opposite to the fork <NUM>. These two-dimensional distance measurement device <NUM> are disposed below the height of the pallet <NUM>. The pallet detection device <NUM> may be configured to detect the pallet <NUM> by a combination thereof. The position and the number of two-dimensional distance measurement devices <NUM> are not limited to the illustrated examples, and may be changed as appropriate.

The configuration of the pallet detection device <NUM> according to an embodiment will now be described. <FIG> is a schematic block diagram of a hardware configuration of the pallet detection device <NUM> according to an embodiment. For example, as shown in <FIG>, the pallet detection device <NUM> is provided by a computer including a processor <NUM> such as a central processing unit (CPU) and a graphics processing unit (GPU), a random access memory (RAM) <NUM>, a read only memory (ROM) <NUM>, a hard disk drive (HDD) <NUM>, an input I/F <NUM>, and an output I/F <NUM>, which are connected via a bus <NUM>. The processor <NUM> of the pallet detection device <NUM> executes a program stored in the memory such as the ROM <NUM> and the RAM <NUM> to implement functions described later.

<FIG> is a schematic block diagram of a functional configuration of the pallet detection device <NUM> according to an embodiment. As shown in <FIG>, the pallet detection device <NUM> includes, as functional units, a point cloud acquisition unit <NUM> configured to acquire point cloud data, a region setting unit <NUM> configured to set a detection target region, a straight line detection unit <NUM> configured to detect a straight line corresponding to the front surface of the pallet <NUM>, a both-end detection unit <NUM> configured to detect both ends of the pallet <NUM>, a line segment detection unit <NUM> configured to detect a line segment indicating the front surface of the pallet <NUM>, a position/orientation acquisition unit <NUM> configured to acquire position and orientation of the pallet <NUM>, and an information acquisition unit <NUM> configured to acquire information about the position.

The flow of a processing performed by the pallet detection device <NUM> according to an embodiment will now be described. Prior to this processing, the information acquisition unit <NUM> acquires information indicating the self-position (the position of the forklift <NUM>) and information indicating the position of the slot <NUM> (the area in which the slot <NUM> is located). The information acquisition unit <NUM> may acquire such information by referring to the memory, or may acquire the information by communicating with a sensor (not shown) or a server device (not shown). For example, the position of the slot <NUM> is stored in a memory or a server device (not shown) in association with a map used in estimation of the self-position, and is read out by the information acquisition unit <NUM>. This association is performed by the user.

<FIG> is a flowchart for describing an example of a processing executed by the pallet detection device <NUM> according to an embodiment. <FIG> is a schematic diagram showing an illustrative example of the point cloud P1 acquired by the pallet detection device <NUM> according to an embodiment. As shown in <FIG>, the pallet detection device <NUM> approaches the vicinity of the pallet <NUM> based on the information indicating the self-position and the information indicating the position of the slot <NUM>, and the two-dimensional distance measurement device <NUM> scans the measurement light in the horizontal direction and measures the reflected light from a certain measurement space region including the pallet <NUM> to acquire point cloud data (e.g., point cloud P1 shown in <FIG>) (Step S1).

Specifically, first, the point cloud acquisition unit <NUM> acquires point cloud data indicating a point cloud P1 measured by the two-dimensional distance measurement device <NUM> on a depth map. The depth map is a map representing the fixed measurement space region in two dimensions in the depth direction and the width direction.

For example, as shown in <FIG>, the point cloud acquisition unit <NUM> acquires a point cloud P1 around the pallet <NUM> to be measured by the two-dimensional distance measurement device <NUM>. At this time, the point cloud acquisition unit <NUM> identifies a region presumed to include the front surface of the pallet <NUM> based on the information acquired by the information acquisition unit <NUM> and acquires the point cloud P1 around the pallet <NUM> based on the identification result. The point cloud P1 acquired by the point cloud acquisition unit <NUM> may be a superposition of point clouds P1 acquired from multiple two-dimensional distance measurement devices <NUM> (e.g., two or more of the three two-dimensional distance measurement devices <NUM> shown in <FIG>). In this case, a dense point cloud P1 can be obtained. At this time, the pallet detection device <NUM> may acquire point cloud data obtained by superimposing point cloud data acquired from multiple two-dimensional distance measurement devices <NUM>.

The pallet detection device <NUM> acquires one or more straight line candidates from the point cloud data. Specifically, first, the region setting unit <NUM> sets a detection target region. The detection target region is a so-called region of interest (ROI). For example, as shown in <FIG>, the region setting unit <NUM> sets a first detection target region A1 having a width larger than the slot <NUM> to extract the point cloud P1 in the first detection target region A1. The straight line detection unit <NUM> acquires one or more straight line candidates for detecting a line segment indicating the front surface of the pallet <NUM>. The straight line detection unit <NUM> assigns, for each of the one or more straight line candidates, a score to the point cloud P1 so that points P1 in front of the straight line candidate at a specified distance or more are assigned with a score with a constant whose priority in selection is lower than the rest of points P1. The straight line detection unit <NUM> selects the straight line corresponding to the front surface of the pallet <NUM> from the one or more straight line candidates based on a score accumulated value obtained by accumulating the score for the point cloud P1 in the region presumed to include the front surface of the pallet <NUM>.

<FIG> is a schematic diagram showing an illustrative example of straight line candidates (straight line candidates L1, L2, L3, L4) acquired by the pallet detection device <NUM> according to an embodiment. For example, as shown in <FIG>, the region setting unit <NUM> may further set a second detection target region A2 having a width smaller than the first detection target region A1, and the straight line detection unit <NUM> may acquire a straight line connecting two points selected from the point cloud P1 in the second detection target region A2 as the straight line candidate. For example, as shown in <FIG>, the straight line detection unit <NUM> acquires four straight line candidates L1, L2, L3, L4 based on the point cloud P1. The width of the second detection target region A2 is set so that the adjacent pallet <NUM> does not enter. The width of the second detection target region A2 may be larger or smaller than that of the slot <NUM>.

Illustrative examples of the method of acquiring straight line candidates will be described. The straight line detection unit <NUM> may be configured to acquire a straight line connecting two point selected by a RANSAC algorithm from the point cloud P1 in the detection target region as the straight line candidate. The detection target region in this case is the second detection target region A2. The detection target region may be the first detection target region A1, or may be a region different from the first detection target region A1 and the second detection target region A2. The RANSAC (RANdom Sample Consensus) is a method of randomly selecting two points and calculating an evaluation value. The detection method of the pallet <NUM> according to the present disclosure is not limited to the method using RANSAC, and other algorithms such as PROSAC (PROgressive Sample Consensus) may be used.

The straight line detection unit <NUM> may be configured to acquire a straight line connecting two points selected from the point cloud P1, one each from right and left sides of the widthwise center of the detection target region, as the straight line candidate. In this case, since the number of combinations is finite, all combinations of two points may be straight line candidates.

The pallet detection device <NUM> calculates the score accumulated value for each of the one or more straight line candidates (step S3). Further, the pallet detection device <NUM> detects the straight line corresponding to the front surface of the pallet <NUM> based on the score accumulated value (step S4). Specifically, the straight line detection unit <NUM> calculates the score accumulated value and detects the straight line corresponding to the front surface of the pallet <NUM> based on the score accumulated value. The straight line detection unit <NUM> may include points P1 outside the second detection target region A2 and within the first detection target region A1 in the straight line candidate having a high priority indicated by the score accumulated value and select (i.e., detect) this straight line candidate as the straight line corresponding to the front surface of the pallet <NUM>. Such points P1 may also be subjected to a subsequent processing. For example, the straight line detection unit <NUM> selects the straight line candidate L1 shown in <FIG> as the straight line corresponding to the front surface of the pallet <NUM> based on the score accumulated value.

The score accumulated value will now be described. For example, in the case that a straight line candidate with a high score accumulated value is preferentially selected as the straight line to be used for line segment detection, "a score with a constant whose priority in selection is low" means a score with a value that decreases the score accumulated value. Conversely, in the case that a straight line candidate with a low score accumulated value is preferentially selected, "a score with a constant whose priority in selection is low" means a score with a value that increases the score accumulated value.

The straight line detection unit <NUM> uses a vertical distance between each measurement point constituting the point cloud P1 and the straight line candidate in calculating the score accumulated value. Specifically, the straight line detection unit <NUM> assigns a higher priority score to points P1 within a specified distance from the straight line candidate than to points P1 in front of the straight line candidate at the specified distance or more (when viewed from the measurement direction of the two-dimensional distance measurement device <NUM>), and the closer to the straight line candidate, the higher the priority of the score. For example, the straight line detection unit <NUM> assigns a score with a negative constant to points P1 in front of the straight line candidate at the specified distance or more, and assigns a score with a positive constant to points P1 within the specified distance. The positive score value is a variable depending on the distance from the straight line candidate.

<FIG> is a graph showing an example of the score assigned by the pallet detection device <NUM> according to an embodiment. The horizontal axis of this graph corresponds to the depth, representing the distance between each point and the straight line candidate, and means that the further to the right in the horizontal direction, the further back (when viewed from the observation direction of the two-dimensional distance measurement device <NUM>). The position where the depth is zero is the position of the straight line candidate. The vertical axis represents the value of the score. In the example shown in <FIG>, points P1 in front of the straight line candidate at a specified distance or more are assigned with a score with a negative constant. Thus, a certain penalty is imposed on points P1 in front of the straight line candidate at a specified distance or more, regardless of the distance between the straight line candidate and each point. Points P1 within the specified distance from the straight line candidate are assigned with a positive score that is greater the closer it is to the candidate straight line.

In calculation of the score accumulated value, the straight line detection unit <NUM> excludes points P1 behind the straight line candidate at a specified distance or more from accumulation of the score accumulated value. The expression "excludes. from accumulation of the score accumulated value" means that such points P1 are substantially excluded from the accumulation target of the score accumulated value; for example, a score of zero may be assigned to these points P1, or no score may be assigned to these points P1. In the example shown in <FIG>, points P1 behind the straight line candidate at a specified distance or more are assigned with a score of zero. When this calculation method of the score accumulated value is adopted, in the example shown in <FIG>, the straight line candidate L1 is selected as the straight line corresponding to the front surface of the pallet <NUM>. The straight line detection unit <NUM> may detect a plurality of straight lines having a high priority indicated by the score accumulated value.

As shown in <FIG>, the pallet detection device <NUM> determines whether the straight line detection unit <NUM> detects the straight line (step S5). If it is determined that the straight line is not detected (step S5; No), the pallet detection device <NUM> performs a processing of step S10. If it is determined that the straight line is detected (step S5; Yes), the pallet detection device <NUM> performs a first both-end detection processing (step S6).

Specifically, in the first both-end detection processing, the both-end detection unit <NUM> extracts one or more line segments from the straight line selected by the straight line detection unit <NUM>, and searches the extracted one or more line segments for one line segment or a combination of two or more line segments having an end-to-end length corresponding to an end-to-end length of the pallet <NUM> determined from design information. The "length corresponding to. " means the length within tolerance. The design information of the pallet <NUM> indicates the dimension of the pallet <NUM> (e.g., end-to-end length, position and spacing of holes <NUM>) and is read from the memory of the pallet detection device <NUM>, for example. If the search in the first both-end detection processing is successful, the both-end detection unit <NUM> estimates both ends of the found line segment or combination of two or more line segments as both ends of the pallet <NUM>.

<FIG> is a schematic diagram showing an example of the first both-end detection processing performed by the pallet detection device <NUM> according to an embodiment. As shown in <FIG>, for example, the both-end detection unit <NUM> extracts three line segments (sets of points P1 surrounded by circles) from the selected straight line L5, and searches for one line segment or a combination of two or more line segments having an end-to-end length W2 corresponding to the end-to-end length of the pallet <NUM>. In the example shown in <FIG>, as a result of search, the end-to-end length of the combination of three line segments is detected.

The pallet detection device <NUM> determines whether both ends of the pallet <NUM> are detected by the first both-end detection processing (step S7). If it is determined that both ends of the pallet <NUM> are detected (step S7; Yes), the pallet detection device <NUM> performs a processing of step S12. If it is determined that both ends of the pallet <NUM> are not detected (step S7; No), the pallet detection device <NUM> performs a second both-end detection processing (step S8).

In the second both-end detection processing, if a gap between adjacent line segment end points of the extracted two or more line segments or a gap between an edge of the detection target region and a line segment end point is equal to or greater than a reference value, the both-end detection unit <NUM> detects both ends of the pallet <NUM> so that an end point of the gap is one end of the pallet <NUM>, and a point that is separated from the one end by a length corresponding to the end-to-end length of the pallet <NUM> is the other end of the pallet <NUM>. The detection target region in this case is preferably the first detection target region A1, not the second detection target region A2, in order to make it easier to find the gap.

<FIG> is a schematic diagram showing an example of the second both-end detection processing performed by the pallet detection device <NUM> according to an embodiment. In this example, another pallet <NUM> (10A) is located to the right of the pallet <NUM> (10B) to be detected. The pallet <NUM> (10B) to be detected and a part of the other pallet <NUM> (10A) are contained within the first detection target region A1. Further, these two pallets <NUM> are close together, and there is almost no gap. In this case, the first both-end detection processing fails.

However, in the second both-end detection processing, if the width W1 of the gap between the left end of the pallet <NUM> (10B) to be detected and the left end of the first detection target region A1 is equal to or larger than a reference value, the end point of the line segment corresponding to the right end point of the gap (the left end of the pallet <NUM> to be detected) is detected as one end of the pallet <NUM> (10B), and a position separated from this end along the line segment by the length W2 corresponding to the end-to-end length of the pallet <NUM> is detected as the other end of the pallet <NUM> (10B). Thus, both ends of the pallet <NUM> (10B) are detected in the example shown in <FIG>.

In <FIG>, if another pallet <NUM> (not shown) is also located to the left of the pallet <NUM> (10B) to be detected, and is partially contained within the first detection target region A1, the gap between adjacent line segment end points of the extracted two or more line segments is examined. For example, if the distance between the right line segment composed of points P1 of the other pallet <NUM> (not shown) and the left line segment composed of points P1 of the pallet <NUM> (10B) to be detected is equal to or larger than a reference value, the left end of the left line segment composed of points P1 of the pallet <NUM> (10B) to be detected is detected as one end of the pallet <NUM> (10B), and a position separated from this end along the line segment by the length W2 corresponding to the end-to-end length of the pallet <NUM> is detected as the other end of the pallet <NUM> (10B).

The pallet detection device <NUM> determines whether both ends of the pallet <NUM> are detected by the second both-end detection processing (step S9). If it is determined that both ends of the pallet <NUM> are detected (step S9; Yes), the pallet detection device <NUM> performs a processing of step S12. If it is determined that both ends of the pallet <NUM> are not detected (step S9; No), the pallet detection device <NUM> performs a third both-end detection processing (step S10).

<FIG> is a schematic diagram showing an example of the third both-end detection processing performed by the pallet detection device <NUM> according to an embodiment. In the third both-end detection processing, for instance as shown in <FIG>, the both-end detection unit <NUM> searches the point cloud P1 for three sets of points having spacings W3, W4 equal to the spacing between ends of the pallet <NUM> or ends of the holes <NUM> of the pallet <NUM>, and detects both ends of the pallet <NUM> so that the forwardmost point (the point closest to the two-dimensional distance measurement device <NUM>) of the three sets of points is one end of the pallet <NUM>, and a point that is separated from the one end by a length corresponding to the end-to-end length (W2) of the pallet <NUM> is the other end of the pallet <NUM>.

In the illustrated example, the two-dimensional distance measurement device <NUM> is located to the left of the pallet <NUM> and scans the pallet <NUM> from a direction oblique to the pallet <NUM>. In this case, the sets of points near the three corners of the pallet <NUM> are acquired as the point cloud. Of these three pairs of points, for example, the leftmost point of the left pair is detected as one end of the pallet <NUM> since it is located furthest forward. A point separated from this end by the length W2 corresponding to the end-to-end length of the pallet <NUM> is detected as the other end. Thus, both ends of the pallet <NUM> are detected. In order to prevent false detections, detection may be determined as failure when there is a plurality of three pairs of points.

The pallet detection device <NUM> determines whether both ends of the pallet <NUM> are detected by the third both-end detection processing (step S11). If it is determined that both ends of the pallet <NUM> are detected (step S11; Yes), the pallet detection device <NUM> performs a line segment fitting (step S12).

<FIG> is a schematic diagram showing an example of position and orientation of the pallet <NUM> acquired by the pallet detection device <NUM> according to an embodiment. In the line segment fitting, first, a line segment indicating the front surface of the pallet <NUM> is detected based on the straight line in which both ends of the pallet <NUM> are detected. In this case, for example, as shown in <FIG>, the line segment detection unit <NUM> may exclude points P1 at the position presumed to be the position of the hole <NUM> of the pallet <NUM> from the point cloud P1, further exclude points P1 away from both ends of the line segment by a predetermined distance or more, and then use only the rest of points P1 to re-detect the line segment indicating the front surface of the pallet <NUM>. In <FIG>, points P2 excluded from the point cloud P1 are represented by white plots. The position of the hole <NUM> of the pallet <NUM> is estimated based on the detected both ends of the pallet <NUM> and the design information of the pallet <NUM>. Further, the line segment detection unit <NUM> may re-detect the line segment indicating the front surface of the pallet <NUM> by the least squares method so as to pass closer to more points P1.

Further, the pallet detection device <NUM> calculates the position and orientation of the pallet <NUM> based on the line segment indicating the front surface of the pallet <NUM> finally detected (step S13). Specifically, the position/orientation acquisition unit <NUM> acquires the position and orientation of the pallet <NUM> based on the line segment detected by the line segment detection unit <NUM>. At this time, the position/orientation acquisition unit <NUM> acquires the orientation of the pallet <NUM> from the slope of the line segment indicating the front surface of the pallet <NUM> finally detected and acquires the position of the pallet <NUM> based on the midpoint of the line segment indicating the front surface of the pallet <NUM> finally detected.

If it is determined that both ends of the pallet <NUM> are not detected (step S11; No), the pallet detection device <NUM> skips the processings of steps S12 and S13. The pallet detection device <NUM> outputs the detection result of the processing to the driving unit (not shown) of the forklift <NUM> (step S14).

The above-described process can solve the various problems described with reference to <FIG>. For example, even when there is almost no gap between the pallet <NUM> (10B) and the pallet <NUM> (10A) as shown in <FIG>, the straight line corresponding to the front surface of the pallet <NUM> (10B) can be detected by the second both-end detection processing shown in <FIG>. Even when the gap between the pallet <NUM> (10B) and the pallet <NUM> (10A) has a width equal to the distance D of the hole <NUM> as shown in <FIG>, the straight line corresponding to the front surface of the pallet <NUM> (10B) can be detected by the first both-end detection processing shown in <FIG>.

As shown in <FIG>, even when the pallet <NUM> (10B) has an angular deviation, the pallet <NUM> (10B) can be transported by inserting the fork <NUM> of the forklift <NUM> based on the calculation result of the position and orientation of the pallet <NUM> (10B) in step S13. As shown in <FIG>, even when the pallet <NUM> (10B) is placed farther back than other pallets <NUM> (10A, 10C), the front surface of the pallet <NUM> can be detected by distinguishing points P1 on the front side from points P1 on the back side based on the calculation result of the score accumulated value of step S3.

The flow of the processing performed by the pallet detection device <NUM> has been described. The order of steps shown in <FIG> can be changed as appropriate. For example, the order of the second both-end detection processing and the third both-end detection processing may be reversed. In the straight line detection in step S4, a plurality of straight lines may be detected. In this case, the processings of steps S6, S8, and S10 may be performed for each of the plurality of straight lines. For example, the pallet detection device <NUM> may select a straight line out of the plurality of detected straight lines in the order of priority indicated by the score accumulation value, and perform a processing for detecting both ends of the pallet <NUM> based on the selected straight line (first both-end detection processing, second both-end detection processing, third both-end detection processing). The position/orientation acquisition unit <NUM> acquires the position and orientation of the pallet <NUM> based on the straight line in which both ends of the pallet <NUM> are detected.

In the processing shown in <FIG>, if the search in the first both-end detection processing fails, an additional processing (second both-end detection processing and third both-end detection processing) different from the first both-end detection processing is performed. Such an additional processing may be omitted, or the processing may be modified to include only one of the two both-end detection processing or the third both-end detection processing.

The present disclosure is not limited to the embodiments described above, but includes modifications to the embodiments described above, and embodiments composed of combinations of those embodiments.

The contents described in the above embodiments would be understood as follows, for instance.

When three-dimensional data is acquired while moving 2D-LiDAR in the vertical direction as in Patent Document <NUM>, a large number of observation points due to reflected light from the front surface of the pallet (<NUM>) are obtained on the same plane, and the straight line corresponding to the front surface of the pallet (<NUM>) can be detected relatively easily. However, when trying to extract the straight line corresponding to the front surface of the pallet (<NUM>) in one scan without moving 2D-LiDAR in the vertical direction, it is difficult to distinguish observation points due to the reflected light from the front surface of the pallet (<NUM>) from observation points due to the reflected light reflected from holes of the pallet, so that a straight line behind the front surface of the pallet (<NUM>) may be extracted.

In other words, if a line segment is detected based on the point cloud in the hole (<NUM>) behind the front surface of the pallet (<NUM>) (point cloud (P1) reflected by the side or top surface in the hole (<NUM>)), the detection accuracy is reduced. In this regard, according to the configuration of claim <NUM>, the score accumulated value is obtained such that a straight line candidate with many points (P1) in front thereof has a low priority in selection, and a straight line is selected based on the score accumulated value. This makes it easier to detect the line segment based on the point cloud (P1) on the front surface of the pallet (<NUM>). As a result, the straight line corresponding to the front surface of the pallet (<NUM>) can be accurately extracted without moving the two-dimensional distance measurement device (<NUM>) in the vertical direction, so that the position and orientation of the pallet (<NUM>) can be detected in a short time.

If the score given to the point cloud (P1) on the front side is a variable such that the priority decreases significantly with a distance from the straight line candidate, there is a risk that a line segment is detected on the front side of the pallet (<NUM>) due to the influence of outliers (for example, the influence of a reflection source by an object other than the target pallet (<NUM>)). In this regard, according to the configuration of claim <NUM>, the score is a constant, which reduces such a risk. Therefore, it is possible to improve the detection accuracy.

According to the configuration of claim <NUM>, a higher score is assigned to points (P1) within the specified distance from the straight line candidate than to points (P1) in front of the straight line candidate at the specified distance or more, and the closer to the straight line candidate, the higher the score. Accordingly, the more points there are in the vicinity of the straight line candidate, the easier it is to select this straight line candidate as the straight line. Thus, it is possible to detect the line segment that is most likely to be the front surface of the pallet (<NUM>).

Points (P1) backward of the front surface of the pallet (<NUM>) are most likely points (P1) in the hole (<NUM>). The positions of such points (P1) are noise in the detection of the front surface of the pallet (<NUM>), and should therefore not be considered as a score used for straight line selection. If they are considered, the accuracy may be reduced. In this regard, according to the configuration of claim <NUM>, points (P1) behind the straight line candidate at the specified distance or more are excluded from accumulation of the score accumulated value. Thus, it is possible to reduce a risk of accuracy degradation due to the influence of points (P1) in the hole (<NUM>).

Even the straight line selected by the straight line detection unit (<NUM>) may not be appropriate as a straight line to be used for line segment detection. In this regard, according to the configuration of claim <NUM>, the processing for detecting both ends of the pallet (<NUM>) is performed by selecting the straight line in the order of the priority indicated by the score accumulated value, and the position and orientation of the pallet (<NUM>) are obtained based on the successful straight line. As a result, the detection accuracy is improved.

According to the configuration of claim <NUM>, the first detection target region (A1) having a width larger than that of the slot (<NUM>) is used to extract the point cloud (P1). Thus, even if the first detection target region (A1) is set at a position slightly displaced from the position of the slot (<NUM>) due to an error, the point cloud (P1) at the position of the slot (<NUM>) can be acquired without omissions.

If the detection target region is wide, the point cloud (P1) of the pallet (<NUM>) adjacent to the target pallet (<NUM>) may enter the detection target region. In this case, even if a straight line suitable for detecting the line segment indicating the front surface of the target pallet (<NUM>) is detected, the priority of the score accumulated value may be reduced by the point cloud (P1) caused by the front surface of the adjacent pallet (<NUM>) located further forward. In this case, the detection accuracy may decrease. In this regard, according to the configuration of claim <NUM>, since the score accumulated value is calculated for the point cloud (P1) in the second detection target region (A2) having a width smaller than the first detection target region (A1), the influence of the adjacent pallet (<NUM>) on the score accumulated value can be reduced. In addition, since the point cloud (P1) in the first detection target region (A1) is also included when detecting the straight line, it is possible to reduce the possibility that the point cloud (P1) of the target pallet (<NUM>) included in the first detection target region (A1) may be omitted from extraction when selecting the straight line.

According to the configuration of claim <NUM>, it is possible to narrow down the region where the point cloud (P1) should be acquired. Further, when multiple pallets (<NUM>) are adjacent to each other, information indicating the position of the slot (<NUM>) can be used to reduce a risk of acquiring the point cloud (P1) of the pallet (<NUM>) whose position and orientation is not the detection target.

According to the configuration of claim <NUM>, it is possible to acquire one line segment or a combination of two or more line segments that are estimated as both ends of the pallet (<NUM>). Since information about the holes (<NUM>) in the pallet (<NUM>) is not used, even in situations where the holes (<NUM>) cannot be recognized, the search by the first both-end detection processing is not affected, thus improving robustness.

It is possible that the first both-end detection processing cannot find one line segment or a combination of two or more line segments that can be estimated as both ends of the pallet (<NUM>). For example, when the target pallet (<NUM>) is close to the adjacent pallet (<NUM>), points (P1) of these pallets (<NUM>) may be detected as one continuous line segment. For example, due to the angle of incidence of the measurement light to the pallet (<NUM>), there is a possibility that only points (P1) at the corner of the pallet (<NUM>) can be observed. In this regard, according to the configuration of claim <NUM>, even if the search in the first both-end detection processing fails, since both ends of the pallet (<NUM>) can be detected by performing the additional processing, it is easy to succeed in detecting both ends of the pallet (<NUM>).

Even if the first both-end detection processing is performed, when the target pallet (<NUM>) is close to the adjacent pallet (<NUM>), points (P1) of these pallets (<NUM>) may be detected as one continuous line segment. In this case, both ends of the pallet (<NUM>) may not be detected by the first both-end detection processing. In this regard, according to the configuration of claim <NUM>, even if the detection by the first both-end detection processing fails, when a gap equal to or larger than a reference value is found, both ends of the pallet (<NUM>) can be detected.

For example, due to the angle of incidence of the laser light to the pallet (<NUM>), there is a possibility that only points (P1) at the corner of the pallet (<NUM>) can be observed. In this case, both ends of the pallet (<NUM>) may not be detected by the first both-end detection processing. In this regard, according to the configuration of claim <NUM>, even if the detection by the first both-end detection processing fails, when two or more sets of points having a spacing equal to a spacing between ends of the pallet (<NUM>) or ends of holes (<NUM>) in the pallet (<NUM>) is found, both ends of the pallet (<NUM>) can be detected.

According to the configuration of claim <NUM>, the pallet (<NUM>) can be detected using 2D-LiDAR for safety of the forklift (<NUM>). This eliminates the need to install a dedicated sensor for pallet detection, thus reducing the cost.

Since the 2D-LiDAR for safety of the forklift (<NUM>) has a fixed height position, it cannot be applied to a configuration that detects the position and orientation of the pallet (<NUM>) while moving the laser sensor in the vertical direction. However, in the configuration of claim <NUM>, such 2D-LiDAR is applicable since it can detect the position and orientation of the pallet (<NUM>) without moving the laser sensor in the vertical direction.

A forklift (<NUM>) according to the configuration of claim <NUM> can transport the pallet (<NUM>) based on the detection result of the pallet detection device (<NUM>).

With a method according to claim <NUM>, the straight line corresponding to the front surface of the pallet (<NUM>) can be accurately extracted without moving the two-dimensional distance measurement device (<NUM>) in the vertical direction, so that the position and orientation of the pallet (<NUM>) can be detected in a short time.

Claim 1:
A pallet detection device (<NUM>), comprising:
a point cloud acquisition unit (<NUM>) configured to acquire point cloud data indicating a point cloud (P1) measured by a two-dimensional distance measurement device (<NUM>) on a depth map;
a straight line detection unit (<NUM>) configured to detect a straight line corresponding to a front surface of a pallet (<NUM>, 10A, 10B, 10C) based on the point cloud (P1) in a region presumed to include the front surface of the pallet (<NUM>, 10A, 10B, 10C) in the point cloud data;
a line segment detection unit (<NUM>) configured to detect a line segment indicating the front surface of the pallet (<NUM>, 10A, 10B, 10C) based on the straight line; and
a position/orientation acquisition unit (<NUM>) configured to acquire position and orientation of the pallet (<NUM>, 10A, 10B, 10C) based on the line segment,
wherein the straight line detection unit (<NUM>) is configured to:
acquire one or more straight line candidates as candidates for the straight line,
assign, for each of the one or more straight line candidates, a score to the point cloud (P1) so that points in front of the straight line candidate at a specified distance or more are assigned with a score with a constant whose priority in selection is lower than the rest of points,
select the straight line from the one or more straight line candidates based on a score accumulated value obtained by accumulating the score for the point cloud (P1) in the region presumed to include the front surface of the pallet (<NUM>, 10A, 10B, 10C), and
in the calculation of the score accumulated value, assign a higher priority score to points within the specified distance from the straight line candidate than to points in front of the straight line candidate at the specified distance or more, and the closer to the straight line candidate, the higher the priority of the score.