Patent Description:
Today's vehicles, including autonomous, driverless, unmanned or uncrewed, etc. ones, commonly use sensors mounted on the vehicle to detect possible obstacles and allow avoidance behavior, such as a complete stop or a trajectory change, or driver alerts, so as to navigate safely through environments. « Sensor » as used herein, designates an opto-electronic apparatus, such as for example a LIDAR sensor, based on the principle of emitting a lightwave beam, catching the reflections of objects (backscattering) in the field of view, and providing relative position information about reflecting objects, with a given scanning frequency in one or multiple scanning layers (or "rows" in the case of a Flash LIDAR). Such a sensor may for example be a sensor available on the market at the time of filing this patent application, such as reference VLP-<NUM> from Velodyne, and any subsequent equivalent technologies becoming available, regardless of their name.

In industrial environments where swap bodies are stored, assistance to autonomy or to a driver may be needed for the navigation of a vehicle used to approach the swap body, before docking under it, picking it up and transporting it elsewhere. Such assistance may take the form, through processing of data from the sensors mounted on the vehicle, of detection of the target swap body after identification of certain of its characteristics, that allows safe approach by the vehicle.

This detection may be particularly challenging where, without limitation:.

In addition, this detection has to be sufficiently fast so as to truly assist with the navigation. "Real time" processing of data provided by the sensors mounted on the vehicle is critical to the safety of the vehicle's docking operation under the swap body. For example, sensors may provide new data at a <NUM> frequency: for example depending on data processing hardware capabilities, the detection software may have to be optimized, for example by making a number of assumptions before calculation so as to simplify it, or by disregarding certain data provided by the sensors so as to limit the processing.

<CIT> discloses a forklift including a fork mounted on a vehicle body, and a laser sensor configured to generate track data for moving the vehicle body to a loading position of the load or the pallet based on distance data as measured by the laser sensor.

Publication entitled "<NPL>, discloses a sensor fusion method by utilizing LiDAR and Camera sensor together to develop a robust drivable road detection system for autonomous vehicles.

Publication entitled "<NPL>et al. , discloses a plane extraction method using point cloud data to perceive an unknown object for a service robot.

German patent application <CIT> discloses a method for the optical display and operation of image-supported navigation in vehicles and a device for using the method for image-supported navigation in vehicles, with which it is possible for the driver to find target objects and then approach quickly and precisely.

Embodiments of the present technology have been developed based on developers' appreciation of shortcomings associated with the prior art, in particular various aspects of the present technology provide a method and system for detection of a target swap body having a front plane, a side plane and four legs, to be picked up by a vehicle equipped with a fusion and labelling system and at least one sensor, wherein the pointcloud received from the fusion and labelling system is analysed toward detecting the front plane, and/or the side plane and/or at least three of the four legs, and a model is updated accordingly and provided to a navigation system of the vehicle.

Various aspects of the present technology provide for a method and system for detection of a target swap body as in the appended claims.

In the context of the present description, unless expressly provided otherwise, a "processor" may refer, but is not limited to, any type of "computing system", "electronic device", "computer-based system", "controller unit", "monitoring device", "server" and/or any combination thereof appropriate to the relevant task at hand, in relation to receiving, storing, processing, and/or forwarding data.

In the context of the present specification, the expression "FPGA" is intended to include Field Programmable Gate Array computing systems, available on the market at the time of filing this patent application, such as references Xilinx VU9P, or Intel Stratix V, and any subsequent equivalent technologies becoming available, regardless of their name, consisting in computing system hardware programmable with software.

In the context of the present description, a "processor" may include a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared. A "processor" may be a general purpose processor, such as a central processing unit (CPU), a processor dedicated to a specific purpose, or a processor implemented in a FPGA. Other hardware and software, conventional and/or custom, may also be included in a "processor".

In the context of the present description, unless expressly provided otherwise, the expression "memory" is intended to include Random Access storage systems, available on the market at the time of filing this patent application, and any subsequent equivalent technologies becoming available, regardless of their name, consisting in computing system media for storing digital information. An example of such memory may be a Quad Data Rate (QDR) Static Random Access Memory (SRAM).

<FIG> depicts a simplified top view of a vehicle in which the present technology may be implemented. Such a Vehicle <NUM>, may be equipped with, for example, five Sensors <NUM>-<NUM>. The respective fields of view 101a-105a of such sensors may cover operating range areas at the front (101a-103a) and at the rear (104a-105a) of the Vehicle <NUM>. As a reminder, fields of view are not represented to scale on <FIG>. Not only the number, but also the mounting location (including height relative to the ground) on the Vehicle <NUM>, of Sensors <NUM>-<NUM> are also depicted as mere examples, and are not limiting with the present technology.

Raw data provided by Sensors <NUM>-<NUM> may, as is known by the person skilled in the art, be fed to a Sensor scan fusion and labelling system <NUM>, which may process them and form a pointcloud therefrom, for example through fusion. The labelling processing itself by such a system may vary from quasi-absent to more sophisticated, and consist for example in a capability to associate with each point, in addition to XYZ coordinates, a probability information that a given point corresponds to the ground. All such optional labelling capabilities are compatible with the present technology.

The Vehicle <NUM> may also be equipped with a Vehicle navigation system <NUM>, which in the case of a driverless vehicle, may action traction and steering of the vehicle, or otherwise may control or alert systems to assist with the driving activity.

The present technology may, in such an example, as Swap body detection system <NUM>, sit between the Sensor scan fusion and labelling system <NUM> and the Vehicle navigation system <NUM> to process the pointcloud output by the Sensor scan fusion and labelling system <NUM>, toward detecting a swap body and forming a model thereof that may be fed to the Vehicle navigation system <NUM>. In such an example, the Swap body detection system <NUM> may also interact with the Vehicle navigation system <NUM> and receive feedback from it.

A "model" as used herein, is information related to a swap body that both is calculated by Swap body detection system <NUM> based on the pointcloud received from the Sensor scan fusion and labelling system <NUM>, and may be used for assistance by the Vehicle navigation system <NUM> when provided to it by the Swap body detection system <NUM>.

In aspects of the present technology, the model may further comprise without limitation other information features related to planes, such as their orientation, or a probability that the model matches a particular swap body type, etc..

Finally, even though not represented on <FIG>, it will be apparent to the person skilled in the art that Vehicle <NUM> may be communicably connected, whether it be an autonomous vehicle or not, to a distant server, or other computing system, providing information to including Swap body detection system <NUM> and/or Vehicle navigation system <NUM>. This communication may take place through wire (for example when Vehicle <NUM> is parked) or wirelessly (for example when Vehicle <NUM> is in motion). This information may for example be related to navigation of Vehicle <NUM> toward a swap body, or expected location or orientation of a particular swap body, or input parameters for the calculations involved in each of the Swap body detection system <NUM> and Vehicle navigation system <NUM>, that may be received at Vehicle <NUM> directly by both systems, or either one of them, and transmitted to the other through interaction between the two.

It will be also appreciated by the person skilled in the art that, although conceptually represented physically separated on <FIG>, blocks such as Sensor scan fusion and labelling system <NUM>, Vehicle navigation system <NUM>, and Swap body detection system <NUM> merely represent functions which may be realized in a single physical entity or in separate physical entities. Conversely, even though represented physically included in the Vehicle <NUM>, certain of these blocks could represent functions that are performed remotely. Each block may comprise hardware and/or software and/or firmware (or a combination thereof), as is known, to execute functions according to the invention, including the processes described in relation to <FIG>.

<FIG> illustrates a swap body that may be detected using the present technology. The assumptions made by the present technology during the detection process, is that a Swap body <NUM> is expected to:.

Apart from possibly the width of the front plane <NUM> (ex: <NUM>), swap bodies' dimensions are very little standardized.

In addition, legs through wear and tear may be damaged, and/or be not strictly perpendicular to the lower plane of the parallelepiped (for example legs are often retractable under the swap body, and may not have been put back up in a complete vertical position).

<FIG> illustrates an industrial environment wherein assistance to the approach of a vehicle toward a swap body to be picked up (as target swap body) may be provided by the present technology. The Swap body <NUM>, to be picked up by the Vehicle <NUM>, may be stored among other stored swap bodies, for example swap bodies <NUM> and <NUM> illustrated on <FIG>. It will be appreciated that other swap bodies may not be perfectly aligned in the industrial environment of storage, with the target swap body.

The Vehicle <NUM> may be uploaded with target swap body expected location and orientation information regarding the location and/or the orientation of the Swap body <NUM>, which may be both hypothetical (the target Swap body <NUM> may actually not be at that location) and approximate (the information of expected location or orientation has a given margin of error). This information may for example consist in a map of the industrial environment. It may also for example be XY coordinates of the centre point of a horizontal plane of the Swap body <NUM>, combined with a vector extending from such point and indicating an orientation for Swap body <NUM>. Such target swap body expected location and orientation will help bring Vehicle <NUM> (whether autonomously or with the assist of a driver) up to point A on <FIG>, ie in the vicinity of target Swap body <NUM>.

Then starting with trajectory <NUM> (from point A to point B), the detection according to the present technology is triggered, along with the interaction between Swap body detection system <NUM> and the Vehicle navigation system <NUM> (from <FIG>). It may be triggered automatically, or manually by an operator even in the case of an autonomous Vehicle <NUM>.

During trajectory <NUM> (from point B to point C):.

It will be apparent to the person skilled in the art that:.

Once the Swap body <NUM> is docked and picked up, the Vehicle <NUM> may follow a trajectory <NUM> (from point C to point D) to transport the Swap body <NUM> to a different location.

It will be apparent to the person skilled in the art that what has been described above for picking up Swap body <NUM> is equally applicable to unloading it (with detection instead that there is no swap body already in the storage spot where the Swap body <NUM> is intended to be unloaded).

<FIG> presents an overview of the method steps in the present technology. At step <NUM>, the overall detection method is triggered for a target swap body. It may be triggered in different ways, all within the scope of the present technology. For example, the method may be triggered manually, for example by a driver, when the vehicle is located in the vicinity of the target swap body to dock under and pick up. For example, the method may be triggered automatically when the vehicle reaches a location in the vicinity of the target swap body to dock under and pick up. For example, the Vehicle <NUM> of <FIG>, in the particular case of an autonomous one, may be uploaded with the expected location and orientation information of the Swap body <NUM>, from which a vicinity location may be calculated (as an example the point A on <FIG>): when Vehicle <NUM> reaches such vicinity location, the method may be triggered automatically.

At step <NUM>, a pointcloud is received that is formed from data provided by at least one sensor mounted on the vehicle. The person skilled in the art knows how to form such a pointcloud, for example through known fusion systems. For example, a fusion of data from all five Sensors <NUM>-<NUM> of Vehicle <NUM> of <FIG> may be performed by Sensor scan fusion and labelling system <NUM>, and provided to the Swap body detection system <NUM>.

In aspects of the present technology, the forming of the pointcloud may be augmented with a labelling of the points, as is known in the art. For example, Sensor scan fusion and labelling system <NUM> may associate with each point, in addition to XYZ coordinates, a probability information that a given point corresponds to the ground, and provide such coordinates and probability information to the Swap body detection system <NUM>. Other such labelling options may be had, that become optional input parameters provided to the Swap body detection system <NUM>, see for example under "Input parameters" below.

At step <NUM>, two detection systems are applied on the received pointcloud: a first system aims at identifying in the pointcloud/detecting a front and a side plane of the target swap body, and a second system aims at identifying in the pointcloud/detecting the front plane and at least three legs of the target swap body. For example, Swap body detection system <NUM> of Vehicle <NUM> of <FIG>, may apply such detection systems on the pointcloud received from Sensor scan fusion and labelling system <NUM>.

At step <NUM>, an assessment of the result for the first detection system is made: if a front and a side plane of the target swap body are identified in the pointcloud/detected, then a model for the target body is updated (or created if none exists). If a front plane only is identified in the pointcloud/detected, this information is kept for use in the next step. For example, Swap body detection system <NUM> of Vehicle <NUM> of <FIG>, may make such assessment relative to the pointcloud received from Sensor scan fusion and labelling system <NUM>, and create/update a model for the target swap body.

At step <NUM>, an assessment of the result for the second detection system is made: if a front plane and at least three legs of the target swap body are identified in the pointcloud/detected, then the model for the target body is updated (or created if none exists). For example, Swap body detection system <NUM> of Vehicle <NUM> of <FIG>, may make such assessment relative to the pointcloud received from Sensor scan fusion and labelling system <NUM>, and create/update the model for the target swap body.

It will be apparent to the person skilled in the art that a different order in the respective detection assessments of front plane, side plane and legs may be had still within the scope of the method according to the present technology.

At step <NUM>, if the model for the target swap body has been created/updated at step <NUM> or <NUM>, then the current version of the model for the target swap body, if any is available, is provided to the vehicle for assistance to navigation. For example, Swap body detection system <NUM> of Vehicle <NUM> of <FIG>, may provide such information to the Vehicle navigation system <NUM>.

In aspects of the present technology, while the current model is provided to a navigation system of the vehicle only when created/updated:.

In other aspects of the present technology, the current model may be provided to the navigation system of the vehicle at each pointcloud formation cycle, regardless of whether it has been created or updated during the cycle.

At step <NUM>, a test is made whether the vehicle has reached a location that enables picking up the target swap body. If it has, the overall target swap body detection is ended at step <NUM>. If not, overall target swap body detection is resumed at step <NUM>, for the next, or a next, cycle of available and received pointcloud that is formed from data provided by the at least one sensor mounted on the vehicle.

The detection of the vehicle reaching the enabling location leading to the overall detection method being stopped, may be achieved in different ways, all within the scope of the present technology. For example, the method may be stopped manually, for example by a driver, when the vehicle has docked under the target swap body. For example, the method may be stopped automatically when it is automatically detected that the vehicle has docked under the target swap body. For example, the Vehicle <NUM> of <FIG>, in the particular case of an autonomous one, may recognize, for example through GPS calculation capabilities, that it has docked under the target swap body (as an example it has reached point C on <FIG>): when Vehicle <NUM> reaches a particular location where assistance to navigation is no longer necessary or helpful to the picking up of the target swap body, the method may be stopped automatically.

We will now go into details of calculation, for example by the Swap body detection system <NUM>, that may be involved with aspects of the present technology, as well as of input parameters, and potential optimization for the calculation.

The person skilled in the art knows of algorithms aimed at processing a pointcloud toward identifying in the pointcloud/detecting planes, for example a front and a side plane. In the particular case of a swap body, such planes are perpendicular. This characteristic may be exploited, and in aspects of the present technology, the identifying in the pointcloud/detecting a front and a side plane of the target swap body may comprise applying an algorithm that allows more accurate grouping of points belonging to the same family (here: to the same front or side plane), ie: Randomized Hough Transform. Such a technique is for example described in the article: "<NPL>. Such a technique allows to better reconstruct normals at a corner, such as at the intersection between the front plane and the side plane. Each point is thus provided with an estimate for its normal.

Points with substantially the same normal may then be grouped in at least two clusters using for example a kd-tree algorithm.

To obtain a line equation once the points are grouped, a RANSAC algorithm may be used.

A front and a side plane may be detected if the vectors corresponding to the at least two line equations are substantially perpendicular.

An inlier rate may be calculated to check the line equation calculation.

The person skilled in the art knows of algorithms aimed at processing a pointcloud toward identifying in the pointcloud/detecting three legs. It also knows how of ways to combine the information of a previously identified in the pointcloud/detected front plane with the operation of identifying in the pointcloud/detecting three feet.

In aspects of the present technology, the identifying in the pointcloud/detecting at least three legs of the target swap body may further comprise:.

Optional input parameters (as defined below) in relation to expected dimensions of legs may be used to filter out potentially unknown objects lying near the target swap body that represent a risk to the vehicle, or avoid detection of legs that are too damaged (for example not quite straight).

Optionally also, the information may be used that at least three legs have been detected even though no front face has been detected, to partially update the model as regard orientation of the target swap body, possibly also to infer that the target swap body matches an expected type for the swap body (if that information is part of input parameters, see below).

In aspects of the present technology, the foregoing algorithms and calculation are fed with certain input parameters that are provided to Vehicle <NUM> as described above, or provided to Swap body detection system <NUM> by another system of Vehicle <NUM>, for example Sensor scan fusion and labelling system <NUM> or Vehicle navigation system <NUM>.

Apart from a poincloud to be processed, necessary input parameters comprise for example:.

Other input parameters may be optional, such as:.

In aspects of the present technology, the foregoing algorithms and calculation may be optimized. This may be necessary when for example the computing in a particular hardware and/or software environment can hardly match the frequency at which pointclouds are received, and generally the requirements of "real time" processing.

For example, the number of actually processed points from the received pointcloud may be limited, as may be afforded by the input parameters pursuant to the section above.

Further, developers of the present technology have, as an option, taken advantage of the situation when:.

<FIG> illustrates a computing system that may be used in the present technology, for example in any one or all of the functional elements Swap body detection system <NUM>, Sensor scan fusion and labelling system <NUM> and/or Vehicle navigation system <NUM> of <FIG>. As will be appreciated by the person skilled in the art, such computing system may be implemented in any other suitable hardware, software, and/or firmware, or a combination thereof, and may be a single physical entity, or several separate physical entities with a distributed functionality.

In some aspects of the present technology, the Computing system <NUM> may comprise various hardware components including one or more single or multi-core processors collectively represented by a Processor <NUM>, a Memory <NUM> and an Input/output interface <NUM>. In this context, the Processor <NUM> may or may not be included in a FPGA. In some other aspect, the Computing system <NUM> may be an "off the shelf" generic computing system. In some aspect, the Computing system <NUM> may also be distributed amongst multiple systems. The Computing system <NUM> may also be specifically dedicated to the implementation of the present technology. As a person in the art of the present technology may appreciate, multiple variations as to how the Computing system <NUM> is implemented may be envisioned without departing from the scope of the present technology.

Communication between the various components of the Computing system <NUM> may be enabled by one or more internal and/or external Buses <NUM> (e.g. a PCI bus, universal serial bus, IEEE <NUM> "Firewire" bus, SCSI bus, Serial-ATA bus, ARINC bus, etc.), to which the various hardware components are electronically coupled.

The Input/output interface <NUM> may enable networking capabilities such as wire or wireless access. As an example, the Input/output interface <NUM> may comprise a networking interface such as, but not limited to, a network port, a network socket, a network interface controller and the like. Multiple examples of how the networking interface may be implemented will become apparent to the person skilled in the art of the present technology.

The Memory <NUM> may store Code instructions <NUM>, such as those part of, for example, a library, an application, etc. suitable for being loaded into the Memory <NUM> and executed by the Processor <NUM> for implementing the method and process steps according to the present technology. The Memory <NUM> may also store a Database <NUM>. The person skilled in the art will appreciate that any of the Database <NUM>, the Code instructions <NUM>, and generally the Memory <NUM>, may also physically reside outside of the Computing System <NUM>, still within the scope of the present technology.

Claim 1:
A method for detection of a target swap body (<NUM>) having a front plane (<NUM>), a side plane (<NUM>) and four legs (<NUM><NUM>-<NUM><NUM>), to be picked up by a vehicle (<NUM>) equipped with a fusion and labelling system (<NUM>) and at least one sensor (<NUM>-<NUM>), wherein the method is triggered (<NUM>) when the vehicle reaches a first location within a range of the target swap body, the method comprising:
- receiving (<NUM>) from the fusion and labelling system (<NUM>) a pointcloud formed from data provided by the at least one sensor;
the method being characterized in that it further comprises:
- processing (<NUM>) the received pointcloud by:
- if (<NUM>) the front and the side plane of the target swap body are detected, creating a model, or updating a current model for the target swap body;
- if (<NUM>) the front plane and at least three of the four legs of the target swap body are detected, creating a model, or updating the current model for the target swap body;
- if at least three of the four legs of the target swap body are detected and no front plane is detected, updating the current model for the target swap body as regard the calculated orientation for the target swap body; and
- if (<NUM>) a model for the target swap body has been created or updated, providing the current model for the target swap body to a navigation system (<NUM>) of the vehicle (<NUM>), wherein the current model comprises a calculated location and orientation for the target swap body; and
- repeating (<NUM>) the receiving and processing to assist the vehicle (<NUM>) navigate from the first location to a second location enabling picking up the target swap body (<NUM>).