Patent Description:
The obstacle avoidance field of mobile robots is an important part of the practical application of mobile robots. To move in an environment as autonomously as human beings, a mobile robot is required to have good obstacle recognition ability. Obstacle avoidance methods have been continuously updated and developed. Recently, corresponding deep learning methods have been proposed using different sensors in order to provide more comprehensive and more robust obstacle detection algorithms.

In external applications, obstacle detection is mainly realized by laser radars or RGBD cameras. For an indoor mobile robot, obstacle detection is implemented mainly by a planar laser radar. Although obstacles at a certain height can be recognized in these solutions, it is very difficult to recognize small obstacles and depressions since it is difficult to mount the camera at a very low position on the robot and ensure that the camera is completely in a horizontal state. However, these small obstacles and depressions will affect the normal operation of the robot. For example, the wheels of a cleaning robot may be entangled by very thin wires, or the cleaning robot may smear a pet's excrement on the floor during operation, or the like.

<NPL>" discloses an algorithm for obstacle segmentation.

<NPL>" discloses and summarizes existing studies on point cloud segmentation.

<NPL>" discloses a prototype of inertial system coupled with mobile system.

<NPL>" discloses a navigation algorithm for mobile robot with high accuracy and robustness.

<CIT> discloses a method and device for obstacle recognition based on point cloud data.

To overcome the problems in the related art, the present invention provides a method, an apparatus and a non--transitory computer-readable storage medium for detecting small obstacles, as defined in independent claims <NUM>, <NUM> and <NUM>, respectively.

The technical solutions provided in the embodiments of the present invention may have the following beneficial effects. In this method, a first 3D point cloud corresponding to image data acquired by a cleaning robot is used, and a part belonging to a ground region is extracted, from the first 3D point cloud, as a target range of interest. An existence range of common obstacles is further extracted from the target range of interest by using the characteristic that height information is carried in the point cloud. Morphological analysis is performed on ground projection information of this range, and a small obstacle range to be detected is selected from each suspected obstacle region. Small obstacles on the ground can be effectively recognized by this method.

It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and not intended to limit the present invention.

The accompanying drawings to be described herein are incorporated into this specification and constitute a part of this specification. These accompanying drawings show the embodiments of the present invention, and are used with this specification to explain the principle of the present invention.

Exemplary embodiments will be described in detail herein, and examples in the exemplary embodiments are shown in the accompanying drawings. When the accompanying drawings are involved in the following description, unless otherwise indicated, identical reference numerals in different accompanying drawings indicate identical or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present invention. Instead, the implementations are merely examples of apparatuses and methods consistent with some embodiments of the present invention as described in the appended claims.

An example embodiment of the present disclosure, which is not in accordance with the claimed invention, provides a method for detecting small obstacles. With reference to <FIG> is a flowchart of a method for detecting small obstacles according to an exemplary embodiment. This method is applied to a cleaning robot. As shown in <FIG>, the method includes:.

Each point in the first 3D point cloud in the step S11 has 3D information that is used for indicating the spatial location of each point. The first 3D point cloud is calculated from the image data acquired by the cleaning robot through coordinate transformation. Since the position of the camera on the cleaning robot cannot be too close to the ground, the images acquired by the camera include images of non-ground parts (e.g., wall, furniture, etc.) in addition to images of ground parts.

In the step S12, the second 3D point cloud is a ground region part in the first 3D point cloud. The second 3D point cloud can be recognized from the first 3D point cloud by different ground region recognition algorithms.

When the set height range in the step S13 is a range containing a positive range and a negative range (e.g., greater than -<NUM> and less than <NUM>), raised obstacles and depressed obstacles on the ground can be simultaneously detected by this method. When the set height range is a range greater than <NUM> (e.g., greater than <NUM> and less than <NUM>), raised obstacles can be detected by this method. When the set height range is a range less than <NUM> (e.g., greater than -<NUM> and less than <NUM>), depressed obstacles can be detected by this method.

In the step S14, during the calculation of the ground projection point cloud of the third 3D point cloud, a ground plane in the third 3D point cloud is firstly determined, and the third 3D point cloud is then projected onto this ground plane to obtain a two-dimensional ground projection point cloud. With reference to <FIG> is a schematic view of a ground projection point cloud. In <FIG>, a continuous part having the maximum gray value is a part of a ground projection point cloud corresponding to a raised obstacle on the ground.

In this method, a first 3D point cloud corresponding to image data acquired by a cleaning robot is used, and a part belonging to a ground region is extracted, from the first 3D point cloud, as a target range of interest. An existence range of common obstacles is further extracted from the target range of interest by using the characteristic that height information is carried in the point cloud. Morphological analysis is performed on ground projection information of this existence range, and a small obstacle range to be detected is selected from each suspected obstacle region. Small obstacles on the ground can be effectively recognized by this method.

An embodiment of the present disclosure further provides a method for detecting small obstacles. In this method, the way to acquire 3D point cloud data in the step S11 shown in <FIG> is one of the following ways.

In the way <NUM> and way <NUM>, the depth image, also referred to as a range image, refers to an image using the distance (depth) from the image collector to each point in the scenario as a pixel value. Each pixel point in the depth image represents the distance from an object closest to the plane of the camera to the plane, the object being at the particular coordinates (x,y) in the field of view of one or more depth sensors to an object.

The process of performing a coordinate transformation on the depth image to obtain the first 3D point cloud data is realized according to a camera calibration principle. The camera calibration principle is formalized as: <MAT> where u and v are any coordinate points in an image coordinate system; u<NUM> and v<NUM> are central coordinates of the image, respectively; xw , yw and zw are three-dimensional coordinate points in a world coordinate system; zc represents a z-axis value of camera coordinates, i.e., a distance from the target to the camera; and, R and T are a 3x3 rotation matrix and a 3x1 translation matrix of external parameter matrices, respectively.

Since the origin of world coordinates and the origin of the camera coincide, that is, there is no rotation or translation, then: <MAT>.

The origin of the camera coordinate system and the origin of the world coordinate system coincide, so a same object in the camera coordinates and the world coordinates has a same depth, that is, zc = zw. Hence, the formula can be further simplified as: <MAT>.

The transformation formula from the image point [u, v]T to the world coordinate point [xw, yw, zw]T can be calculated from the above transformation matrix:.

The image acquired by the camera mounted on the cleaning robot generally includes a ground region and a non-ground region. To recognize small obstacles, it is necessary to obtain the ground region accurately. The small jitter or jolt of the cleaning robot during its movement due to uneven ground or unsmooth start/stop will lead to a large recognition error of the ground region, thereby affecting the subsequent recognition of small obstacles.

To solve the problem of the large recognition error of the ground region due to the jitter or jolt of the cleaning robot during running, an embodiment of the present disclosure further provides a method for detecting small obstacles. With reference to <FIG> shows a method for detecting small obstacles according to an exemplary embodiment, not forming part of the present invention.

As shown in <FIG>, the method in <FIG> further includes:
Step S31: First inertial measurement unit data collected by one or more sensors of the cleaning robot is acquired. The inertial measurement unit data includes: angular velocity and/or acceleration.

The step S31 in <FIG> is located between the steps S11 and S12, but the execution position of the step S31 is not limited to that shown in <FIG>. The step S31 may be performed before the step S11.

As shown in <FIG>, the step S12 in <FIG> includes:.

The plane information is used for indicating a plane in a world coordinate system. For example, in the world coordinates (x,y,z), a unique plane can be determined according to ax+by+cz+d=<NUM>. The plane information may be a set of specific values of a, b, c and d.

The mapping set includes a mapping relationship between second inertial measurement unit data and plane information. The mapping set is constructed according to the historical second internal measurement unit data of the cleaning robot during operation and the corresponding plane information. If there are more mapping relationships in the mapping set, the accuracy of the second inertial measurement unit data is higher, and the ground region recognized by using the mapping set is more accurate.

The inertial measurement unit data is continuous data with a preset duration. In the processing of searching the second inertial measurement unit data closest to the first inertial measurement unit data, a k-Nearest Neighbor algorithm is used.

To solve the problem of the large recognition error of the ground region due to the jitter or jolt of the cleaning robot during running, an embodiment of the present disclosure further provides a method for detecting small obstacles. With reference to <FIG> shows a method for detecting small obstacles according to an embodiment of the present invention.

As shown in <FIG>, the method in <FIG> further includes:
Step S31: First inertial measurement unit data acquired by one or more sensors of the cleaning robot is acquired.

The step S31 in <FIG> is located between the steps S11 and S12, but the execution position of the step S31 is not limited to that shown in <FIG>. In another embodiment, the step S31 may be performed before the step S11.

The first model is a preset model for recognizing plane information according to the inertial measurement unit data.

The first model may also be a sustainable learning model. Specifically, a learning model is trained by using a training data set to obtain the first model. Training input data and training output data in one-to-one correspondence in the training data set are inertial measurement unit data and plane information corresponding to ground data in a 3D point cloud acquired by the cleaning robot at a same historical moment, respectively. For example, the first model is a multilayer neural network, including an input layer, one or more hidden layers and an output layer. The number of neurons in each hidden layer is adjusted according to the dimensionality of the inertial measurement unit data, the training accuracy and other information.

In this embodiment, by a learning model capable of self-learning, the ground region can still be accurately recognized when the cleaning robot jitters or jolts during operation, and training samples are increased continuously by using historical data of the cleaning robot in different scenarios, so that the accuracy, stability and robustness of the learning model can be continuously improved.

An embodiment of the present disclosure further provides a method for detecting small obstacles. With reference to <FIG> shows a method for detecting small obstacles according to an exemplary embodiment. As shown in <FIG>, the method in <FIG> further includes the following step between the steps S14 and S15. Step S51: The ground projection point cloud is updated. Specifically, ground fitting is performed on the ground projection point cloud to remove noise data other than the ground.

The ground fitting may be performed in one of the following ways: a random sample consensus (RANSAC) algorithm and a nonlinear optimization algorithm.

An embodiment of the present disclosure further provides a method for detecting small obstacles. With reference to <FIG> shows a method for detecting small obstacles according to an exemplary embodiment. As shown in <FIG>, the method in <FIG> further includes the following step between the steps S15 and S16. Step S61: Noise elimination is performed on the ground projection point cloud by a morphological opening operation and a morphological closing operation. Thus, a connection area (which can be considered as noise) having an area smaller than the area of small obstacles to a large extent is removed, the recognition accuracy of small obstacles is improved, and the processing efficiency is improved.

An example embodiment of the present disclosure, which is not in accordance with the claimed invention, further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. This apparatus is applied to a cleaning robot. As shown in <FIG>, the apparatus includes:.

An embodiment, not forming part of the present invention,
further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. Based on <FIG>, this apparatus further includes:.

An embodiment of the present disclosure further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. Based on <FIG>, this apparatus further includes:.

An embodiment of the present disclosure further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. Based on <FIG>, this apparatus further includes:
a training module configured to train a learning model by using a training data set to obtain the first model.

Training input data and training output data in one-to-one correspondence in the training data set are inertial measurement unit data and plane information corresponding to ground data in a 3D point cloud acquired by the cleaning robot at a same historical moment, respectively.

An embodiment of the present disclosure further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. Based on <FIG>, this apparatus further includes:
a fitting processing module configured to perform fitting processing on the ground projection point cloud by any one of the following methods: a random sample consensus algorithm and a nonlinear optimization algorithm.

An embodiment of the present disclosure further provides an apparatus for detecting small obstacles. With reference to <FIG> is a structure diagram of an apparatus for detecting small obstacles according to an exemplary embodiment. Based on <FIG>, this apparatus further includes:
a morphological processing module configured to perform noise elimination processing on the ground projection point cloud by a morphological opening operation and a morphological closing operation.

An example embodiment of the present disclosure, which is not in accordance with the claimed invention, further provides a non-transitory computer-readable storage medium storing instructions that, when executed by a processor of a mobile terminal, enable the mobile terminal to execute a method for detecting small obstacles, the method including:.

Other embodiments of the present invention will be readily apparent to those skilled in the art upon considering this specification and the practices of inventions disclosed herein. The present application is intended to encompass any variations, uses or adaptations of the present invention, and these variations, uses or adaptations follow the general principle of the present invention and include the common knowledge or conventional technical means in the art that are not disclosed herein. This specification and the embodiments are merely exemplary, and the real scope of the present invention are defined by the following claims.

Claim 1:
A method for detecting obstacles, applied to a cleaning robot, comprising:
acquiring a first 3D point cloud corresponding to image data acquired by the cleaning robot (S11);
extracting, from the first 3D point cloud, a second 3D point cloud belonging to a ground region (S12);
extracting, from the second 3D point cloud, a third 3D point cloud having a height value in a set height range (S13), wherein the set height range is a range containing a positive range and/or a negative range;
calculating a ground projection point cloud of the third 3D point cloud (S14);
the method is characterized by:
additionally acquiring first inertial measurement unit data collected by one or more sensors of the cleaning robot (S31);
determining morphologically-connected regions in the ground projection point cloud (S15); and
using a morphologically-connected region having an area less than a preset value as a region where an obstacle is located (S16);
wherein the extracting, from the first 3D point cloud, a second 3D point cloud belonging to a ground region (S12) comprises:
inputting the first inertial measurement unit data into a first model to obtain plane information output by the first model (S121'); and
extracting, from the first 3D point cloud, a second 3D point cloud belonging to a ground region, according to the plane information (S122');
wherein the plane information is used for indicating a plane in a world coordinate system.