Patent Description:
The usefulness of a mobile construction robot on a construction site increases with its autonomy. It is thus of particular interest if the mobile construction robot is capable of inferring its position on a construction site in an autonomous, precise and robust manner.

Numerous efforts have been made to this end to date. One approach consists in using map data of the construction site, e. building information model (BIM) data in combination with, for example, a total station or the like. Nonetheless, such solutions lack flexibility and usability and often require manual interventions.

Even if BIM data are available for a specific construction site, a localization solely based on such BIM data and, for example, distance measurements, regularly fails due to interfering objects like persons, power tools, material depositions, or the like on the construction site.

Therefore, it is an object of the present invention to provide a robust and precise method for localizing a mobile construction robot on a construction site.

This is achieved in a number of aspects of the present invention, in which a first aspect is a method of localizing a mobile construction robot on a construction site, wherein environment data of the construction site is captured, wherein a pose of the mobile robot is inferred using the environment data, wherein the method comprises a step of semantic segmentation, wherein a semantic classifier having an updatable semantic model classifies the environment data into at least two semantic classes, wherein the semantic model is updated at least once by assigning points of environment data to one of the two semantic classes, if a distance of the points to a BIM model data surpass a threshold, and otherwise assign the points of environment data to the other one of the two semantic classes.

The pose may comprise a position and/or orientation of the mobile construction robot, in particular in reference to the construction site.

Particularly preferably the environment data comprises 3D environment data. The environment data may be obtained by distance measurement sensors, e. by a LIDAR scanner or a time-of-flight camera.

Thus, it is an idea of the invention to semantically segregate different parts of the captured environment data.

In particular the environment data may be classified in regard to the usefulness of the data for localizing the mobile construction site robot. Thus, noisy or even interfering data can be detected and discarded from the environment data. Hence, it permits a robust and precise localization based on a purposive selection of the environment data, in particular in the presence of, for example, clutter on the construction site.

The environment data may be captured by the mobile construction robot itself. All sensor equipment may thus be on-board of the mobile construction robot. Hence, the autonomy of the mobile construction robot may be increased.

In a preferred variant of the method the semantic model may be updated using the environment data. Thus, the environment data may be used for localizing the mobile construction robot and for improving the semantic classifier. This, in turn, may further augment the robustness and/or precision of the localization.

The semantic classifier classifies the environment data into a set of classes. The set of classes may contain a finite number of semantic classes.

Preferably, at least one of said semantic classes may correspond to a construction site background and at least one of said semantic classes may correspond to a construction site foreground.

The classes, in particular the background and foreground classes, may relate to the immutability of objects detected in the environment data, e. walls versus tables or moving animals; and / or, preferably, refer to the probability that the environment data is mapped in an available set of map data, e. "Background" may comprise, for example, environment data of walls, ceilings, floors, or the like. In particular, it may comprise elements that usually are represented in map data or, respectively, BIM data. "Foreground" may comprise other elements, in particular inferring objects like persons, furniture, equipment, or the like.

Preferably, the pose may be inferred using a subset of the environment data that is classified as construction site background. So, environment data may be stripped off data classified as construction site foreground. Unclassified data may also be stripped off. Subsequently, the localization may be executed based on the remaining environment data, in particular solely based on environment data classified as background. Of course, if finer classifications are available, the additional semantic information may also be taken into consideration. For example, if a plurality of background classes is available, e. separate classes for walls, ceilings, floors, or the like, this additional semantic information may facilitate a matching of the environment data with available BIM data.

The quality, in particular the rate of correct classifications, may be improved if the semantic segmentation of the environment data is refined at least once using at least one additional data set, preferably an image data set. For this, superpixels may be generated, for example by using a simple linear iterative clustering (SLIC)-algorithm.

Preferably, the pose may be inferred using map data of the construction site, e. In particular, the environment data or a selection thereof may be matched to the map data in order to infer the position and / or the orientation of the mobile construction robot.

The semantic classifier may be trainable. Thus, in order to adapt the semantic classifications to different domains, the semantic model may be updated multiple times, preferably regularly. The update may use current or previous semantic segmentation data and / or current or previous environment data or a selection thereof.

To counteract catastrophic forgetting the semantic model may be updated with a mix of current and previously gathered semantic segmentation data.

Whereas traditional methods of semantic segmentation allow for the introduction of new classes across different semantic segmentation tasks, the invention focuses on classifying environment data, in particular according to its usefulness for localization. Thus, it is preferred that the set of said semantic classes is fixed for all semantic segmentations. In this sense, the invention tackles the problem of domain adaptation, in which the goal is to tune the semantic classifier towards better generalization to a new domain (target) which possibly has a large semantic gap from the one of which the semantic classifier is trained (source).

Another aspect of the invention relates to a construction robot system comprising a mobile construction robot, configured for working on a construction site, for example for drilling, chiselling, grinding, plastering and/or painting, and a control unit, wherein the control unit is configured to localize the mobile construction robot using the method according to the invention.

The mobile construction robot may comprise a mobile base for moving at least on one of the construction site's floor, its ceiling, or one of its walls. The mobile base may be unidirectional, multidirectional or omnidirectional. It may comprise wheels and/or tracked wheels. Alternatively, it may also be or comprise a legged mobile base. Additionally, or alternatively, the mobile construction robot, in particular the mobile base, may be an aerial vehicle, in particular an unmanned aerial vehicle, also known as a "construction drone".

It may be configured to at least one of grasping or moving an object, drilling, chiselling, grinding, plastering, or painting.

The mobile construction robot may comprise the control unit. Additionally, or in the alternative, the control unit may be, or at least partly, external to the mobile construction robot. The control unit may be, at least partly, part of a cloud-based computing system. The control unit may be configured to control a plurality of mobile construction robots. In particular, the same semantic classifier or at least its semantic model may be used for a plurality of mobile construction robots. Thus, a training of the semantic classifier may be beneficial to several ones of the plurality of mobile construction robots.

In a preferred embodiment the mobile construction robot comprises a 3D sensor, e. a LIDAR scanner or a stereoscopic camera system, and at least one camera. Thus, the 3D sensor may provide 3D environment data as environment data, which is enriched by image data, e. RGB-image data, provided by the at least one camera.

Another aspect of the invention relates to a computer program product including a storage readable by a control unit of a construction robot system, the storage carrying instructions which, when executed by the control unit, cause the control unit to execute the method according to the invention.

The invention will be described further, by way of example, with reference to the accompanying drawings which illustrate preferred variants thereof, it being understood that the following description is illustrative of and not limitative of the scope of the invention. The features shown there are not necessarily to be understood to scale and are presented in such a way that the special features of the invention are clearly visible. The various features may be realized individually or in combination in any desired way in variants of the invention.

As far as possible, same reference signs are used for functionally equivalent elements within the description and in the figures.

<FIG> schematically shows an overview of the method <NUM> according to the invention, which will be described in more detail in the following section.

The method <NUM> and, hence, the construction robot system applying the method <NUM> according to the invention combine the concepts of continual learning and of self-supervision.

Environment data <NUM> is captured by a LIDAR scanner. By matching the environment data <NUM> with a BIM model <NUM>, representing, e. a floorplan of the construction site, an initial pose estimate and an initial set of pseudo-labels <NUM> are generated during a localization step.

A semantic classifier <NUM> comprising a continually learned semantic model serves as an input filter to the localisation, semantically segmenting a scene <NUM> observed by a camera system <NUM>, comprising one or more cameras, in this embodiment one camera monitoring the rear left, one camera monitoring the center, and one camera monitoring the rear right, into segments, e. superpixels, classified as construction site foreground or as construction site background. This refined segmentation is fed forward into the localization.

The set of semantic classes are not arbitrary class labels, but purposively selected by the observable affordance that some parts of a current scene are mapped (construction site background) and some are not (construction site foreground). Therefore, pseudo-labels are created based on the localisation in reference to the BIM model <NUM> to train the semantic classifier <NUM>, whereas the segmentation into construction site foreground and construction site background informs the localisation.

This creates a feedback loop that can yield improvements for both parts of the procedure, in particular for the quality of the semantic classification and for the robustness and the precision of the localisation, i. the pose detection. On average, experiments showed improvements of semantic segmentation by <NUM>% and median localisation error by <NUM>% during deployment.

Thus, a life-long self-supervised learning of semantic classification of scenes <NUM> is achieved. The invention tackles several topics, of which the following topics will be discussed in more detail:.

The mobile construction robot is localized based on aligning the environment data <NUM> from the LIDAR scans with the given floorplan in the form of the BIM model data <NUM>.

Given a building model mesh M representing the BIM model <NUM>, a point-cloud P representing the environment data <NUM>, and an initial alignment <MAT> subsequent mobile construction robot poses may be found as <MAT> where ICP is a point-to-plane ICP according to <NPL>. The ICP filters out points of large distance and other criteria.

The point-to-plane ICP is preferably run with three nearest neighbors and initialised on the previously solved pose. Furthermore, multiple filters are applied to the input. These filters may also be applied after the semantic filtering:.

Furthermore, for very cluttered 3D environments or, respectively, construction sites, additional filters may be enforced in order to localise without segmentation: For example, only four degrees of freedom, in particular x, y, z, yaw, may be inferred. Normal directions may be based on <NUM> nearest neighbours. Points may only be associated to the BIM model data <NUM> if the angle between the normals is below <NUM> rad.

To further divide the point-cloud P into construction site foreground and construction site background points, image data from the camera system <NUM> is used as additional information.

After having the image data from the camera system <NUM> semantically classified, a static point-cloud Pstatic is extracted from the point-cloud P as those points p ∈ P whose reprojected pixel in the image frame is classified as construction site background.

Accordingly, localisation, i. pose detection, is based on <MAT>.

<FIG> show different stages during the generation of pseudo-labels and, hence, during the semantic classification.

For each camera of the camera system <NUM> semantic classifications ("pseudo-labels") are generated by labelling the captured point-cloud of the environment data <NUM> leveraging a current pose estimate as well as the BIM model data <NUM> of the construction site.

As an example, <FIG> schematically shows a mesh according to BIM model data <NUM> and a point-cloud corresponding to environment data <NUM> from a LIDAR scan as an overlay image.

In this variant of the method <NUM>, the data is classified into three different classes: construction site foreground, construction site background and unknown.

The classes are created in two steps. First, for each point of the localised environment data <NUM>, we calculate the distance to the closest plane of the mesh according to the BIM model data <NUM> using a fast intersection and distance computation according to <NPL>.

If the distance surpasses a given threshold δ, the point is assigned the construction site foreground class, otherwise it is assigned the construction site background class. Empirically, the distance threshold is set to δ = <NUM>. <FIG> schematically shows the result of this preliminary classification as an image of the camera system <NUM> overlaid with a projection of the classified point-cloud. Pixels in black along the LIDAR scan lines correspond to the class unknown, pixel in light grey along the LIDAR scan lines the correspond to construction site foreground and the remaining pixels along the LIDAR scan lines correspond to construction site background. Thus, according to <FIG> an area comprising a worker and some equipment has been classified as foreground on the right half of <FIG>.

In the second stage, the projection is refined using superpixels created with a simple linear iterative clustering (SLIC-) algorithm, which utilizes k-means clustering. <FIG> schematically shows an example of a resulting oversegmented image with a mesh according to the superpixels. In particular, this mesh may correspond to the outlines of the superpixels.

In particular, the image is oversegmented into a superpixel set S. A superpixel s ∈ S is then assigned a class according to a majority voting of the contained projected classes. The segmentation is further improved by discarding superpixels whose depth variance surpasses a given threshold.

Superpixels with a depth variance that surpasses <NUM> are discarded. Images are smoothed with a Gaussian kernel (σ = <NUM>). The images are oversegmented into approximately <NUM> superpixels with SLIC parameter compactness = <NUM>.

Instead of the SLlC-algorithm, other superpixel algorithms may be employed alternatively, e. the SCALP-algorithm according to<NPL>.

Furthermore, depending on the type of environment the standard deviation threshold may be increased to <NUM> or more, in particular in case if large amounts of clutter are to be expected.

<FIG> schematically shows an example of a resulting semantic classification, wherein black pixels are classified by the majority voting as unknown, pixels in dark grey are classified as construction site foreground, and pixels in light grey are classified as construction site background.

It has been found out that the quality of classification or, respectively, of the pseudo-labels, is sufficient to be used as a training signal for retraining the semantic model of the semantic classifier <NUM>. A particular advantage of the method <NUM> according to the invention is that the classification can be generated on-the-fly without requiring any external supervision.

To improve the semantic classification, the semantic classifier <NUM> having a semantic model in the form of a neural network architecture is trained on different data sources.

To cater to the goal of online learning, a lightweight network architecture based on Fast-SCNN according to <NPL>, is used for the semantic classifier <NUM>.

The semantic model is pre-trained with labelled data <NUM> (see <FIG>), e. the NYUDepth v2 dataset (<NPL>), which contains <NUM> images extracted from video sequences of indoor scenes, each with per-pixel semantic annotations. Of this data set the classes "wall", "ceiling", and "floor" were mapped to the construction site background class, whereas the remaining classes were mapped to the construction site foreground class.

A pre-training step <NUM> (see <FIG>) is performed as an initial step to allow the semantic model to acquire prior knowledge that can then be leveraged as an inductive bias to perform the same semantic segmentation task on subsequently captured data.

The semantic model is then fine-tuned with self-supervision through the pseudo-labels generated on the real-world scene in which the mobile construction robot is deployed. Thus, the semantic model is continuously updated.

With reference to the nomenclature often used in continual learning, each new environment is named a "task", and "task boundaries" are assumed to be known. Every time the mobile construction robot is moved to a new environment, the same scheme as above is applied, i.e., the semantic model trained on the previous environments is provided as new training data the classes or, respectively, the pseudo-labels generated from the current environment.

In order to achieve this domain adaptation and allow the semantic model to improve the semantic segmentation accuracy on the current environment in which it is deployed without forgetting information learned on previous tasks, a method is adopted based on memory replay buffers.

When adapting to a new environment, each training batch is filled with frames collected in the current environment, along with a small fraction of images collected in the previous environments, so that the semantic model is, in particular continuously, updated with a mix of current and previously gathered semantic segmentation data.

Hence, in each training step, the semantic model is to be jointly optimized over data from current and previous environments. However, storing all observations or sematic segmentation data from past environments in memory would require a large memory space.

To solve this problem, according to the invention a memory buffer for each of the previous environments only contains a random subset of all image and pseudo-label or label attributions.

Training batches are then filled from the memory buffers of previous environments alongside with self-supervised labels of the current environment.

According to a preferred variant of the method <NUM> according to the invention, a batch of size <NUM> may be used. The semantic model may be trained for, e. up to <NUM> epochs. Preferably, early stopping with a patience of <NUM> epochs based on a validation loss may be used. In different variants the replay buffers may be filled according to a ratio between the target and the NYU data of between <NUM>:<NUM> and <NUM>:<NUM>, e. <NUM>:<NUM>, <NUM>:<NUM>, <NUM>:<NUM>, <NUM>:<NUM>, <NUM>:<NUM>, or <NUM>:<NUM>. The replay buffers may also be filled according to a replay fraction of between <NUM>% and <NUM>%, e. <NUM>% or <NUM>%. Preferably a replay fraction of <NUM>% is used.

The semantic model based on Fast-SCNN preferably has a total of <NUM> to <NUM> × <NUM><NUM>, e. approximately <NUM>,<NUM> × <NUM><NUM>, trainable parameters.

Group normalization may be used in all layers, which, in general, has been found to perform better in regard to the required transfer-learning tasks than alternative normalization methods like batch normalization.

In variants of the method <NUM> different strategies for memory replay may be used. A first strategy, which is the one adopted in the variant of the method <NUM> as described before, on each source (e. NYU data set) -to-target (e. construction site) transfer, the replay buffer may be refilled with a fraction of randomly selected samples from the source dataset(s), here the NYU dataset. Then, the training batches are filled from the replay buffer and the target dataset according to their relative sizes.

According to an alternative, second strategy, the replay buffer may be filled with the full source dataset, whereas training batches are filled with a pre-defined target-to-source ratio. For instance, a ratio Construction site to NYU of <NUM>:<NUM> with a batch size of <NUM> indicates that batches on average contain <NUM> images from the construction site and two images from NYU.

Larger target-to-source ratios, or smaller replay fractions, have shown to achieve higher performance on the target domain, but cause the amount of information retained from the source domain to drop. At the same time, the segmentation quality on the pseudo-labels of the target dataset follows an inverse trend, generally increasing for smaller amounts of replay. This highlights the trade-off between the accuracy on the target and on the source domain.

Further variants of the method <NUM> according to the invention may make use of alternative continual learning methods, for example regularization methods. One example may be a distillation approach, in which a weighted regularization term Ld is added to a cross entropy loss Lce to encourage the semantic model to retain knowledge from previous tasks. This distillation may be either applied to output logits produced by the semantic model (output distillation) or on intermediate features extracted from the model architecture before a final classification module (feature distillation).

A further alternative may be an elastic weight consolidation (EWC), in which deviations of network parameters across tasks are penalized.

Overall, replay buffers have shown to be very effective in minimizing the amount of forgetting on the NYU dataset, and generally allow attaining a good trade-off with the segmentation quality on the pseudo-labels from the target domain.

<FIG> shows a construction robot system <NUM> on a construction site, the construction robot system <NUM> comprising a mobile construction robot <NUM> and a control unit <NUM>, which is schematically represented in <FIG>.

In this embodiment, the control unit <NUM> is arranged inside the mobile construction robot <NUM>. It comprises a computing unit <NUM> and a computer program product <NUM> including a storage readable by the computing unit <NUM>. The computing unit <NUM> comprises a neuronal network unit configured as semantic classifier <NUM> as described in reference to <FIG>. The storage carries instructions which, when executed by the computing unit <NUM>, cause the computing unit <NUM> to execute the method according to the invention and as described previously.

Furthermore, the mobile construction robot <NUM> comprises a robotic arm <NUM>. It comprises an end effector, on which a power tool <NUM> is detachably mounted. The power tool <NUM> may be a drilling machine. It may comprise a vacuum cleaning unit for an automatic removal of dust. The robot arm <NUM> and/or the power tool <NUM> may also comprise a vibration damping unit.

The control unit <NUM> is configured to control the robotic arm <NUM>. It may comprise further modules, for example a communication module, in particular for wireless communication, e. with an external cloud computing system (not shown in <FIG>), a display unit, or the like.

Furthermore, the control unit <NUM> is configured to control a mobile base <NUM> of the mobile construction robot <NUM>. In this embodiment, the mobile base <NUM> is a wheeled ground vehicle, on which other parts of the mobile construction robot <NUM> are mounted.

The mobile construction robot <NUM> may be configured for drilling floors, walls and / or ceilings. For this, the robotic arm <NUM> may, preferably, be capable of moving according to at least <NUM> degrees of freedom.

The mobile construction robot <NUM> comprises a plurality of sensors and localization support elements. In particular, it comprises a camera system <NUM> comprising three 2D-cameras. It further comprises a LIDAR scanner <NUM>. Both elements <NUM> and <NUM> are aligned to each other and connected to the control unit <NUM>. Furthermore, the cameras of the camera system <NUM> are time-synchronized with the LIDAR scans from the LIDAR scanner <NUM>.

The mobile construction robot <NUM> may further comprise other pose-aware sensors or elements, e. an inertial measurement unit (not shown in <FIG>).

Claim 1:
Method (<NUM>) of localizing a mobile construction robot (<NUM>) on a construction site (<NUM>), wherein environment data (<NUM>) of the construction site is captured, wherein a pose of the mobile robot (<NUM>) is inferred using the environment data (<NUM>), wherein the method (<NUM>) comprises a step of semantic segmentation, wherein a semantic classifier (<NUM>) having an updatable semantic model classifies the environment data (<NUM>) into at least two semantic classes, wherein the semantic model is updated at least once by assigning points of environment data to one of the two semantic classes, if a distance of the points to a BIM model data surpass a threshold, and otherwise assign the points of environment data to the other one of the two semantic classes.