Patent Description:
The invention aims at minimizing human effort to scan regions of interest while also minimizing the time needed by the robot to acquire the data. <CIT> discloses a method in which the areas to be scanned are defined on the basis of external data. However, this data may be cumbersome to retrieve and/or a user may have difficulty finding the most relevant data.

<CIT> discloses a method for surveying an environment by a movable surveying instrument in its environment along a substantially random trajectory, with a progressional capturing of 2D images by at least one camera at the surveying instrument and applying a visual simultaneous location and mapping algorithm (VSLAM) or a visual inertial simultaneous location and mapping algorithm (VISLAM) with a progressional deriving of a sparse evolving point cloud of at least part of the environment, and a progressional deriving of a trajectory of movement.

It would be desirable to allow a user to have areas of interest in an environment scanned without having to carry the scanner through the environment. Also, it would be desirable that the user does not have to wait at the area of interest until the scan has been performed.

<CIT> discloses several systems and methods for automated facility surveillance using robots having sensors including laser scanners and being configured for autonomously moving through an environment. Autonomous exploration of areas of interest by robots is generally known in the art. Numerous aspects of this field are disclosed, e.g., in: <NPL>; <NPL>; <NPL>; and <NPL>.

However, these methods are focused on complete exploration of an area and assign every part of the scene the same importance. Consequently, a large amount of time is potentially spent on exploring areas which might be of no particular interest to a user.

Other approaches follow a "teach-and-repeat" scheme, for instance <NPL>. Other approaches include a human in the loop to select interest points (so-called waypoints) while leaving the robot to autonomously find a way to reach these points. For instance such an approach is disclosed by <NPL>. However, such a representation requires to know the waypoints in a global reference frame, hence, some way of establishing this reference frame is required. In a GPS-denied area, such as interiors of buildings, this requires a localization algorithm and a map to localize the robot in the environment.

It would be desirable to use an autonomous robot to carry out the scanning of the area of interest without the need of further user interaction after the definition of the areas of interest. It would also be desirable that the robot only scans the areas of interest.

It is therefore an object of the present invention to provide an improved method and system for generating 3D scan data of one or more areas of interest in an environment.

It is a further object to provide such a method and system that minimize the human effort, particularly that avoid the necessity for a human to move a scanner through the environment or to set-up a scanner at the one or more areas of interest.

It is a further object to provide such a method and system that minimize the time scanning the areas of interest, particularly to minimize the time for a mobile robot to identify user-defined areas of interest and to travel through the environment towards the areas of interest.

At least one of these objects is achieved by the method of claim <NUM>, the system of claim <NUM>, and/or the dependent claims of the present invention.

A first aspect of the present invention pertains to a method for generating three-dimensional scan data of one or more areas of interest in an environment. Said method comprises a user defining the one or more areas of interest using a mobile device in the environment, and a scanning device performing a scanning procedure at each defined area of interest to generate the scan data of the respective area of interest. Defining the areas of interest comprises, for each area of interest, generating identification data, which at least comprises generating image data of the respective area of interest. The scanning procedure at each defined area of interest is performed by a mobile robot comprising the scanning device and being configured for autonomously performing a scan of a surrounding area using the scanning device, the mobile robot having a SLAM functionality for simultaneous localization and mapping and being configured to autonomously move through the environment using the SLAM functionality. The identification data is provided to the mobile robot, and, in the course of each scanning procedure, the mobile robot navigates to the respective area of interest using the identification data, detects the respective area of interest using the identification data, and uses the scanning device to scan the respective area of interest to generate the three-dimensional scan data.

According to some embodiments of the method, generating the identification data comprises generating position data related to a determined position of the mobile device at the respective area of interest. The position of the mobile device is determined relative to the environment, e.g. relative to a local coordinate system of the environment, and/or relative to a global coordinate system, e.g. using GNSS data of a global navigation satellite system receiver of the mobile device. For instance, the determined position can be a position of the mobile device while capturing the image data.

According to some embodiments of the method, generating the identification data comprises generating pose data related to a pose of the mobile device while capturing the image data - for instance using IMU data of an inertial measuring unit of the mobile device - the pose comprising at least the attitude in three degrees-of-freedom.

According to some embodiments of the method, the mobile device tracks its path in the environment, and generating the identification data comprises generating path data related to the path of the mobile device. For instance, navigating to the respective area of interest may comprise using the path data, and/or the mobile robot may generate, based on the path information, a route for the mobile robot through the environment.

In one embodiment, tracking the path comprises using a SLAM functionality of the mobile device. For instance, for tracking the path the SLAM functionality of the mobile device may use at least one of IMU data of an inertial measuring unit of the mobile device, and image data continuously captured by at least one camera of the mobile device.

According to some embodiments of the method, the identification data is generated using environment data comprising at least one of image data, 2D data or 3D data of the environment. For instance, the environment data may be retrieved from an external data source, and/or may be used for determining a position of an area of interest based on the image data. Optionally, the image data may comprise depth information.

According to some embodiments of the method, the mobile robot has access to environment data comprising 3D data of the environment, wherein the mobile robot.

The 3D data of the environment in particular may have a lower resolution than the scan data of the areas of interest.

According to some embodiments of the method, the mobile device comprises a display, at least one camera and an image-capturing functionality for generating, upon a trigger by the user of the mobile device and using the at least one camera, the image data. Optionally, also position data and/or pose data may be generated upon the trigger.

According to some embodiments of the method, the identification data is generated and provided to the mobile robot directly after generating the image data.

According to some embodiments of the method, the mobile robot starts a scanning procedure upon receiving the identification data.

According to some embodiments of the method, the image data comprises depth information. For instance, the mobile device comprises at least one time-of-flight camera and/or a 3D camera arrangement. In one embodiment, the identification data is generated using environment data comprising 3D data of the environment, wherein the environment data is used for determining a position of an area of interest based on the depth information; in another embodiment, the mobile robot detects the respective area of interest based on the depth information, for instance wherein the mobile robot comprises at least one time-of-flight camera and/or a 3D camera arrangement.

A second aspect of the invention pertains to a system for generating three-dimensional scan data of one or more areas of interest in an environment, e.g. according to the method of the first aspect, the system comprising a mobile device and a mobile robot. The mobile device comprises a camera for capturing images of the one or more areas of interest and for generating image data, and the mobile robot has a SLAM functionality for simultaneous localization and mapping and a scanning device for performing a scan at the one or more areas of interest and generating the scan data of the one or more areas of interest. The system is configured to generate, using at least the image data, identification data for each of the one or more areas of interest, the identification data allowing identifying the respective area of interest, and to provide the identification data to the mobile robot. The mobile robot is configured to autonomously.

In some embodiments of the system, the scanning device comprises at least one laser scanner. In some embodiments, the scanning device comprises at least one structured-light scanner. In some embodiments, the scanning device comprises at least one time-of-flight camera.

In some embodiments of the system, the mobile robot is configured as a legged robot, comprising actuated legs for moving through the environment. In some embodiments, the mobile robot is configured as a wheeled robot, comprising actuated wheels for moving through the environment. In some embodiments, the mobile robot is configured as an unmanned aerial vehicle, e.g. a quadcopter, comprising actuated rotors for moving through the environment.

According to some embodiments of the system, the mobile device comprises a display, at least one camera and an image-capturing functionality for generating the image data upon a trigger by the user of the mobile device and using the at least one camera. In one embodiment, the image-capturing functionality is provided by a software application installed on the mobile device, wherein the display is configured as a touchscreen and the software application allows the user to mark an area in an image displayed on the display to define as an area of interest. In one embodiment, the mobile device comprises an inertial measuring unit, a compass and/or a GNSS receiver. In one embodiment, the at least one camera is configured as a time-of-flight camera and the image data comprises depth information. In one embodiment, the mobile device is configured for detecting a position of the mobile device while capturing the image data, and the system is configured to generate the identification data using position data related to the detected position. In one embodiment, the mobile device is configured for detecting a pose of the mobile device while capturing the image data, and the system is configured to generate the identification data using pose data related to the detected pose. In one embodiment, the mobile device is configured for tracking a path through the environment, e.g. using a SLAM functionality of the mobile device, IMU data of an inertial measuring unit of the mobile device, and/or image data continuously captured by the at least one camera, and the system is configured to generate the identification data using path data related to the path.

According to some embodiments of the system, the mobile robot is configured to receive environment data comprising 3D data of the environment, and the mobile robot is configured to autonomously move through the environment using the environment data and the SLAM functionality, to navigate to the areas of interest using the environment data and the determined positions, and/or to detect the areas of interest based on the image data and the 3D data, for instance wherein the image data comprises depth information.

According to some embodiments of the system, the mobile device comprises a SLAM functionality for simultaneous localization and mapping of the mobile device and is configured to track the path using the SLAM functionality. Optionally, for tracking the path the SLAM functionality uses IMU data of an inertial measuring unit of the mobile device, and/or image data continuously captured by at least one camera of the mobile device.

The invention in the following will be described in detail by referring to exemplary embodiments that are accompanied by figures, in which:.

<FIG> shows a layout of an apartment. The apartment is an example of an environment <NUM>, in which there are areas of interest that a person would like to have scanned. In the shown example, there are three areas of interest <NUM>, <NUM>, <NUM>. A first area of interest <NUM> is situated in a bedroom and comprises a wall including a window. A second area of interest <NUM> is situated in a bathroom and comprises appliances including a bathtub. A third area of interest <NUM> is situated in a combined kitchen and living room and comprises a built-in kitchen unit including a stove.

For instance, the window, the bathtub and the kitchen unit have recently been installed in the apartment, and an existing 3D model of the apartment needs to be updated with new 3D data of these areas. In this case, for instance, the person who would like to have the scans performed at the areas of interest may be a contractor or craftsman that have installed the applications or an owner of the apartment or an architect that have commissioned the installations. Alternatively, there is no 3D model of the environment <NUM>, and only 3D data of the areas of interest <NUM>, <NUM>, <NUM> may be needed.

Conventionally, the person who would like to have the scans performed would haul a scanner through the apartment and place it at each area of interest to perform the scans. Alternatively, the person could define the areas of interest and then have someone else perform the scanning.

In the approach suggested by the present application, the areas of interest are specifically defined by a user of a mobile device. Data comprising identification information that allows identifying the defined areas is made available to a mobile robot that will perform the scans.

<FIG> shows the user <NUM> of an exemplary embodiment of the mobile device <NUM> at the third area of interest <NUM> of the environment of <FIG>. The mobile device <NUM> has a camera <NUM> and is used by the user <NUM> to capture an image <NUM> of the area of interest <NUM>.

<FIG> shows the front side of the mobile device <NUM> of <FIG>. It comprises a display <NUM> and an image capturing functionality to capture images <NUM> of the areas of interest, thereby generating digital image data that may be provided to the mobile robot. The image may be captured upon receiving a trigger by the user. For instance, the trigger may comprise the user pushing a button of the mobile device or a digital button on the touch-sensitive display <NUM>. Also a position of the mobile device may be determined and position data may be generated upon receiving the trigger. The identification data to be provided to the mobile robot may comprise the image data and the position data or be generated based on the image data and the position data.

Optionally, the user may define an area <NUM> in the image (e.g. using the touch-sensitive display <NUM>) as the area of interest. Then, for instance, this information is included in the identification data. Alternatively, only the image data related to the user-defined area <NUM> in the image is included in generating the identification data.

The identification data needs to include data that allows the mobile robot to determine at least a rough position of each area of interest within the environment <NUM>. For instance, an absolute or relative position of the mobile device may be determined while capturing the image or a path to that position may be tracked. The identification data further needs to include data that allows the mobile robot to detect the area of interest at this rough position. In particular, this information may include the image data of the area of interest and/or pose data regarding a pose of the mobile device while capturing the image data. Alternatively, a precise position of the area of interest may be derived by comparing the image data and existing environment data.

<FIG> shows an exemplary path of the user of the mobile device through the environment <NUM>. The user captures a first image <NUM> of the first area of interest from a first position <NUM>, then moves along a path <NUM> to a second position <NUM> to capture an image <NUM> of the second area of interest and finally moves along the path <NUM> to a third position <NUM> to capture an image <NUM> of the third area of interest.

While the user moves along the path <NUM>, the mobile device may track this path and generate path information. For instance, tracking the path <NUM> may involve using one or more cameras and/or an inertial measuring unit (IMU) and a simultaneous localization and mapping (SLAM) functionality of the mobile device. Also, the mobile device may comprise a compass and/or a global navigation satellite system (GNSS) receiver that may be involved in tracking the path <NUM>. The path information may be part of the identification data or used to generate the identification data for an area of interest, particularly path information relating to the path <NUM> from the previous area of interest.

The identification data of each area of interest may be generated and sent to the mobile robot directly after capturing the respective image. Alternatively, generating and/or sending the identification data may require a further user input, e.g. on the mobile device.

<FIG> shows the scanning by the mobile robot <NUM> in the environment <NUM>. The mobile robot <NUM> has a SLAM functionality for simultaneous localization and mapping that allows the mobile robot to autonomously move through the environment <NUM>. The mobile robot uses the received identification data to autonomously navigate to the user-defined areas of interest.

In the shown example, having received the identification data of the three areas of interest, the mobile robot moves to a first scanning position <NUM> at the first area of interest and performs a first scan. Then, the mobile robot moves along a path <NUM> to a second scanning position <NUM> and to a third scanning position to perform a second and third scan.

The scanning positions are selected based on the received identification data. It is not necessary that the scanning position is the same as the position at which the image of the respective area of interest has been captured. Sometimes, it may be even necessary to use a different position for the scanning than for capturing the image.

By also using the information of the user's path <NUM>, the robot can quickly navigate between interest points, as the planning required basically consists only of local obstacle avoidance. Furthermore, because of the user's selections, the robot is aware of what is important and does not spend time on places in which the user is not interested. Hence, the time-efficiency of the robot is also increased and can approach that of a teach-and-repeat workflow (where the "planning" is entirely up to the operator).

Another possibility is that if an environment model, e.g. a CAD model of the environment, is available, the robot can try to localize itself with respect to this model and do the same operations as if the CAD model was a previously recorded scan.

<FIG> shows an exemplary embodiment of the mobile robot <NUM> at the third area of interest <NUM> of the environment of <FIG>. The mobile robot has a scanning device <NUM> to capture the 3D data of the area of interest <NUM>.

For moving through the environment, the mobile robot may use different kinds of locomotion, each having its own advantages and disadvantages depending on the kind of environment. For instance, as shown here, the mobile robot <NUM> may be embodied as a legged robot, e.g. comprising four actuated legs. Alternatively, the robot <NUM> may be configured as a wheeled robot, i.e. comprising actuated wheels (and/or tracks), or as an unmanned aerial vehicle (UAV), particularly a quadcopter comprising actuated rotors.

The scanning device <NUM> of the mobile robot may comprise any suitable scanning means, in particular at least one laser scanner, at least one structured-light scanner, and/or at least one time-of-flight camera.

<FIG> illustrates the generation, use and flow of data within an exemplary embodiment of a system according to the invention while performing an exemplary embodiment of a method according to the invention. The mobile device <NUM> captures image data <NUM> of the area of interest and optionally further data, such as position data <NUM> and pose data <NUM> related to a position and pose of the device <NUM> while capturing the image data <NUM>. The mobile device <NUM> may also generate path data <NUM> from tracking a path to the area of interest. Also, the image data <NUM> may comprise RGB and depth information.

This data <NUM>, <NUM>, <NUM>, <NUM> captured by the mobile device <NUM> is used to generate identification data <NUM> that will be provided to the mobile robot <NUM>. Generating the identification data <NUM> may be done on the mobile device <NUM> or on an external computing unit of the system. It may comprise using existing environment data <NUM> of the environment. This may comprise 2D, 3D or image data of the environment.

The mobile robot <NUM> receives the identification data <NUM>, identifies the area of interest and generates the scan data <NUM> of the area of interest. For facilitating identification of the area of interest, the mobile robot optionally may use existing environment data <NUM> of the environment.

<FIG> shows a flow chart illustrating an exemplary embodiment of a method <NUM> according to the invention. The approach of the described method <NUM> allows to efficiently operate and not spend time on areas that are not of particular interest. Also, it includes the advantage of a teach-and-repeat workflow which allows the robot to quickly navigate as it has a strong prior on the path it can take (i.e. the user's trajectory). The approach can be seen as a sort of a teach-and-repeat workflow with additional information added, namely the areas of interest defined by the user. Therefore, it can be placed between a fully autonomous exploration and a simple path following algorithm.

As shown here, the approach consists of two stages. In a first stage (definition stage <NUM>), a user, using a mobile device, takes images of areas of interest, e.g. those places that should be scanned thoroughly. The mobile device captures the images, thus generating <NUM> image data of the areas of interest. Optionally, the mobile device also captures other data, e.g. regarding a position or pose of the device while capturing the image. For instance, the device may record the user's motion by means of an odometry system (e.g. ARKit, ARCore) to determine a trajectory between two areas of interest. Based on the image data and the other data, identification data is generated <NUM> and provided <NUM> to the mobile robot.

In a second stage (scanning stage <NUM>), the mobile robot identifies the areas of interest based on the images taken by the user and references itself relative to the position of the mobile device when taking the image.

The scanning stage <NUM> comprises the robot using the identification data <NUM> to autonomously navigate <NUM> towards the respective area of interest and to detect <NUM> the respective area of interest. Optionally, in order to efficiently navigate between the areas of interest, the robot may use the user's trajectory as a basis for its own path planning. Then, the robot uses its scanning device to scan <NUM> the area of interest to generate the three-dimensional scan data.

During the first stage <NUM>, the user can use a large variety of lightweight devices (almost any modern smartphone/tablet), which allows to quickly go through the scene and select (define) the areas of interest. A software application ("app") may be installed on the mobile device that automatically provides the captured data (or identification data that is generated based on the captured data) to the mobile robot - either directly or via a server computer. The app may also automatically track the user's movement between two areas of interest. Optionally, the app may also receive a user input, e.g. on a touchscreen of the mobile device, to define the area of interest more precisely in the captured image.

Claim 1:
Method (<NUM>) for generating three-dimensional scan data (<NUM>) of one or more areas of interest (<NUM>, <NUM>, <NUM>) in an environment (<NUM>), the method comprising
- a user (<NUM>) defining (<NUM>) the one or more areas of interest (<NUM>, <NUM>, <NUM>) using a mobile device (<NUM>) in the environment (<NUM>); and
- a scanning device (<NUM>) performing a scanning procedure (<NUM>) at each defined area of interest (<NUM>, <NUM>, <NUM>) to generate the scan data (<NUM>) of the respective area of interest,
- defining (<NUM>) the areas of interest (<NUM>, <NUM>, <NUM>) comprises, for each area of interest,
generating (<NUM>) identification data (<NUM>), wherein generating the identification data at least comprises generating (<NUM>) image data (<NUM>) of the respective area of interest; and
- the scanning procedure (<NUM>) at each defined area of interest (<NUM>, <NUM>, <NUM>) is performed by a mobile robot (<NUM>) comprising the scanning device (<NUM>) and being configured for autonomously performing a scan of a surrounding area using the scanning device (<NUM>), the mobile robot (<NUM>) having a SLAM functionality for simultaneous localization and mapping and being configured to autonomously move through the environment (<NUM>) using the SLAM functionality,
wherein the identification data (<NUM>) is provided (<NUM>) to the mobile robot (<NUM>), and, in the course of each scanning procedure (<NUM>), the mobile robot (<NUM>)
- navigates (<NUM>) to the respective area of interest (<NUM>, <NUM>, <NUM>) using the identification data (<NUM>);
- detects (<NUM>) the respective area of interest (<NUM>, <NUM>, <NUM>) using the identification data (<NUM>); and
- uses the scanning device (<NUM>) to scan (<NUM>) the respective area of interest (<NUM>, <NUM>, <NUM>) to generate the three-dimensional scan data (<NUM>).