Patent Description:
Modern manufacturing plants are increasingly automated or digitalized. The term "Industry <NUM>" has come to refer to the "fourth industrial revolution" and, in particular, to the digital transformation of manufacturing and production processes. In many conceptions of Industry <NUM>, humans and robots are sharing the same workplace and are operating side by side, in what might be referred to as hybrid workplaces.

A main goal of new industrial technologies is increasing productivity using automated machines that operate <NUM> hours a day, seven days a week. To maximize productivity requires optimizing the "uptime" of these machines, i.e., maximizing the amount of time in which these machines are operating. This is especially important in high-volume manufacturing processes where the fabrication machinery is very expensive, such as semiconductor fabrication. Each production stop in semiconductor production reduces the output and the performance of the whole facility, and reduces the return on the very large investments needed to build the production line.

Therefore, safe collaboration between automation systems (e.g., robots) and humans becomes more and more important. In various collaborative environments, humans control the progress of every operation performed by robots, and humans adjust robots parameters to enhance their performance. Typically, however, this collaboration between robots and human is free from bidirectional interaction, with the collaboration instead being limited to humans acting on the machines.

While robots are normally equipped with various sensors, conventional robots have no "brains," and use sensor data directly, for automated decision making. In particular, this sensor data may be used to trigger alarms and/or stop operation of the machine, in response to sensor data indicating, for example, that an unknown object has entered into a protected region.

Current state-of-the art solutions, however, only detect objects in 2D data, using, for example, laser or infrared-based safety curtains. These curtains may be scanned, in some cases, to provide something that resembles three-dimensional (3D) coverage.

The machines perform different predefined tasks in a loop and need continuous guidance from humans. This means that the human operator is the only part of the manufacturing system that can proactively react when something unusual happens, to eliminate or reduce machine stoppage time. The machine part of the manufacturing system, in contrast, can conventionally only react, based on direct sensor data, performing reactive mitigation strategies such as reducing operation speed or stopping operator altogether, e.g., if a predefined threshold is reached or if a danger factor is otherwise detected from the sensor data.

Literature <NPL> discloses a system presented which relies on the fusion of data coming from a Time of Flight (ToF) sensor and a <NUM> radar sensor. The data are analyzed and evaluated using deep learning (DL) algorithms. The purpose is to detect humans and track their movements in the observed area. The resulted perception system can be installed somewhere in a room or on a moving system.

Literature <NPL> discloses a multi-sensor system for accurate and power-efficient dynamic car-driver hand-gesture recognition, using a short-range radar, a color camera, and a depth camera, which together make the system robust against variable lighting conditions. The procedure jointly calibrates the radar and depth sensors. It is employed a convolutional deep neural network to fuse data from multiple sensors and to classify the gestures.

Literature <NPL> discloses an open hardware and software platform to efficiently fuse heterogeneous sensor data in an automotive/robotic context. The framework presented in this paper provides researchers a base platform in order to develop and evaluate sensor fusion strategies. In contrast to similar approaches, this framework exploits in particular the raw radar data and enables the fusion at low-level. The proposed system utilizes low-level data from radar sensors as well as indirect (e.g. 3D imaging) and direct (e.g. LIDAR) Time-of-Flight (ToF) sensors. After a configurable amount of preprocessing at sensor-level, the sensor data is transferred to a centralized platform and aligned temporally and spatially.

Patent Literature <CIT> tracking a current and/or previous position, velocity, acceleration, and/or heading of an object using sensor data may comprise determining whether to associate a current object detection generated from recently received (e.g., current) sensor data with a previous object detection generated from formerly received sensor data.

Improvements in these manufacturing systems are needed.

There may be a demand for providing an improved system for a method and an apparatus. The present technology relates to a method of detecting and tracking human activities in the vicinity of a robot and an apparatus for detecting and tracking human activities in the vicinity of a robot as set out in the appended set of claims.

<FIG> illustrates principles of the application of machine learning to the context of human-robot interaction.

As noted above, smooth collaboration between humans and machines in a manufacturing environment can be difficult. One problem in developing smooth collaboration results from the fact that state-of-the-art systems often rely on automated responses based on two-dimensional (2D) data, such as data obtained from 2D laser-based line scanners, despite that the human-machine collaboration is taking place in a three-dimensional (3D) room. For improved collaboration, it is necessary to observe this room in real time, during the operation and collaboration process, with appropriate sensors and an appropriate control solution.

For 3D room protection, the system needs 3D sensor information. This may mean, for example, the use of at least a three-laser line scanner for horizontal and vertical directions. In many cases, more scanners are needed to have a full and safe 3D environment observation around the machine/robot area. If any one of these scanners detects an obstacle, the system will immediately stop operation, or reduce the machine's movement speed.

Other challenges in human-machine collaboration result from interactions between humans and robots and arise from a lack of intelligence in the machine side of the collaboration while performing a shared task. For example, when a human works nearby and enters the comfort zone of a collaborative robot, the robot will automatically stop. In many cases, the machine's operation needs to be relaunched by a manual action by the human. This causes a delay or stop of the production line and causes financial loss for the company.

Two key requirements for automated systems are in tension with one another. First, when automated systems have to work in the same area or in direct collaboration with humans, it is necessary to have a system that is fully compliant with safety regulations concerning man/machine collaboration. However, production lines need to run with a high uptime to be productive. Interaction between machine and human cannot be synchronized and produce unexpected machine stops. This is a general contradiction and is valid for all kind of interaction between machine and humans in industry-, service-, home- and health applications.

A technical problem for existing automated manufacturing systems is that the data coming from a laser scanner, for example, cannot readily be transformed into real 3D information characterizing the type, movement direction, and speed of a variety of different of objects or obstacles, such as humans, other autonomous guided vehicles (AGVs), transport systems, tables, etc. This means the systems will stop or reduce their operating speed in every case of obstacle detection, even when the intrusion is small and/or short. This leads to many small stops and reduces the productivity of the whole automation system. Another problem with existing systems is the high cost of the scanner system solutions. These problems prevent the increase of collaborative automation solutions for industry and home applications.

Some industries find alternative solutions by using advanced 3D red-green-blue (RGB) cameras for detection and classification purposes. A deep interpretation of the environment enabled by the use of such cameras has been demonstrated, compared to using passive 2D sensors. Although the usage of 3D RGB cameras provides a better perception of the surrounding area, these cameras remain very expensive and have limitations, especially regarding the special coverage of the entire environment.

The techniques, apparatuses, and systems described herein address these problems by use of a fusion system composed of multiple different sensors. The combination of data coming from the sensors can be used to obtain a much better perception of the surrounding area.

In particular, the combination of the data, i.e., the "fused" sensor data, helps to understand the surrounding mechanisms that present an ambiguity before. Two or more sensors processing together in a harmony not only cut with the previous passive use of the data but also extend the perception coverage on the temporal and special domains. This approach also increases the global resolution and improves measurements that, when used alone, might be poor representations of the system's environment.

Additionally, artificial intelligent (AI) applications may be used to filter the sensor data, thus facilitating efficient processing of large quantities of data that may include a lot of useless information. The term AI as used here is meant to refer to machine-learning (ML) techniques used to manipulate the data received from the sensors and to put that data to use for various purposes. More particularly, these ML techniques include deep learning algorithms, e.g., as implemented using adaptive neural networks (ANNs) or, more specifically, convolutional neural networks (CNNs) or recursive neural networks (RNNs) such as long short-term memory (LSTM) models. The output from these ML algorithms may include signals and alarms dedicated to monitor a shared collaborative area between humans and robots. Such a system can be installed in a fixed place and triggered each time a strange situation occurs, such as a production stoppage, unusual traffic, unexpected accident in the production line, unusual human behavior, etc. In this case the signal is mainly sent to operating robot for guidance and for correction of a further misunderstanding of a certain situation. The basic system mechanism for an application of machine-learning in the context of automated manufacturing is illustrated in <FIG>.

An automated manufacturing system using fused sensor data according to the techniques described herein may rely, in some embodiments, on combined information from one time-of-flight (ToF) sensor and one radar sensor. ToF sensors, alternatively referred to as ToF cameras, are imaging systems that employ time-of-flight measurement techniques to determine a distance between the sensor and an imaged subject, e.g., for each of several points of the image. Devices may utilize photonic mixer devices (PMDs). In these devices, an outgoing light source is modulated, e.g., with a radio-frequency (RF) carrier, and the relative phase shift of the light reflected from the subject is determined, to determine the round-trip distance traveled by the modulated light. Radar systems, of course, transmit an RF signal, typically pulsed or modulated, and measure the round-trip travel time from the reflected radar system. Doppler-based processing techniques may be used to determine a relative speed of the object from which the radar signal is reflected, in a direction towards or away from the radar sensor.

Thus, the ToF sensor gives information about the shape and the location of an imaged human body or other moving obstacle, yielding two-dimensional greyscale and depth images, while the radar gives information about the x, y, z and velocity of different points of the detected human, with, typically, far less resolution between the different points. Samples of ToF sensor data and radar sensor data obtained in various scenarios may be collected and used to train a neural network to detect peoples' presences in a room as well as their directions of movement and their speeds. In operation, the trained neural network can be used in real time, to detect and react to expected and unexpected scenarios, e.g., by triggering alarms or other automated actions in response to detecting unusual scenarios or known dangerous scenarios.

The training dataset is composed of frames obtained by combining data from both sensors. For example, each frame may be composed of three fused layers, comprising a greyscale image obtained from the ToF sensor, a depth image obtained from the ToF sensor, and a radar image or radar data points obtained from the radar sensor. The system may be tested in real time to track human behavior in an indoor environment.

Using the fused dataset described above as input for training the neural network, safety hazards and other robot-human conflicts can be avoided, by actively changing the behavior of the automated system based on predictions provided by the trained neural network's outputs. Proper predictive and active system behavior can be defined by the following behaviors:.

To achieve this behavior, the automation system needs sensors and accompanying systems to detect obstacles by type, movement direction, and speed, in a 3D space around the work area. The sensor information must be available in real time with the exact constraints on the timeliness of the information depending on tool operation speed, the number of humans in the work area, and the layout of the automated system.

This can be done with an intelligent combination of data from both of the sensor types described above, i.e., the ToF sensor and a radar sensor. As was described above, the sensor data is fused in a way that the information from both sensor can be used to train a neuronal network to classify obstacles in front of the sensor view. Additional sensor information can be used to detect velocity and movement direction of classified objects as well. It is possible to combine a number of double sensors for a view that fully surrounds an automated system or a workspace, to protect up to <NUM> degree sensing. Processing of the fused sensor data can thus provide a high-resolution 3D perception of the environment with a wide spatial and temporal coverage, e.g., up to <NUM> degrees of coverage, at distances up to and greater than <NUM> meters, at frame rates up to and exceeding <NUM> frames per second (fps).

The fused sensor data will be delivered to the system for active reaction, to prevent conflict situations or to control temporary work process adaption.

A system based on sensor fusion as described herein can be either implemented in a fixed place on top of a hybrid workplace or on mobile robots. For example, in the semiconductor industry or in the automotive industry, there is a close interaction between humans and robots in different phases of the whole manufacturing process. Six-degree-of-freedom (6DOF) robots are widely used and can present a danger for humans. This is because these robots sometimes apply large forces to fulfill their tasks, regardless of the external conditions. Therefore, safety plays an important role in a shared workplace, i.e., a workplace in which both humans and robots are working. While in many industries and applications robots are equipped with sensor-based systems using laser scanners, these devices are not the best choice to establish an on-going active communication between both humans and robots.

Ideally, a robot system would have senses and a brain that operate very much like that of a human. In this case, the robot could communicate with the human and even protect himself from him. The systems described herein mimic this human-like brain, in that the "senses" correspond to the fused sensor data provided by the sensors while the "brain" is the additional intelligence added on top, to process the fused sensor data and to provide the robot with the ability to recognize and classify the objects that surround his environment. The result is a system like a safety eye that might guide the robot's movements (loading/unloading, assembly, pick & place, etc.) or help an AGV to navigate smoothly in a complex environment without causing accidents. The robot is able to recognize and locate objects and humans and track their movements.

In areas and applications where the safety requirements are less strong, e.g., because the power of the AGV or robotics systems is much less, in comparison to heavy industrial applications, sensor fusion can be applied using relatively cheap sensors including edge computing solutions - for example for service robots in medical or home/outpatient care. These robots can also be found in shops for cleaning or making inventory jobs or at home for entertainment or for taking care of kids, for example. Again, these systems might be applied to a robot operating in a fixed workspace, or to a mobile AGV, in various embodiments. There are thus various scenarios in which the inventive system described herein can be used, e.g., for industrial robots and semiconductor fab automation. The systems can be used in mobile or stationary robotics. In mobile robots, the problems are similar to those in autonomous cars. In autonomous environments, robots and humans are sharing the same working area. In these applications, the sensors and supporting systems may be attached to the autonomous moving vehicle. In other implementations, the sensors and supporting systems can be placed so as to obtain field of view that includes a full field of view of the working area of the stationary or limited linear motion robotics system. In either case, the system can detect unexpected obstacles and any humans. This principle can be used in many different applications in the same way.

For example, the systems described herein can be placed in any environment like in underground stations and maybe entrance of factories to detect and count humans and also we can extract their behavior in the environment. The systems can be placed in cars to detect humans also. Indeed, using radar, it is possible for the systems to detect vital signals of the humans and check whether they are sleeping while driving, or possible injured or unwell, etc. The systems can be used in service robotics to detect the owner of the systems or another assigned person to give the robot an ability to help that person. Using dynamic background detection and classification techniques, e.g., based on neural network technologies, the system can differentiate between allowed obstacles (including any movement of those obstacles) and unexpected obstacles.

An example implementation of the concepts described herein is a ToF/Radar fusion system for human detection and tracking. In this system, preprocessed data from both sensors is fused and used to train a model used to detect people and recognize there behaviors. The approach may be divided into two phases, a training phase and a testing/use phase.

During the training phase, depth and amplitude (grayscale) images are received from the ToF sensor(s). Contemporaneously, raw radar signals are received, e.g., from each of four receiving antennas. The angle of arrival of each signal is determined, and then the 3D coordinates (x,y,z) of each detected target are calculated (the number of targets depends on the number of used chirps). The doppler effect is also used to calculate the velocity of those targets, i.e., by detecting a frequency shift in the received signals, relative to the transmitted radar signal. After calibration of data coming from both sensors, it is possible to mask the targets on radar with the ToF image and construct a new image called radar image. This image includes the same objects, with shapes like those detected in the ToF image, but with additional information about the target velocity.

In the end, there are three channels, depth, grayscale and radar, fused in one image. Each of these <NUM>-channel images corresponds to one sample in the training dataset for a neural network. These images can be collected for a variety of operating conditions for the automated system/robot, with a variety of human and other articles in the environment. The so-created samples and the considered neural network architecture for human are presented in <FIG>.

An optimized training set of data could be between might be composed of more than <NUM> samples, for example, structured as follows: <NUM> % for training, <NUM>% for validation and <NUM>% for testing. The dataset result is then used to train a neural network for human detection. Since a typical dataset may be limited in terms of number of samples, the training for a particular environment may be done with a pre-trained network. For example, one might choose to work with a CNN-based object detection network. Experiments show that this may be trained with sample sets like those described above to achieve performance with <NUM>% mean average precision and <NUM> detection frame rate for <NUM>% intersection over union (tested on a standard GPU).

After training, a person model is an output. This person model may then be used for real time detection of running persons. Once detected, the system may define a region of interest around the person, e.g., a "bounding box. " From the first channel, of each three-channel frame in which the person may be detected, the range at which the person is standing or moving may be detected. From the second channel, the optical flow of moving points may be detected and then the general movement of the person may be detected. This general movement may be represented by an arrow on a displayed image of the system's output, for example. Finally, the speed of movement of the detected person (v) may be read from the third channel. Data from successive frames may be filtered, in some embodiments, to smooth the results and suppress noise.

In view of the detailed examples and explanation provided above, it will be appreciated that <FIG> illustrates an example method of detecting and tracking human activities in the vicinity of a robot, in accordance with several embodiments of the inventive concepts described herein. It should be appreciated that the illustrated method is intended to encompass examples of the techniques described above, and thus where the terms and terminology used below to describe the illustrated method differ from terms and terminology above, the latter should be interpreted to encompass the related terms above, except where the context clearly indicates otherwise.

As shown at block <NUM> of <FIG>, the method comprises combining one or more two-dimensional images obtained from a time-of-flight (ToF) sensor with contemporaneously obtained data from a radar sensor, to obtain fused sensor data. As shown at block <NUM>, the method further comprises detecting the presence of a human in the vicinity of the robot, based on the fused sensor data. Finally, as shown at block <NUM>, the method comprises estimating direction of motion and speed of motion of the human, based on the fused sensor data.

The method further comprises combining the one or more two-dimensional images and data obtained from the radar sensor with contemporaneously obtained data from one or more additional sensors. In other words, the various embodiments of the method shown in <FIG> are not limited to use of only a ToF sensor and a radar sensor. Examples of the one or more additional sensors that might be used include any of: a second ToF sensor; a second radar sensor; a magnetic sensor; a photoelectric sensor; a sound sensor; an acceleration sensor; a vibration sensor; a chemical sensor; a humidity sensor; and a lidar sensor.

The detecting and estimating shown at blocks <NUM> and <NUM> are performed using a machine-learning model, the machine-learning model having been trained using two-dimensional ToF images and radar sensor data representative of an environment for the robot. This machine-learning model may be a deep learning model, e.g., an artificial neural network (ANN) or the like, such as a convolutional neural network (CNN). Note that the training may, in some instances or embodiments, be trained on the actual environment for the robot - this, of course, would be "representative" of the robot's environment. In other instances or embodiments, however, the machine-learning model could be trained on a similar environment, elsewhere.

The combining discussed above comprises aligning velocity and/or range data obtained from the radar sensor with depth and/or amplitude data in the two-dimensional images, for input into the machine-learning model. The method further comprises filtering this velocity and/or range data and the depth and/or amplitude data, to detect distinct features in the combined data, prior to the detecting and estimating steps. This filtering might comprise background subtraction.

In some embodiments or instances, the estimating of motion and speed of motion may comprise estimating direction of motion in three dimensions. Depending on the number and quality of sensors that are used, this may require the use of a series of the two-dimensional images, where, for example, the radar data is used to detect and measure motion in a direction normal to the two-dimensional images, while comparison of object location in successive images is used to detect and measure motion in directions lying in the plane of the image.

In various embodiments and instances, the method may further comprise controlling one or more actions of the robot, based on said detecting and estimating. This is shown at block <NUM> in <FIG>, which is illustrated with a dashed outline to indicate that it need not appear in all instances or embodiments of the illustrated method. As discussed above, this might involve stopping or starting operation of the robot, or altering a direction of motion, for example.

In some embodiments or instances, the method comprises generating an image, where the generated image includes a graphical feature identifying a position of the detected human and one or more graphical features indicating a direction and/or speed of the detected human. This is illustrated at block <NUM> in <FIG>. These graphical features might include, for example, the directional arrow described above; a length or other feature of the arrow might indicate a speed.

In some embodiments, the method comprises triggering an alarm and/or sending an alarm message, based on said detecting and estimating. This is shown at block <NUM> in <FIG>. This alarm might be designed to deter further movement of a human towards the robot, or to alert operators that robot operation has been suspended, in various embodiments or instances.

<FIG> is a block diagram of an apparatus for detecting and tracking human activities in the vicinity of a robot, corresponding generally in its operation and capabilities to the method illustrated in <FIG>. This apparatus comprises a time-of-flight sensor <NUM> and a radar sensor <NUM>, as well as a processing circuit <NUM> operatively coupled to the time-of-flight sensor and radar sensor. This processing circuit may comprise one or several microprocessors, microcontrollers, digital signal processors, or the like, and may comprise one or more application-specific circuits, customized digital logic, etc., in various embodiments. In the illustrated example, processing circuit comprises a microprocessor <NUM> coupled to memories <NUM> - it will be appreciated that this microprocessor and memories may take any of various well-known forms and utilize any of a wide variety of semiconductor technologies.

The processing circuit <NUM>, in various embodiments, is configured, e.g., with program code stored in memories <NUM> for execution by microprocessor <NUM>, to combine one or more two-dimensional images obtained from a time-of-flight (ToF) sensor with contemporaneously obtained data from a radar sensor, to obtain fused sensor data, to detect the presence of a human in the vicinity of the robot, based on the fused sensor data, and to estimate direction of motion and speed of motion of the human, based on the fused sensor data. It will be appreciated that the variations of the method described above for <FIG> are applicable to the operation of processing circuit <NUM>, as well.

Thus, for example, processing circuit <NUM> is configured to combine the series of two-dimensional images and data obtained from the radar sensor with contemporaneously obtained data from one or more additional sensors, in some embodiments or instances. The one or more additional sensors may comprise any one of the following, for example: a second ToF sensor; a second radar sensor; a magnetic sensor; a photoelectric sensor; a sound sensor; an acceleration sensor; a vibration sensor; a chemical sensor; a humidity sensor; and a lidar sensor.

Processing circuit <NUM> is configured to perform the detecting and estimating using a machine-learning model, the machine-learning model having been trained using two-dimensional ToF images and radar sensor data representative of an environment for the robot. The machine-learning model may be instantiated in program code and data stored in memory <NUM>, for example, as shown in <FIG> as ML model <NUM>.

The processing circuit <NUM> is configured to align velocity and/or range data obtained from the radar sensor with depth and/or amplitude data in the two-dimensional images, for input into the machine-learning model. Processing circuit <NUM> is configured to filter the velocity and/or range data and the depth and/or amplitude data, to detect distinct features in the combined data, prior to the detecting and estimating. This filtering may comprise background subtraction, in some examples.

The processing circuit <NUM> may be configured to estimate direction of motion of the detected human in three dimensions, in some embodiments or instances. Processing circuit <NUM> may be configured to control one or more actions of the robot, as described above, based on the detecting of the human the estimating of its motion. In some embodiments or instances, processing circuit <NUM> may be configured to generate an image, the generated image including a graphical feature identifying a position of the detected human and one or more graphical features indicating a direction and/or speed of the detected human. Still further, processing circuit <NUM> may be configured to trigger an alarm and/or send an alarm message, based on the detecting and estimating.

Claim 1:
A method for detecting and tracking human activities in the vicinity of a robot, the method comprising:
combining (<NUM>) one or more two-dimensional images obtained from a time-of-flight (ToF) sensor (<NUM>) with contemporaneously obtained data from a radar sensor (<NUM>), to obtain fused sensor data;
detecting (<NUM>) the presence of a human in the vicinity of the robot, based on the fused sensor data; and
estimating (<NUM>) direction of motion and speed of motion of the human, based on the fused sensor data,
wherein said detecting (<NUM>) and estimating (<NUM>) are performed using a machine-learning model, the machine-learning model having been trained using two-dimensional ToF images and radar sensor data representative of an environment for the robot; and
aligning velocity and/or range data obtained from the radar sensor (<NUM>) with depth and/or amplitude data in the two-dimensional images, for input into the machine learning model; and
filtering the velocity and/or range data and the depth and/or amplitude data, to detect distinct features in the combined data, prior to said detecting and estimating.