Patent Description:
Various entities' operations may include performance of industrial tasks, such as asset inspection, asset repair, asset manipulation, asset maintenance, environmental monitoring, etc. Such assets may include physical or mechanical devices, structures, or facilities which may, in some instances, have electrical and/or chemical aspects as well. Such assets may be used or maintained for a variety of purposes and may be characterized as capital infrastructure, inventory, or by other nomenclature depending on the context. For example, assets may include distributed assets, such as a pipeline or an electrical grid as well as individual or discrete assets, such as an airplane, a wind turbine generator, a radio tower, a steam or smoke stack or chimney, a bridge or other structure, a vehicle, and so forth. During operations, the assets may be manipulated (e.g., valve actuated, one or more actuators actuated, one or more components moved, etc.). Further, the assets may be subject to various types of defects (e.g., spontaneous mechanical defects, electrical defects, as well as routine wear-and-tear) that may impact their operation. For example, over time, the asset may undergo corrosion or cracking due to weather or may exhibit deteriorating performance or efficiency due to the wear or failure of component parts.

Typically, one or more human operators may perform the industrial task. For example, the operator may actuate a component of the asset, may locate corrosion on the asset, may locate and quantitatively or qualitatively assess cracks or defects on the asset, may assess an asset for the degree of wear-and-tear observed versus what is expected, may repair an asset, and so forth. However, depending on the location, size, and/or complexity of the asset, having one or more human operators performing industrial tasks may take away time for the operators to perform other tasks or may otherwise be time consuming and labor intensive, requiring personnel time that might be more productively spent elsewhere. Additionally, some industrial tasks may be dull, dirty, dangerous, or may be otherwise unsuitable for a human to perform. For instance, some assets may have locations that may not be accessible to humans due to height, confined spaces, or the like.

Document <CIT> discloses a system to train an industrial manipulator using a virtual reality environment.

Document <CIT> discloses a system of controlling a UAV in order to automatically carry out inspections of assets.

The invention is defined by independent claims <NUM> and <NUM>.

Further preferred embodiments are defined by the dependent claims.

Furthermore, any numerical examples in the following discussion are intended to be nonlimiting, and thus additional numerical values, ranges, and percentages are within the scope of the disclosed embodiments.

As discussed herein, the present approach relates to performance of industrial tasks using robots, unmanned vehicles, or drones and/or inspections implemented by automated or computer-based routines that learn how to perform a task based on human demonstration of the task. By way of example, such industrial tasks may be performed using unmanned or robotic devices, such as ground-based mobile robots, including those with legs, wheels, tracks, etc., unmanned aerial vehicles (UAVs), including fixed wing and rotary wing vehicles, unmanned submersible vehicles (USVs), which may swim or move along the floor of the body of liquid, or other autonomously moving vehicles that may be characterized as drones or robots. As used herein, the terms "drone" and "robot" are intended to encompass all variations, of UAVs, USVs, robotic devices, and so forth that are capable of programmable movement with limited human oversight. Such programmable movement can be based on either locally generated path waypoints or guidance or path guidance and waypoints generated by a remote system and communicated to the robot. Thus, as used herein, such devices move during an operational phase or period with limited human intervention or oversight. In accordance with present approaches, such devices may be operated to move along a flight plan, along which the devices acquire data, such as video or still image data, LIDAR data, acoustic data, spectroscopic data, temperature or pressure data, chemical samples, smells, or other data that can be acquired by sensors or cameras that can be affixed to a device moving along the flight plan. In general, such industrial tasks may be performed on one or more assets including, but not limited to, power generation assets, communication assets, transportation assets, mining or underground pumping assets, manufacture or construction assets and so forth.

Though the phrase "flight plan" is used generally herein, it should be appreciated that this phrase does not necessitate aerial movement, but instead relates to any onedimensional (1D) (such as along a track), two-dimensional (2D) (such as along a defined or undefined planar route), or three-dimensional (3D) (such as movement in the air, under water, or on a structure in where depth or altitude is also traversable), or four-dimensional (4D) (such as where there are defined temporal aspects that may characterize a velocity, acceleration, or a time on station at a waypoint) path or route along which a drone moves as part of an flight plan. Thus, a "flight plan" as used herein may be characterized as any 1D, 2D, 3D, or 4D route or path along which device such as a drone or robot is moved to perform an industrial task, including inspection tasks. Such a path may be adaptive and/or may consist of one or more waypoints along which the robot proceeds in an ordered fashion, with the sequence and location of the waypoints defining the path or route. It should be appreciated that such a flight plan may also incorporate not only temporal and/or spatial locations, but also orientation and/or alignment instructions for movement along the path and/or to exhibit at a given waypoint. Thus, the flight plan may also specify parameters such as roll, pitch, and yaw for the drone to exhibit at different points along the flight plan as well as two- or three-dimensional alignment characteristics that may relate to the direction in which a sensor or camera is pointing at a point along the flight plan. Thus, the flight plan may address not only where or when a robot is with respect to an industrial task site but, at a given location or waypoint, the direction the robot is facing or otherwise oriented with respect to. Further, even at the same waypoint and orientation, images may be acquired at different magnifications, wavelengths, or other optical parameter such that effectively the image constitutes a different view. As discussed herein, the present approach facilitates the performance of an industrial task by acquired sensor data gathered during performance of the task.

In addition, in accordance with certain aspects, prior knowledge may be leveraged during the industrial task. For example, prior knowledge may be used in generating or modifying an adaptive task plan or flight plan. In certain aspects, machine learning approaches may be employed to learn from human demonstration and/or human reviewer decisions (e.g., regarding asset condition, data sufficiency, decision oversight, mission planning, etc.), thereby creating a trained artificial neural network based on this prior knowledge that can facilitate future data sufficiency decisions.

To facilitate explanation and provide useful real-world context, various examples such as wind turbine generators, radio transmission towers, smokestacks, and so forth are provided herein. It should be appreciated however that such examples are provided merely to facilitate explanation, and the present approach is suitable for use with a wide range of other assets and at various other types of sites. Thus, the present approach is not intended to be limited to the context of the present examples.

With the preceding in mind, and turning to the figures, <FIG> depicts aspects of a system <NUM> for performing industrial tasks. As shown, the system <NUM> employs one or more robots <NUM> suitable for performing an industrial task on or near one or more assets <NUM>, such as a smokestack, or other suitable asset.

The system <NUM> also includes a remote server <NUM>, accessible via a cloud <NUM> (e.g., a network interface for accessing one or more remote servers, virtual machines, etc. for storage, computing, or other functionality), which may communicate with the one or more robots <NUM> to coordinate operation of one or more robots <NUM>, such as for performance of the task. In one implementation, the robot(s) <NUM> have onboard cellular or network connectivity and can communicate with the remote server <NUM> at least prior to and/or during performance of the task. In certain implementations the cellular or network connectivity of the robot(s) <NUM> allow communication during a task, allowing inspection data to be communicated to the remote server <NUM> and/or allowing the remote server <NUM> to communicate with a given robot <NUM>.

As shown, in some embodiments, the system <NUM> may also include a docking station <NUM> (e.g., robot garage), disposed on or near the asset <NUM>, for short term or long term storage of the robot <NUM> before and/or after performance of the task. In some embodiments, the docking station <NUM> may be in communication with the remote server <NUM> via the cloud <NUM>. If the robot <NUM> relies on a battery for power, the docking station <NUM> may also include a power source for charging the robot's <NUM> battery.

In the depicted example, the remote server <NUM> is a remote computing device accessible by the robot(s) <NUM> via the cloud <NUM>. Though only a single remote server <NUM> is shown in <FIG>, it should be understood that the functions performed by the remote server <NUM> may be performed by multiple remote servers <NUM> and/or by virtualized instances of a server environment. In the instant embodiment, the remote server <NUM> includes a data processing system <NUM>, which may include a memory component <NUM> and a processor <NUM>, for processing data received from the robot <NUM>. As is described in more detail below, in some embodiments, the robot <NUM> may provide raw data to the remote server <NUM> for processing. In other embodiments, the robot <NUM> may pre-process or partially process the data before passing it to the remote server <NUM>. In further embodiments, all of the data processing may be performed by the robot <NUM>.

The remote server <NUM> also includes a searching/parsing component <NUM>, which may also include a memory <NUM> and a processor <NUM>, for searching, parsing, and otherwise interacting with data stored on the remote server <NUM>. A user interface <NUM> may receive inputs from a user. For example, the data processing system <NUM> may utilize machine learning (e.g., a trained artificial neural network) that uses inputs from a user provided via the user interface <NUM> or from another computing device. The user inputs may be demonstrative (e.g., demonstrate how to perform a task), supervisory (e.g., the user monitors the robot and provides guidance as the robot performs the inspection), corrective (e.g., the user corrects the robot when the robot does something incorrectly), reinforcing (e.g., the user tells the robot when the robot does something correctly), decisive (e.g., the user makes a decision for the robot when prompted), instructive (e.g., identifying features in the asset), etc. In some embodiments, the user may interrupt the robot's performance of a task with feedback. In other embodiments, the robot may prompt the human operator by requesting feedback. A network interface <NUM> facilitates communication between the robot(s) <NUM> via the cloud <NUM>. As shown, the remote server <NUM> may store and maintain one or more databases <NUM>. These databases <NUM> may include instructions for performing various tasks, inspection data, configuration files, models of assets and/or areas surrounding assets, task files, algorithms, etc..

During or following performance of an industrial task, the robot <NUM> may send collected data to the remote server <NUM> for processing, analysis, and/or storage. By way of example, videos, images, LIDAR data, depth sensor data, acoustic data, spectroscopic data, or other relevant sensor or camera data acquired by the one or more robots <NUM> during a task may be uploaded to the database <NUM> as acquired or as a batch after an inspection flight plan is completed. Alternatively, in other implementations, the collected data may be provided to the database <NUM> by other means or channels, such as via direct transmission from the robot <NUM> and/or via other intermediary communication structures, such as a dedicated inspection data communication circuit.

In the depicted example, the data processing system <NUM>, the database <NUM>, and the searching/parsing component <NUM> are depicted as part of a single or common processor-based system. However, the depicted functionalities may be implemented in a distributed or dispersed manner, with certain aspects being local to the asset <NUM>, to the robot <NUM>, to an operational facility and/or other locations remote from the asset <NUM>. In such distributed implementations, the depicted aspects may still be communicatively linked, such as over one or more network connections.

As shown, the system <NUM> also includes one or more workstations <NUM>. In the illustrated embodiment, the workstations <NUM> are separate from the remote server <NUM>. However, in some embodiments, the operations of the remote server <NUM> and the workstations <NUM> may be performed by a single computing device. The workstations may include, for example, a smart phone, a tablet computer, a laptop computer, a desktop computer, a virtual reality system (including a virtual reality headset <NUM> and one or more controllers <NUM>), etc. As described in more detail below, one or more sensors (e.g., cameras, infrared sensors, proximity sensors, etc.) disposed on or around the one or more robots <NUM> may be used to generate a virtual recreation of the one or more robots and the environment around the one or more robots. In some embodiments, models of the asset, various components of or around the asset, and/or the surrounding area may be used in the virtual recreation. The virtual environment may then be displayed to an operator <NUM> via the workstation <NUM> (e.g., via the virtual reality headset, a display of the work station <NUM>, a television, a projection, etc.). The operator may then use one or more input devices (e.g., the controllers <NUM>, a mouse, a joystick, a remote control, a keyboard, etc.) to control the robot and demonstrate how the task is performed. The task may include, for example, inspecting an asset, actuating a component of an asset (e.g., actuating a valve), collecting samples of a gas, liquid, or solid, performing chemical tests, repairing an asset or a component of an asset, assembling industrial parts, polishing and blending parts, pulling or pushing controllers, cutting pipes, welding parts, etc. In some embodiments, the user's control of the one or more robots <NUM> via the input devices may result in movement of the one or more robots <NUM> on or around the asset <NUM>. In other embodiments, the user's <NUM> movement only controls virtual instantiations of the one or more robots <NUM> within the virtual environment (i.e., the actual robots <NUM> do not move in response to the user's control inputs). In some embodiments, the user may perform the industrial task more than once in order to cover various embodiments, permutations, or techniques for performing the task. Based on the user's <NUM> demonstration of the task, the system <NUM> learns how to perform the task such that one or more robots <NUM> may perform the task with minimal input and/or supervision from the human operator <NUM>. In some embodiments, the learning may be done by the server <NUM>, or a component of the server <NUM>, that subsequently controls the one or more robots to perform the industrial task. In other embodiments, the learning is done by the robots themselves based on the human operator's demonstration. The robots may then proceed to perform the industrial task. In other embodiments, the learning functions may be distributed between multiple servers <NUM>, or the robot <NUM> and one or more servers <NUM>. In some embodiments, however, a human operator <NUM> may utilize the workstation <NUM> to monitor and/or supervise the robots <NUM>, and to provide feedback and/or inputs to the robots when requested. For example, once the robots <NUM> have learned how to perform the tasks, the system <NUM> may generate a virtual recreation of the environment surrounding the robot <NUM>, or a video of the robot <NUM>, as the robot <NUM> performs the task. The human operator <NUM> may monitor the performance of the task via the workstation <NUM> and provide feedback (e.g., when prompted or otherwise) during the performance of the task.

<FIG> is a schematic of an embodiment of the robot <NUM> shown in <FIG>. It should be understood, however, that other embodiments of the robot <NUM> are envisaged having additional components, fewer components, and/or different combinations of components. As shown, the robot <NUM> includes a power supply <NUM> to provide power for the operation of the robot <NUM>. The power supply <NUM> may include a replaceable or rechargeable battery, a combustion engine, a generator, and electric motor, a solar panel, a chemical-reaction based power generation system, etc., or some combination thereof.

The robot may include a user interface <NUM>, by which a user may set up or adjust various settings of the robot <NUM>. The user interface may include one or more input devices (e.g., knobs, buttons, switches, dials, etc.) and in some cases may include a display (e.g., a screen, array of LEDs, etc.) for providing feedback to the user. Though previously discussed embodiments receive user feedback via the user interface of the remote server, embodiments in which user feedback is provided via the user interface <NUM> of the robot <NUM> are also envisaged.

A network interface <NUM> enables communication with the remote server via the cloud, or other devices (e.g., the docking station, a remote controller, a smart phone, a computing device, a tablet, etc.). For example, the network interface <NUM> may enable communication via a wireless network connection, a wired network connection, cellular data service, Bluetooth, Near Field Communication (NFC), ZigBee, ANT+, or some other communication protocol. In some embodiments, data sent or received via the network interface may be encrypted. For example, standard data encryption techniques may be utilized, such as hashing, key exchange encryption, symmetric encryption methods, asymmetric encryption methods, or a combination thereof. In some embodiments, the network interface <NUM> may allow the robot <NUM> to connect to a remote controller, a computing device, a smart phone, a tablet computer, etc..

A sensing system <NUM> may include one or more sensors <NUM> (e.g., tactile, chemical, ultrasound, temperature, laser, sonar, camera, a red, blue, green, depth (RGB-D) camera, etc.) configured to sense various qualities and collect data corresponding to the asset during performance of an industrial task. As previously discussed, some of the one or more sensors <NUM> may be used to generate a virtual recreation of the environment surrounding the robot.

A drive system <NUM> may actuate movement of the robot <NUM> through the air, through a liquid, along a surface, or some combination thereof. As shown, the drive system <NUM> may include one or more motors <NUM> and one or more encoders <NUM>. The one or more motors <NUM> may drive propellers, legs, wheels, tracks, etc. The one or more encoders <NUM> may sense one or more parameters of the one or more motors <NUM> (e.g., rotational speed) and provide data to a control system <NUM>.

The control system <NUM> may include one or more memory components <NUM> and one or more processors <NUM>. A motion control system <NUM> may receive a signal from the one or more encoders <NUM> of the drive system <NUM>, and/or commands from the human operator via the network interface <NUM>, and output a control signal to the one or more motors <NUM> to control the movement of the robot <NUM>. Similarly, a data collection control system <NUM> may control the operation of the sensing system <NUM> and receive data from the sensing system <NUM>. A data processing and analysis system <NUM> may receive data collected by the sensing system <NUM> and process or analyze the collected data. In some embodiments, the data processing and analysis system <NUM> may completely process and analyze the data and make a determination as to the condition of the asset. In other embodiments, the data processing and analysis system <NUM> may perform pre-processing of the data or a partial processing and analysis of the data and then send the data to the remote server for the remainder of processing and analysis. In further embodiments, the robot <NUM> may send raw data to the remote server. When user feedback is provided, the data processing and analysis system <NUM> may take the user inputs into account when processing and/or analyzing the inspection data. In some embodiments, user feedback may be communicated back to the robot <NUM>.

In one implementation, the control system <NUM> may also include a mission planning component <NUM>. The mission planning component <NUM> generates a mission plan and executes the mission plan by coordinating the various other components of the control system <NUM> and the robot <NUM>. The mission planning component <NUM> may utilize, for example, control inputs of the human operator controlling the robot <NUM> in real time or near real time, data describing how industrial tasks are to be performed (e.g., based on demonstration by the human operator), models of the asset, components of the asset, components in the vicinity of the robot, and/or models of the surrounding area. The mission planning component <NUM> may then learn from the human operator's demonstration of the task, and generate a plan for the robot <NUM> to perform the industrial task autonomously or semi-autonomously. As described in more detail below with regard to <FIG>, in some embodiments, the robot may include a manipulator that enables the robot to interact with other components. For example, the manipulator may be an arm or some other structure that the robot utilizes to interact with objects in its surrounding environment to perform the industrial task.

<FIG> generally illustrates a block diagram of example components of a computing device <NUM> that could be used as the remote server and/or the workstation. As used herein, a computing device <NUM> may be implemented as one or more computing systems including laptop, notebook, desktop, tablet, or workstation computers, as well as server type devices or portable, communication type devices, such a cellular telephones, and/or other suitable computing devices.

As illustrated, the computing device <NUM> may include various hardware components, such as one or more processors <NUM>, one or more busses <NUM>, memory <NUM>, input structures <NUM>, a power source <NUM>, a network interface <NUM>, a user interface <NUM>, and/or other computer components useful in performing the functions described herein.

The one or more processors <NUM> are, in certain implementations, microprocessors configured to execute instructions stored in the memory <NUM> or other accessible locations. Alternatively, the one or more processors <NUM> may be implemented as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or other devices designed to perform functions discussed herein in a dedicated manner. As will be appreciated, multiple processors <NUM> or processing components may be used to perform functions discussed herein in a distributed or parallel manner.

The memory <NUM> may encompass any tangible, non-transitory medium for storing data or executable routines, including volatile memory, non-volatile memory, or any combination thereof. Although shown for convenience as a single block in <FIG>, the memory <NUM> may actually encompass various discrete media in the same or different physical locations. The one or more processors <NUM> may access data in the memory <NUM> via one or more busses <NUM>.

The input structures <NUM> are used to allow a user to input data and/or commands to the device <NUM> and may include mice, touchpads, touchscreens, keyboards, controllers, and so forth. The power source <NUM> can be any suitable source for providing power to the various components of the computing device <NUM>, including line and battery power. In the depicted example, the device <NUM> includes a network interface <NUM>. Such a network interface <NUM> may allow communication with other devices on a network using one or more communication protocols. In the depicted example, the device <NUM> includes a user interface <NUM>, such as a display configured to display images or date provided by the one or more processors <NUM>. The user interface <NUM> may include, for example, a monitor, a display, a virtual reality headset, a television, a projector, or some combination thereof. As will be appreciated, in a real-world context a processor-based systems, such as the computing device <NUM> of <FIG>, may be employed to implement some or all of the present approach, such as performing the functions of the remote server and/or the workstations shown in <FIG>.

<FIG> is a schematic illustrating interactions between the robot <NUM>, the remote server <NUM>, and the workstation <NUM> in performing an industrial task. The robot <NUM>, the remote server <NUM>, and the workstation <NUM> may communicate with one another via respective network interfaces (e.g., communication servers). In the illustrated embodiment, some functions are shared between the robot <NUM> and the server <NUM>. It should be understood that these functions may be performed entirely by the robot, or may be shared between the robot <NUM> and the server <NUM>. As previously described, the sensing system <NUM> includes one or more sensors that collect information about the environment surrounding the robot <NUM>. The collected data is provided to the video detection and processing system <NUM>, where the collected data is processed, and then passed to a task planning component <NUM> of the mission planning system <NUM>. The task planner <NUM> transmits the data to the work station <NUM> via the network interface <NUM> (e.g., robot server) of the robot <NUM> and the remote server <NUM>. The data is received by a network interface <NUM> of the workstation <NUM> (e.g., via the remote server <NUM>, the cloud <NUM>, or both), and then passed to a states processing component <NUM>. The states processing component <NUM> uses the received data to generate a visualization <NUM> of the environment surrounding the robot <NUM>, and a visualization <NUM> of the robot <NUM> itself within the environment. The visualizations are then presented to the human operator <NUM> via a user interface <NUM> (e.g., the virtual reality headset <NUM>). Using one or more controllers <NUM> of the user interface <NUM>, the user <NUM> controls the robot visualization <NUM> within the environmental visualization <NUM> to perform the task. The inputs by the human operator <NUM> may then be sent back to the robot <NUM> (via the respective network interfaces <NUM>, <NUM>). The robot <NUM> may then generate one or more control signals based on the inputs of the human operator <NUM> to control one or more components (e.g., the drive system <NUM>, an industrial manipulator <NUM>, an actuator, etc.) of the robot <NUM>. As shown, the workstation <NUM> may include a task definitions database <NUM> for various tasks performed, such that the human operator can associate various actions demonstrated via the virtual reality system <NUM> with one or more industrial tasks.

As the human operator <NUM> demonstrates the task, a learning mechanism breaks down the demonstrated activity into component parts (e.g., actions), generalizes the actions, and learns how to perform the actions autonomously or semi-autonomously. Data for how to recreate the demonstrated task (e.g., configuration of the robot, environmental state, goals, executed motions, sequence of motions, goals, etc.) may be stored in a knowledge base <NUM> for retrieval later.

When the robot <NUM> is instructed to perform the demonstrated task, the task planner <NUM> of the mission planning system <NUM> retrieves the data associated with the task from the knowledge base <NUM> and plans to perform the task autonomously or semi-autonomously. As shown, the mission planning system <NUM> may also include a robot manipulation planner <NUM>, which may plan and control an industrial manipulator <NUM> (e.g., arms, actuators, etc.) that may manipulate an asset or a component of an asset, as well as a robot navigation planner <NUM>, which plans the flight plan for the robot <NUM>. In some embodiments, the robot navigation planner <NUM> may rely on input from a localization or simultaneous localization and mapping (SLAM) component <NUM> to identify the robot's <NUM> position. The robot <NUM> may then proceed to perform the planned task autonomously or semi-autonomously. As the task is being performed, the sensing system <NUM> may collect video, which is passed through the video detection and processing component <NUM> and the network interface <NUM> to the workstation <NUM>. As previously described, the data is used to create a visualization <NUM> of the surrounding environment and a visualization <NUM> of the robot <NUM> within the environment for display to the human operator (e.g., via the user interface <NUM>). The human operator <NUM> may then supervise the robot <NUM> as it performs the task. At times, the robot <NUM> may request input from the human operator <NUM> as the robot <NUM> performs the task. Other times, the human operator <NUM> may provide feedback unprompted.

<FIG> schematically depicts an example of the learning mechanism <NUM> (e.g., artificial neural network) that may be trained as a deep learning model as discussed herein. In this example, the network <NUM> is multi-layered, with a training input <NUM> (e.g., task demonstration and/or user feedback) and multiple layers including an input layer <NUM>, hidden layers <NUM>, <NUM>, and so forth, and an output layer <NUM> and the training target <NUM> present in the network <NUM>. Each layer, in this example, is composed of a plurality of "neurons" or nodes <NUM>. The number of neurons <NUM> may be constant between layers or, as depicted, may vary from layer to layer. Neurons <NUM> at each layer generate respective outputs that serve as inputs to the neurons <NUM> of the next hierarchical layer. In practice, a weighted sum of the inputs with an added bias is computed to "excite" or "activate" each respective neuron of the layers according to an activation function, such as rectified linear unit (ReLU), sigmoid function, hyperbolic tangent function, or otherwise specified or programmed. The outputs of the final layer constitute the network output <NUM> which, in conjunction with a target value or construct <NUM>, are used to compute some loss or error function <NUM>, which will be backpropagated to guide the network training.

The loss or error function <NUM> measures the difference between the network output <NUM> and the training target <NUM>. In certain implementations, the loss function may be a mean squared error (MSE). Alternatively, the loss function <NUM> could be defined by other metrics associated with the particular task in question, such as a softmax function.

With the preceding in mind, the neural network <NUM> may be trained for use in learning how to perform and planning industrial tasks.

<FIG> is a flow chart of a process <NUM> for performing an industrial task based on a human demonstration of the task. In block <NUM>, the human operator demonstrates performance of the task. As previously described, data from one or more sensors disposed on or near the robot may be used to create a virtual recreation of the robot and its surrounding environment. The data may include, for example, video or still image data, LIDAR data, acoustic data, spectroscopic data, temperature or pressure data, chemical samples, smells, or other data that can be acquired by sensors or cameras that can be affixed to a device moving along the flight plan. The virtual environment may be displayed to the human operator via a workstation (e.g., virtual reality headset, computer monitor, television, projector, etc.). The human operator may then control the virtual robot within the virtual environment (including motion and analytics) via one or more controllers to demonstrate performance of the task. In some embodiments, the human operator's inputs are used to control the actual robot performing the task at a remote location. In some embodiments, the human operator may set sub-goals for the robot to accomplish. The robot may plan motion to accomplish the sub-goals, which may be selected and/or validated by the human operator. After selection and/or confirmation, the robot may perform the planned motion. In some embodiments, the robot may control the position of the robot, while the human operator controls the manipulator to perform the task. In some embodiments, tasks may be divided into sub-tasks. At block <NUM>, the industrial task being demonstrated is defined. A task is defined as: starting state, goal state, and constraints. A robot needs to drive the state, including the robot itself and the environment, from the starting state to the goal state, while satisfying the constraints defined in the task. At block <NUM>, the various states of the robot during the demonstration of the task are represented. The states describe the status of the current system, either a geometrical relationship and/or a representational description of objects and robot when a task is executing. Robots may group or ungroup the representation of the states when necessary to facilitate the semantic understanding of the demonstrations. At block <NUM>, the data from the demonstration is saved. The saved data may include, for example, robot configuration, environmental state, robot position and/or movement, manipulator position and/or motion, operator inputs, goals, sequence of events, timing, forces and/or torques applied, etc. For example, the human operator may move the robot base (i.e., the body of the robot) to a position near a component to be moved or actuated. The position may be such that a manipulator has access to the component. The human operator may then use the manipulator (e.g., an arm) to move or actuate the component to perform the task.

At block <NUM>, the demonstration is generalized. For example, the motion plans may be generalized to find the distribution that represents the most plausible and applicable solution. Generalization may include finding the most common constraints of executing a task using a motion plan. At block <NUM>, the system learns how to perform the task. For example, learning the task may include machine learning techniques. Based on the demonstration of the task by the human operator, the system may generate a behavior sequence or adapt an existing behavior sequence to match the demonstration. The learned knowledge is represented in the task space and defined as a finite number of states. For example, behavior sequence generation may include finding a solution to achieve a task goal based on learned knowledge. Given a task, the system finds the subtasks to achieve the task goal based on learned knowledge. The goals of each subtask are then extracted based on the learned knowledge. In some embodiments, the goals may be validated by a human operator as an interim step. At block <NUM>, the system generates a motion plan to perform each subtask. The motion plan may be associated with prior knowledge, but may also be adapted to new task-relevant situations. The candidate motion plan may then be displayed to a user for review. For example, visualizations (e.g., animations, images, video, etc.) of hypothetical execution of the motion plan may be generated and displayed (e.g., via a virtual reality mask, display, television, or projector of the work station) to the user for review. At block <NUM>, the human operator may provide feedback via the controllers or user inputs of the workstation.

If the motion plan is approved by the human operator, the robot may proceed to perform the task. In some embodiments, the sensors on or around the robot may continue to collect data. The collected data may be used to recreate a virtual recreation of the robot and its environment to be displayed to the human operator as the robot performs the task. The human operator may supervise the robot and provide feedback as the robot performs the task (block <NUM>). The system may use the human operator feedback to modify its motion plan. The feedback may also be taken into account as the system re-learns the task taking into account the human operator's feedback. As such, the process <NUM> may return to block <NUM>. The process <NUM> may then repeat the task, or move on to a new task.

The presently disclosed techniques include teaching one or more robots to perform an industrial task autonomously or semi-autonomously based on human demonstration of the task. Specifically, one or more sensors disposed on or around the one or more robots may collect data (e.g., video, images, etc.) corresponding to the robots and their surrounding environment. The collected data are then used to generate a virtual recreation of the robots and their surrounding environments, which are displayed to a human operator (e.g., via a virtual reality headset, computer monitor, etc.). The human operator then uses a controller to control the robots within the virtual environment to perform the task. The actual robots may also be controlled by the user based on the user's inputs. The robot then learns how to perform the task based on the human operator's demonstration of the task. The robot may then go on to perform the task autonomously or semi-autonomously. In some embodiments, the sensors may continue to collect data and generate the virtual recreation of the robots and their surrounding environments by which the human operator may supervise the one or more robots as they perform the task autonomously or semiautonomous. The human operator may periodically provide feedback, which may allow the robots to refine their understanding of how to perform the task.

Claim 1:
A system (<NUM>) for performing industrial tasks, comprising:
a robot (<NUM>), comprising one or more sensors configured to collect data corresponding to the robot (<NUM>) and an environment surrounding the robot (<NUM>); and
a computing device disposed remote from the robot (<NUM>), and wherein the robot (<NUM>) and the computing device are configured to communicate with one another via a cloud interface (<NUM>), the computing device comprising:
a user interface (<NUM>);
a processor (<NUM>); and
a memory (<NUM>) comprising instructions that, when executed by the processor (<NUM>), cause the processor (<NUM>) to:
receive, via the cloud interface (<NUM>), the collected data from the robot (<NUM>);
generate a virtual recreation of the robot (<NUM>) and the environment surrounding the robot (<NUM>);
receive, via the cloud interface (<NUM>), inputs from a human operator controlling the robot (<NUM>) to demonstrate an industrial task, wherein the robot (<NUM>) is disposed remote from the human operator;
wherein the system (<NUM>) is configured to:
learn how to perform the industrial task based on the human operator's demonstration of the task;
generate a motion plan for performing the industrial task;
display the motion plan to the human operator for approval;
perform, via the robot (<NUM>), the industrial task autonomously or semi-autonomously upon approval of the motion plan;
receive feedback from the human operator during the performance of the task or following performance of the task; and
relearn how to control the robot (<NUM>) to perform the industrial task based on the feedback provided by the human operator.