Patent Description:
Many modern robots and other machines are designed to operate with increased autonomy and are less reliant on well-trained operators to safely operate. Some of these modern robots are manned while others are unmanned. In particular, a variety of unmanned vehicles include unmanned ground vehicles (UGVs), unmanned aerial vehicles (UAVs), unmanned surface vehicles (USVs), unmanned underwater vehicles (UUVs), unmanned spacecraft and the like. The use of unmanned vehicles has grown in recent years and these unmanned vehicles are employed in a wide variety of applications, including both military and civilian uses.

One focus in the field of robotics is in the improvement of autonomy, which often includes multiple examples of robot operation. These examples of robot operation include automatic control of a given robot to support remote human control. Another example is optimization systems (and associated methods) to determine how, for a given robot or set of robots, tasks should be ordered and/or allocated. And yet another example of robot operation is automatic, real-time or near real-time data processing, and exploitation in support of automatic route planning, mission execution and other activities.

Despite advancements, existing autonomy systems are typically configured to address only one example of these activities, thereby focusing its design of the underling autonomy algorithms and software architecture on a narrow mission set. This limits the extensibility of existing autonomy systems. Furthermore, it is generally desirable to improve existing systems to enhance their efficiency and operation.

<NPL>, in accordance with its abstract, states in most complex missions, unexpected situations arise that may interfere with the planned execution of mission tasks. These situations result in the generation of contingency tasks that need to be executed before the originally planned tasks are completed. Potential contingency tasks may not always affect mission tasks due to the inherent uncertainty in the environment. Deferring action on a potential contingency task may incur a penalty in terms of wasted time due to idle robots if the contingency task becomes a bottleneck in the future. On the other hand, immediate action on a potential contingency task may incur a penalty in terms of wasted time if the contingency task did not actually impact the mission. When a contingency task is reported, the planner generates an updated plan that minimizes expected mission completion time by taking into account the probability of the contingency task impacting mission tasks, its effect on the mission, and its spatial location.

Further background may be found in <CIT>; <NPL>; <NPL>; and <NPL>.

<NPL>, in accordance with its abstract, states "Civil Aviation Authorities are elaborating a new regulatory framework for the safe operation of Unmanned Aircraft Systems (UAS). Current proposals are based on the analysis of the specific risks of the operation as well as on the definition of some risk mitigation measures. In order to achieve the target level of safety, we propose increasing the level of automation by providing the on-board system with Automated Contingency Management functions. The aim of the resulting Safe Mission Manager System is to autonomously adapt to contingency events while still achieving mission objectives through the degradation of mission performance. In this paper, we discuss some of the architectural issues in designing this system. The resulting architecture makes a conceptual differentiation between event monitoring, decision-making on a policy for dealing with contingencies and the execution of the corresponding policy. We also discuss how to allocate the different Safe Mission Manager components to a partitioned, Integrated Modular Avionics architecture. Finally, determinism and predictability are key aspects in contingency management due to their overall impact on safety. For this reason, we model and verify the correctness of a contingency management policy using formal methods.

Therefore it would be desirable to have a system and method that takes into account at least some of the issues discussed above, as well as other possible issues.

There is described herein a method, optionally a computer-implemented method, of causing one or more robots to execute a mission in an environment. The method comprises, for a robot of the one or more robots, accessing mission data for the mission including or associated with tasks that are executable to cause the robot to execute maneuvers. The method further comprises executing the tasks according to the mission data to cause the robot to execute the maneuvers and, as the tasks are executed, monitoring at least one of the robot or the environment. The method further comprises detecting an event based on the monitoring, during execution of a task of the tasks. The method further comprises mapping the event to a level of contingency of a plurality of levels of contingency. The method further comprises transitioning from the task to another of the tasks according to the level of contingency. The method further comprises executing the other of the tasks. The mission data includes a task graph in which the mission is modeled, the task graph expressed as a directed graph and including task nodes representing the tasks that are connected by edges representing transitions between the tasks. Executing the tasks includes traversing the task graph including the task nodes. When a task node representing the task is visited, calling on a task library of the tasks that are executable to cause the robot to execute the maneuvers, thereby causing the robot to execute a maneuver. Transitioning from the task to the other of the tasks includes transitioning from the task node on an edge that connects the task node to another of the task nodes representing the other of the tasks, the edge associated with the level of contingency.

There is also provided a computer program comprising computer program instructions that, when executed by a computer processor, cause the computer processor to perform the method of the preceding paragraph, and a computer-readable storage medium having stored thereon such a computer program. There is also provided an apparatus for causing one or more robots to execute a mission, the apparatus comprising: a memory having computer-readable program code stored therein; and processing circuitry configured to access the memory, and execute the computer-readable program code to cause the apparatus to perform the method of the preceding paragraph.

A mission is a deployment of a robot (one or more robots) to achieve one or more mission objectives. A mission may be decomposed into behaviors or actions involving the robot maneuvering with optional sensor and/or effector scheduling; and a mission may be enhanced by contemplating contingency events whose resolution makes use of other behaviors. In various examples, a maneuver may be defined as a path through space, a path and velocity through space, rate or acceleration commands, direct actuator and propulsion commands, or the like. Contingency events may include a strong tail wind, actuator failure, propulsion degraded mode, propulsion failure, computer redundancy failure, navigation degraded mode (e.g., GPS loss), navigation failure, communication loss, sensor degraded mode, sensor failure, or the like.

According to some examples, a robot may be managed to execute tasks to implement these behaviors with specific parameters and capabilities, and these tasks may be modeled in a task graph of nodes interconnected through transition logic. The task graph may decouple the semantic of a specific mission from its general structure; and therefore, decouple mission planning from mission execution. This may in turn separate non-predictive decision making from predictive mission making. A mission planning and mission execution may together realize autonomic behavior.

The robot and/or its environment is monitored for events such as contingency events. When an event is detected, the event is mapped to a level of contingency, and the event is resolved according to the level of contingency.

In some examples in which a mission is modeled in a task graph, feasibility of a path in the task graph may be determined before the mission is executed, as well as during execution of the mission. In particular, a predicted performance of the robot for the path may be determined, and the feasibility of the path may be determined from the predicted performance of the robot. A mission on a feasible path may continue uninterrupted, while a mission that is on an infeasible path may cause any of a number of different responses. An infeasible path may be replanned, its determination may be treated as a contingency event to be resolved, or it may prompt an operator to intervene.

In an example, not currently claimed, there is provided a method of causing one or more robots to execute a mission, the method comprising identifying the mission including a nominal sequence of selected tasks that are executable to cause the one or more robots to execute maneuvers to achieve a mission objective; determining a task graph in which the mission is modeled, the task graph expressed as a directed graph and including selected task nodes representing the selected tasks that are connected by edges representing transitions between the selected tasks; and causing the one or more robots to execute the mission using the task graph and a task library of tasks including a selected task executable to cause the one or more robots to execute a maneuver.

The selected task in the nominal sequence of the selected tasks may cause the one or more robots to execute a nominal maneuver, or an alternate maneuver when a task-specific contingency event occurs during execution of the nominal maneuver, and wherein causing the one or more robots to execute the mission may include causing the one or more robots to execute the mission using the task library that includes the selected task executable to cause the one or more robots to execute the nominal maneuver, or the alternate maneuver when the task-specific contingency event occurs.

The mission may further include one or more alternate tasks to be executed when a contingency event occurs during execution of the selected task in the nominal sequence of selected tasks, and wherein determining the task graph may include determining the task graph further including one or more alternate task nodes representing the one or more alternate tasks.

The mission may be associated with one or more alternate tasks to be executed when a contingency event occurs during execution of any one of the selected tasks in the nominal sequence of selected tasks, and wherein determining the task graph may include determining an alternate task graph including one or more alternate task nodes representing the one or more alternate tasks, each selected task node associated with the alternate task graph.

Causing the one or more robots to execute the mission may include for a robot of the one or more robots accessing mission data including the task graph in which the mission is modeled; traversing the task graph including the selected task nodes; and when a selected task node representing the selected task is visited, calling on the task library to execute the selected task and thereby cause the robot to execute the maneuver.

The mission may further include an alternate task to be executed when a contingency event occurs during execution of the selected task, the task graph may further include an alternate task node representing the alternate task, and the tasks in the task library may further include the alternate task, and wherein the method may further comprise for the robot of the one or more robots detecting occurrence of the contingency event during execution of the selected task; transitioning in the task graph from the selected task node to the alternate task node; and calling on the task library to execute the alternate task and thereby cause the robot to execute another maneuver.

The mission may be associated with one or more alternate tasks to be executed when a contingency event occurs during execution any one of the selected tasks, the task graph may be associated with an alternate task graph including one or more alternate task nodes representing the one or more alternate tasks, and the tasks in the task library may further include the one or more alternate tasks, and wherein the method may further comprise for the robot of the one or more robots detecting occurrence of the contingency event during execution of any one of the selected tasks; transitioning to the alternate task graph; and calling on the task library to execute the one or more alternate tasks and thereby cause the robot to execute one or more other maneuvers.

These and other features, examples, and advantages of the present disclosure will be apparent from a reading of the following detailed description together with the accompanying figures, which are briefly described below. The present disclosure includes any combination of two, three, four or more features or elements set forth in this disclosure, regardless of whether such features or elements are expressly combined or otherwise recited in a specific example described herein. This disclosure is intended to be read holistically such that any separable features or elements of the disclosure, in any of its examples, should be viewed as combinable unless the context of the disclosure clearly dictates otherwise.

It will therefore be appreciated that this Brief Summary is provided merely for purposes of providing a basic understanding of some examples of the disclosure. Accordingly, it will be appreciated that the above described examples are merely examples and should not be construed to narrow the scope of the disclosure in any way. Other examples and advantages will become apparent from the following detailed description taken in conjunction with the accompanying figures which illustrate, by way of example, the principles of some described examples.

Having thus described examples of the disclosure in general terms, reference will now be made to the accompanying figures, which are not necessarily drawn to scale, and wherein:.

Some examples of the present disclosure will now be described more fully hereinafter with reference to the accompanying figures, in which some, but not all examples of the disclosure are shown. Indeed, various examples of the disclosure may be embodied in many different forms and should not be construed as limited to the examples set forth herein. For example, unless specified otherwise or clear from context, references to first, second or the like should not be construed to imply a particular order. A feature described as being above another feature (unless specified otherwise or clear from context) may instead be below, and vice versa; and similarly, features described as being to the left of another feature else may instead be to the right, and vice versa. As used herein, unless specified otherwise or clear from context, the "or" of a set of operands is the "inclusive or" and thereby true if and only if one or more of the operands is true, as opposed to the "exclusive or" which is false when all of the operands are true. Thus, for example, "[A] or [B]" is true if [A] is true, or if [B] is true, or if both [A] and [B] are true. Further, the articles "a" and "an" mean "one or more," unless specified otherwise or clear from context to be directed to a singular form. Furthermore, it should be understood that unless otherwise specified, the terms "data," "content," "digital content," "information," and similar terms may be at times used interchangeably.

Examples of the present disclosure relate to robotics and, in particular, to one or more of the design, construction, operation or use of robots. As used herein, a robot is a machine designed and configurable to execute maneuvers in its environment. The robot may be manned or unmanned. The robot may be fully human-controlled, or the robot may be semi-autonomous or autonomous in which at least some of the maneuvers are executed independent of or with minimal human intervention. In some examples, the robot is operable in various modes with various amounts of human control.

A robot designed and configurable to fly may at times be referred to as an aerial robot. A robot designed and configurable to operate with at least some level of autonomy may at times be referred to as an autonomous robot, or an autonomous aerial robot in the case of an autonomous robot that is also designed and configurable to fly. Examples of suitable robots include aerobots, androids, automatons, autonomous vehicles, explosive ordnance disposal robots, hexapods, industrial robots, insect robots, microbots, nanobots, military robots, mobile robots, rovers, service robots, surgical robots, walking robots and the like. Other examples include a variety of unmanned vehicles, including unmanned ground vehicles (UGVs), unmanned aerial vehicles (UAVs), unmanned surface vehicles (USVs), unmanned underwater vehicles (UUVs), unmanned spacecraft and the like. These may include autonomous cars, planes, trains, industrial vehicles, fulfillment center robots, supply-chain robots, robotic vehicles, mine sweepers, and the like.

<FIG> illustrates one type of robot, namely, a UAV <NUM>, that may benefit from examples of the present disclosure. As shown, the UAV may include a fuselage <NUM>, wings <NUM> extending from opposing sides of the UAV in a mid-section of the fuselage, and an empennage or tail assembly <NUM> at a rear end of the fuselage. The tail assembly includes a vertical stabilizer <NUM> and two horizontal stabilizers <NUM> extending from opposing sides of the UAV. Rotors <NUM> and <NUM> are mounted to respectively the wings and the end of the tail assembly for lifting and propelling the UAV during flight.

<FIG> illustrates a system <NUM> that includes any of a number of different subsystems (each an individual system) for performing one or more functions or operations. As shown, in some examples, the system includes a control station <NUM> and one or more robots <NUM> (e.g., one or more UAVs <NUM>). The control station provides facilities for communication with or control of the one or more robots, such as by wired or wireless data links directly or across one or more networks <NUM>. In some examples, the control station may be a ground station, and not in all cases control the robots. In this regard, the control station may be configured to monitor the robots. The control station may initiate mission, but the control station may not control the robots to maneuver. At times, then, the control station may enable or provide a distributed network/server of software functions.

The robot <NUM> includes a robot management system (RMS) <NUM> and a mission management system (MMS) <NUM>. The RMS is a robot-specific subsystem configured to manage subsystems and other components of the robot. These subsystems and other components include, for example, maneuver controls, landing gear, onboard environmental systems, electrical, pneumatic and hydraulic systems, communications systems, navigation systems and other subsystems and components for controlling operation and maneuvering of the robot. The RMS is configured to accept maneuver commands such as waypoints and/or steering commands, and control the robot to follow those maneuver commands. In the context of a vehicle, the RMS is at times referred to as a vehicle management system (VMS).

The MMS <NUM> is a subsystem configured to manage missions of the robot <NUM>. A mission is a deployment of the robot (one or more robots) to achieve one or more mission objectives. A mission may be decomposed into maneuvers of the robot with optional sensor and/or effector scheduling, and the MMS may execute tasks to manage the robot to execute maneuvers with specific parameters and capabilities. In some examples, a mission may also include idle tasks in which the vehicle may idle while other mission systems operate. These idle tasks may include, for example, on-ground tasks, system startup tasks, tasks employing sensors or effectors, and the like. The MMS <NUM> includes subsystems to process sensor data to situational awareness, plan tasks for the robot <NUM> (or multiple robots), coordinate with teams to assign tasks, execute assigned tasks. The MMS is also configured to interface with the RMS <NUM>, and in some examples the control station <NUM>. Although the MMS is shown on the robot <NUM>, the MMS may instead be at the control station; or in some examples, the MMS may be distributed between the robot and the control station.

In some examples, the MMS <NUM> provides a complete, end-to-end autonomy architecture with open system architecture standards and parameterized to allow rapid extension and reapplication to a variety of robots. The flexibility of the MMS enables an operator to code it once, but to apply it anywhere. The MMS may therefore be applied to virtually any robot that applies, or benefits from, autonomy. The MMS may include an adaptable autonomy architecture that is applicable to a variety of robots, including those identified above. A benefit of the MMS is therefore not only in the specific contents, but also in the specific details of the architecture, its subroutines, and in the interfaces between those subroutines and other systems/devices that support rapid extensibility and adaptability of the MMS to a variety of domains.

<FIG> more particularly illustrates the MMS <NUM>. The MMS may include any of a number of different subsystems (each an individual system) for performing one or more functions or operations. As shown, in some examples, the MMS includes an interface subsystem <NUM>, a situational awareness subsystem <NUM>, a mission planning subsystem <NUM>, a mission coordination subsystem <NUM>, and a mission execution subsystem <NUM>. As suggested above, in some examples, the subsystems of the MMS may be on the robot <NUM>, at the control station <NUM>, or distributed between the robot and the control station. The subsystems may be configured to communicate with one another directly, over a communication bus <NUM>, or across the network(s) <NUM> in examples in which the MMS is distributed between the robot and the control station.

The subsystems enable the MMS <NUM> of the robot <NUM> to interface with the system <NUM>, perform situational awareness, plan a mission including a plurality of tasks, coordinate the plurality of tasks and thereby the mission with other robots <NUM>, and execute the mission. For example, the MMS may use the interface subsystem <NUM> to interface with various sensors onboard the robot, the RMS <NUM>, the control station <NUM> and/or other robots. The MMS may use the situational awareness subsystem <NUM> to acquire sensor data and maintain an awareness of the state of the environment in which the robot is operating. The MMS may use the mission planning subsystem <NUM> to plan a mission including or associated with a plurality of tasks, and which may incorporate rules of engagement, tactics and other constraints on operations. The MMS may likewise use the mission planning subsystem to dynamically replan a mission in which changes to the mission are made in real-time or near real-time as the mission is executed. The MMS may use the mission coordination subsystem <NUM> to coordinate the plurality of tasks of the mission with other robots and users, where agreed-upon tasks may then be executed by the MMS using the mission execution subsystem <NUM>.

According to some examples, various ones of the subsystems of the MMS <NUM> are configured to implement various software functionality or functionalities (at times referred to as services) to perform their respective functions. <FIG> is a diagram of services <NUM> that may be implemented by the mission planning subsystem <NUM> to plan or dynamically replan a mission. As shown, the services may include a task planner service <NUM> and a task allocation service <NUM>.

A basic part of autonomy is a cost function to make decisions on any given set of actions, and a simple cost function is typically based on distance, time, or fuel is used by decision making algorithms. This simplifies the cost function to something trivial (yet still useful) to allow research into the more difficult generation and optimization of action sets. The services <NUM> of some examples determine a probability of success of a mission including sequence of selected tasks to be executed by one or more robots. The services provide a library of probabilities of events that impact success of tasks, which may be used to determine the probability of a single task. The probability of success may be determined for selected tasks across a mission, which may then be used in a cost function to evaluate the mission.

In particular, the task planner service <NUM> may acquire information about tasks and their characteristics, including performance data that includes expected results of task selection that can help decide whether or not tasks should be selected for the mission. This performance data may be used as a decision basis for other services of the mission planning subsystem <NUM> such as the task allocation service <NUM>. The task allocation service reviews the performance data, and selects tasks and arranges the tasks in sequence. The task allocation service may also determine which robots <NUM> should execute the selected tasks. For these operations, the task allocation service may use evaluation criteria such as cost (in the context of a cost function).

The task planner service <NUM> is configured to receive inputs from a variety of sources,
including a probability of success service <NUM> of the mission planning subsystem <NUM>. As explained in greater detail below, the probability of success service may be used to determine probabilities of success of selected tasks. The performance data from the task planner service to the task allocation service <NUM> may include these probabilities of success. The task allocation service may use the probabilities of success of the selected tasks to determine a cost of the mission, which may be used to evaluate a mission.

According to examples, the task planner service <NUM> and the task allocation service <NUM> cooperate to plan a mission including a sequence of selected tasks to be executed by one or more robots <NUM> to achieve a mission objective. The task allocation service is configured to determine a probability of success of the mission objective from selected tasks of the sequence of tasks.

More particularly, the probability of success service <NUM> is configured to access a library <NUM> of probabilities of events that impact success of tasks of the robot(s); and identifying, from the library, respective probabilities of those of the events that impact success of the selected tasks. The probability of success service is configured to determine probabilities of success of the selected tasks from the respective probabilities of those of the events that impact success of the selected tasks.

The probabilities of events in the library <NUM> may be expressed in any of a number of different manners, and may be expressed in the same manner or in different manners across the events. The probability of an event may be expressed in the library as a value (e.g., a number between <NUM> and <NUM>), or the probability may be expressed as a mathematical function form which the value may be determined. This may allow the probability of success service <NUM> to determine a probability of success of a selected task based on one or more conditions of selected task.

In some examples, the library <NUM> includes probabilities of events common to various tasks. These events may depend on the type of robot <NUM>. Some examples of suitable events include operation critical failure (any operation critical subsystem failure, e.g., flight critical failure), mission critical failure (any mission critical subsystem failure), collision, bingo fuel, sensor detection of target, sensor tracking target, and the like.

A more particular example of a suitable event that may be covered in the library <NUM> in the case of an aerial robot is midair collision, and the probability of this event may be expressed as a function to determine the probability of violating midair collision distances based on current traffic conditions. Traffic position and velocity may be modeled as a distribution around the measurement. As the robot distribution overlaps with known traffic distributions in forward predictions, the probability of midair collision violation increases. A well-clear violation is another example of a suitable event covered in the library. This is similar to midair collision but with larger boundaries. The well-clear boundaries may be defined by regulatory authority (e.g., FAA) to express how close vehicles are allowed to operate without reporting the incident to the regulatory authority.

Other examples of suitable events that may be covered in the library <NUM> include route execution, fuel burn, ground collision, threats and the like. Route execution (or flight path execution in the case of an aerial robot) may be expressed as a function to determine the probability that a robot <NUM> will not be able to track a given path, which may be described as a series of waypoints that define a route the robot will travel. There may be a number of issues that could arise when a robot is following a route, each factoring into the probability of success. One issue may include catastrophic failure, and each robot may have a probability of catastrophic failure per hour (e.g., <NUM>-<NUM> for passenger vehicle), which integrated over the route may be the probability of a catastrophic failure. Other issues may include lower-level hazards that affect mission performance such as failure of the MMS <NUM> or one or more sensors, which may also be integrated over the route.

Fuel burn as an event may be accurately estimated and checked. In some examples, fuel burn may also factor in weather variation such as wind and turbulence that could affect how much fuel burn is required to reach the end of a route. Over the route of the robot <NUM> with an appropriate weather model, the probability of failure due to fuel burn can be determined.

Ground collision may be an issue for a robot <NUM> operating on or close to the ground. When the robot is on or close to the ground there may be a number of issues that can cause ground collision, such as navigation accuracy, ground map accuracy, maneuvering limits, reaction time, visibility and the like. Each of issues may add to the risk that determines the probability of success of a task executing on or close to the ground.

In some scenarios such as some military scenarios, threats may reduce the probability of success of a task. The probability for a known threat may be expressed as a function to determine the probability of the known threat incapacitating the robot <NUM>. This function may include factors such as threat range, tracking, targeting, firing, interception probabilities and the like.

Again, the probability of success service <NUM> is configured to determine probabilities of success of the selected tasks from the respective probabilities of those of the events from the library <NUM> that impact success of the selected tasks (e.g., navigating route, enough fuel, finding target, prosecuting target). In some examples, the probability of success service is configured to determine the probability of success of a first selected task from the respective probabilities of those of the events that impact success of the first selected task. In mathematical notation, the probability of success for the first selected task T<NUM> may be expressed as follows: <MAT> where Em is an event that impacts success of the selected task.

Likewise, in some examples, the probability of success service <NUM> is configured to determine the probability of success for a selected task that follows a first selected task from the respective probabilities of those of the events that impact success of the selected task. For the selected task that follows the first selected task, the respective probabilities are conditional given success of an immediately preceding selected task in the sequence of selected tasks. The probability of success for the selected task Tn, n > <NUM> that follows the first selected task T<NUM> may be expressed as follows: <MAT> where <MAT>.

The task planner service <NUM> in some examples is configured to provide the probabilities of success of the selected tasks from the probability of success service <NUM> to the task allocation service <NUM>, which is configured to determine the probability of success of the mission objective from the probabilities of success of the selected tasks. The probability of success of the mission objective may be expressed as follows: <MAT> where Ok is objective k and Tq is a final task that achieves objective k. There may be multiple sequences of selected tasks to achieve an objective, each of which may have a final task that aggregate to accomplish an objective. The above expression assumes that every selected task in the sequence is completed to achieve the objective.

The task allocation service <NUM> is configured to determine a cost of the mission from the probability of success of the mission objective, which may be used to evaluate the mission. In this regard, the robot(s) <NUM> may be caused to execute the mission when the cost of the mission meets a predetermined selection criterion.

In some examples in which the mission has a plurality of mission objectives, the mission includes respective sequences of selected tasks for respective ones of the plurality of mission objectives. In some of these examples, the task allocation service <NUM> is configured to determine probabilities of success of the plurality of mission objectives, and determine the cost of the mission from the probabilities of success of the plurality of mission objectives. In some further examples, the task allocation service is configured to determine the cost of the mission as a product of the probabilities of success of the plurality of mission objectives, or as a weighted mean of the probabilities of success of the plurality of mission objectives.

In particular, for example, the probability of success of a mission M may the probability of all objectives being achieved, which may be expressed as follows: <MAT> According to a weighted approach, the cost of the mission may be determined as: <MAT> where Wk is a weight for P(Ok). In this expression, the cost of the mission is expressed as a sum of weighted probabilities rather than a product, which may more intuitively scale in the case an objective is not achievable.

In some examples in which the robot(s) <NUM> are caused to execute the mission, the task allocation service <NUM> is configured to construct a task graph in which the mission is modeled. The task graph is expressed as a directed graph and includes selected task nodes representing the selected tasks that are connected by edges representing transitions between the selected tasks. In some of these examples, the task allocation service is further configured to provide mission data including the task graph to the robot(s), and the task graph is accessible onboard the robot(s) to cause the robot(s) to execute the mission.

Returning to <FIG>, in some even further examples, the mission data is provided from the mission planning subsystem <NUM> to the mission execution subsystem <NUM> of the MMS <NUM> (directly or via the mission coordination subsystem <NUM>). The mission and in particular the selected tasks of the mission may then be executed through the mission execution subsystem using the mission data including the task graph. This may include the MMS configured to send one or more maneuver commands to the RMS <NUM> to control the robot to follow the maneuver commands and thereby execute maneuvers.

In some examples, the MMS <NUM> accounts for the possibility that the mission fails to execute exactly as planned because the environment or other robots <NUM> can experience unexpected and/or expected state changes. In some of these examples, the mission execution subsystem <NUM> periodically (or dynamically) checks a current predicted outcome of the mission plan throughout mission execution, and may call on one or more of the services <NUM> of the mission planning subsystem <NUM>. If the probability of success of the mission or one or more selected tasks falls below a predetermined threshold value (e.g., a predetermined success percentage), a re-plan may be initiated.

The mission is modeled in a task graph, which is a specific directed graph of tasks with defined transitions, assigned to one or more robots <NUM>, which if executed, will accomplish objectives intrinsic to those tasks. A task may be executed by the mission execution subsystem <NUM> to cause one or more robots to execute one or more maneuvers with specific parameters and capabilities, such as by way of one or more maneuver commands. A task may include internal logic represented through one or more state machines, and may therefore be deterministic. In some examples, the internal logic of a task is defined during design of the system <NUM>; and in some of these examples, the internal logic does not change during mission planning or execution.

A task may exhibit a list of available predecessors that serves to indicate possible predecessor-successor transition logic. During a mission, the mission execution subsystem <NUM> may cause the one or more robots <NUM> to execute maneuvers and thereby the mission task-by-task. If a task can be executed to its end, the mission execution subsystem may next call a task connected to "nominal completion" exit criteria of the task; otherwise, the mission execution subsystem may next call a task connected to specific non-nominal completion of the task. Nominal completion of a task means that a task completed as expected, with the exit criteria being the desired result that occurs in this case. The exit criteria may include, for example, arriving at a point, completing takeoff, completing landing, finishing a search pattern, finding an object being searched for, completing a map of an area, dropping a package at a delivery point, or the like. Non-nominal means that a task exited, but did not complete with its desired result, such as when an event occurs that results in the task terminating early. When the task connected to the specific non-nominal completion of the task is called, it may be because something has prevented the internal logic to flow along the nominal path and the execution ended up into one of the many predefined possible conclusions.

Predecessors and exit criteria for a task may be defined during its design of a task, and considered during mission planning. In this regard, MMS <NUM> may use the mission planning subsystem <NUM> to plan a mission in one or more phases. In one phase, a mission may be composed by a nominal sequence of selected tasks connected through nominal completion exit criteria and available predecessors. Task attributes such as routes may be added. In this regard, a task may be described by program code and may need one or more inputs to execute. These inputs are task attributes and they may at times relate to completion of the task. For example, if a task follows a waypoint route, then an attribute may be the waypoint route to follow. If the task is to search an area, the attribute may be the box defining the area to search. <FIG> illustrates an example of a task graph <NUM> for including selected task nodes <NUM> representing the selected tasks in a nominal sequence that are connected by edges <NUM> representing transitions between the selected tasks. As shown, sequences of edges connecting respective ones of the task nodes define paths in the task graph.

In another phase, specific behaviors may be assigned to failure modes of certain identified tasks. Local contingency events include contingency events that relate to specific task execution, and that are associated with specific tasks. One or more contingency paths may be added to create alternatives to the nominal sequence of selected tasks when one or more tasks the nominal sequence cannot be successfully executed. In some examples, a local contingency event may prevent a certain identified task from achieving its nominal completion exit criteria, and instead lead to a specific non-nominal completion of the task. Resolving a local contingency event may occur outside of the identified task by making use of one or more alternate tasks within the mission.

<FIG> illustrates the task graph <NUM> further including alternate task nodes <NUM> representing alternate tasks to be executed when a local contingency event occurs during execution of a selected task in the nominal sequence of selected tasks. For the selected task node and each alternate task node representing each alternate task, the task graph includes an edge <NUM> connecting the selected task node to the alternate task node that represents a transition from the selected task to the alternate task. For example, consider the "approach" task, and that the task cannot be successfully executed on a specified approach path when a strong tail wind (contingency event) occurs. The mission may be planned to include an alternative "hold" task to transition to a holding pattern, and/or a "transition up" task, to enable the robot <NUM> to use specific airport runways based on wind conditions.

In yet another phase, one or more alternate task graphs of alternate tasks assigned to global contingency events may be added. Global contingency events include contingency events that usually have nothing to do with the specific task execution and are instead related to one or more failures of the robot <NUM> (e.g., engine failure, navigation failure, link loss, battery failure). That is, global contingency events include contingency events that are not associated with a specific task. Global contingency events may also include contingency events in which a task fails execution in such a way that would normally produce a local contingency event, but the task graph does not otherwise include a contingency path for the local contingency event.

<FIG> illustrates three alternate task graphs <NUM>, <NUM>, <NUM> including respective sequences of alternate tasks nodes representing alternate tasks to be executed when a global contingency event occurs during execution of any one of the selected tasks in the nominal sequence. The alternate task graphs may be organized by levels of contingency. In this regard, a first alternate task graph <NUM> for a first global contingency event may call for the robot <NUM> to land immediately. A second alternate task graph <NUM> for a second global contingency event may call for the robot to land as soon as practical at a landing zone from which the robot took off. And a third alternate task graph <NUM> for a third global contingency event may call for the robot to land as soon as practical at a designated or selected landing zone.

<FIG> illustrates the task graph <NUM> connected to, for example, the third alternate task graph <NUM> for the third global contingency event. The selected task nodes <NUM> are associated with the third alternate task graph, shown for one of the selected task nodes as a dashed edge <NUM> that represents association of the selected task with the alternate task graph.

Returning to <FIG>, in some examples in which a mission is modeled in a task graph, tasks executable to cause a robot <NUM> or a type of robot to execute maneuvers (or idle in the case of idle tasks) may be designed, and a task library of tasks executable to cause the robot or type of robot to execute the maneuvers (or idle) may be developed. During mission planning, then, MMS <NUM> may use the mission planning subsystem <NUM> to plan a mission including a nominal sequence of selected tasks that are executable to cause one or more robots (of the type of robot) to execute maneuvers (and/or idle) to achieve a mission objective. The mission data provided by the mission planning subsystem to the mission execution subsystem <NUM> may include the task graph, and the mission execution subsystem may use the task graph and the task library to execute the mission. This may include the MMS configured to execute the selected tasks and send one or more maneuver commands (e.g., waypoints, steering commands) to the RMS <NUM> to control the robot to follow the maneuver commands and thereby execute the maneuvers.

<FIG> is a diagram of additional or alternative services <NUM> that may be implemented by one or more subsystems of the MMS <NUM>. As shown, the services may include a mission manager <NUM> service that may be implemented by the mission execution subsystem <NUM>. The mission manager service may be configured to identify a mission including a nominal sequence of selected tasks that are executable to cause one or more robots <NUM> to execute maneuvers to achieve a mission objective. In some examples, the mission is identified from mission data <NUM>.

The services <NUM> also include a task manager <NUM> service configured to access mission data <NUM> for the mission, and execute the selected tasks according to the mission data. In some examples, the task manager service is configured to determine a task graph <NUM> in which the mission is modeled, which in some examples may include the task manager service configured to access the task graph from the mission data. The task graph is expressed as a directed graph <NUM> and includes selected task nodes <NUM> representing the selected tasks that are connected by edges <NUM> representing transitions between the selected tasks. The task manager service is configured to execute the selected tasks using the task graph and a task library <NUM> of tasks <NUM> including the selected tasks 618A that are executable to cause the robot(s) to execute the maneuvers.

In some more particular examples, for a robot <NUM> of the robot(s), the task manager <NUM> service is configured to access the mission data <NUM> including the task graph <NUM> in which the mission is modeled. The task manager service is configured to traverse task graph including the selected task nodes <NUM>. And when a selected task node representing a selected task is visited, the task manager service is configured to call on the task library <NUM> to execute the selected task 618A and thereby cause the robot to execute the maneuver.

As suggested above, a mission contemplates the occurrence of various contingency events. These contingency events may include, for example, task-specific contingency events, local contingency events and/or global contingency events. A task-specific contingency event may be expressed in internal logic of a task <NUM>. A local contingency event may be expressed in the task graph <NUM>. A global contingency event may be expressed in an alternate task graph 608A.

For a task-specific contingency event, in some examples, a selected task in the nominal sequence causes the robot(s) <NUM> to execute a nominal maneuver, or an alternate maneuver when the task-specific contingency event occurs during execution of the nominal maneuver. In some of these examples, the task library <NUM> includes the selected task 618A executable to cause the robot(s) to execute the nominal maneuver, or the alternate maneuver when the task-specific contingency event occurs.

In the case of a local contingency event, in some examples, the mission further includes one or more alternate tasks to be executed when a local contingency event occurs during execution of a selected task in the nominal sequence. In some of these examples, the task graph <NUM> further includes one or more alternate task nodes <NUM> representing the one or more alternate tasks. The task graph also includes for the selected task node and each alternate task node representing each alternate task, an edge <NUM> connecting the selected task node to the alternate task node that represents a transition from the selected task to the alternate task. And further, the tasks <NUM> in the task library <NUM> further include the one or more alternate tasks 618B.

For a global contingency event, in some examples, the mission is associated with one or more alternate tasks to be executed when the global contingency event occurs during execution of any one of the selected tasks in the nominal sequence. In some of these examples, the selected task nodes <NUM> in the task graph <NUM> are associated with one or more alternate task graphs 608A of alternate task nodes <NUM> representing the one or more alternate tasks. This association may be represented for each selected task node and each alternate task graph by a dashed edge <NUM> connecting the selected task node to the alternate task graph. Similar before, the tasks <NUM> in the task library <NUM> also include these one or more alternate tasks 618B.

In some examples, the services <NUM> further include a contingency monitor <NUM> service, a contingency manager service <NUM> and/or a feasibility <NUM> service, which may be implemented by one or more subsystems of the MMS <NUM>, such as the mission planning subsystem <NUM> or the mission execution subsystem <NUM>. The contingency monitor service is configured to monitor for contingency events during the mission, and the contingency manager is configured to manage those contingency events. This may include the contingency monitor configured to determine at least one of a state of the robot <NUM>, a status of the robot, or a state of the environment, some or all of which may be reflected in or determined from input data. The state of the robot is or includes its position, orientation, and time derivatives of those variables; and often, the state is the position, velocity, orientation, and orientation rates of the robot. The status of the robot refers to other information the robot may report, such as actuator modes, flight modes, navigation mode, GPS lock, communication throughput, and the like. In some examples, at least some of the input data may be provided by or determined from data provided by various sensors onboard the robot, the RMS <NUM> and/or the control station <NUM>, which may interface with the MMS <NUM> using the interface subsystem <NUM>. The state of the environment may in some examples be provided by or determined from data provided by the situational awareness subsystem <NUM> of the MMS.

The contingency monitor <NUM> service is configured to detect a contingency event, and report the contingency event to the task manager <NUM> service, directly or via the contingency manager <NUM> service. That is, in some examples, the contingency monitor service is configured to detect a contingency event, and report the contingency event to the task manager <NUM>service. In other examples, the contingency monitor service is configured to detect a contingency event, and report the contingency event to the contingency manager, which is in turn configured to notify the task manager service.

For a local contingency event detected during execution of a selected task, in some examples, the task manager <NUM> service is configured to transition in the task graph <NUM> from the selected task node to the alternate task node <NUM> (on edge <NUM>), and call on the task library <NUM> to execute the alternate task 618B and thereby cause the robot to execute another maneuver (nominal for the alternate task). For a global contingency event detected during execution of any one of the selected tasks, in some examples, the task manager service is configured to transition to the alternate task graph 608A with one or more alternate task nodes <NUM> (and thereby transition to the one or more alternate task nodes), and call on the task library to execute the one or more alternate tasks 618B and thereby cause the robot to execute one or more other maneuvers.

According to some examples, events that are detectable by the contingency monitor <NUM> service may be may be classified into levels of contingency. In some examples, events may be classified into levels of contingency corresponding to the alternate task graphs <NUM>, <NUM>, <NUM> shown in <FIG> for some global contingency events. In other examples, the events may be classified into levels of contingency as follows:.

The levels of contingency may be ordered such as in order of priority (e.g., land immediately being the highest, followed by land as soon as possible, followed by and as soon as practical, and so forth).

In some examples, then, the task manager <NUM> service is configured to access mission data <NUM> for the mission including or associated with tasks that are executable to cause the robot <NUM> to execute maneuvers, and execute the tasks according to the mission data. As the tasks are executed, the contingency monitor <NUM> service is configured to monitor at least one of the robot or the environment, and detect an event during execution of a task of the tasks. The event may be or include, for example, a local contingency event or a global contingency event. The contingency manager <NUM> service is configured to map the event to a level of contingency of a plurality of levels of contingency, and report the level of contingency to the task manager service. In response, the task manager service is configured to transition from the task to another of the tasks according to the level of contingency, and execute the other of the tasks.

In some examples in which the plurality of levels of contingency are in order of priority, that order may be used when multiple events are detected by the contingency monitor <NUM> service. In this regard, in some examples, the contingency manager <NUM> service is configured to map the multiple events to the respective levels of contingency, and report to the task manager service <NUM>, the level of contingency that among the respective levels of contingency is higher (or highest) in the order of priority. The task manager service, then, is configured to transition from the task to the other of the tasks according to the higher level of contingency.

In some examples, the contingency monitor <NUM> service is configured to detect an event (another event) that the contingency manager <NUM> service is configured map to a non-maneuver action or procedure of the robot. The contingency manager service is configured cause the robot to perform the non-maneuver action or procedure. This may include the MMS <NUM> configured to send one or more commands to the RMS <NUM> to control the robot to perform the non-maneuver action or procedure.

The mission data <NUM> includes a task graph <NUM> in which the mission is modeled, and in which the edges <NUM>, <NUM>, <NUM> representing transitions between the tasks (represented by the task nodes <NUM>, <NUM>, <NUM>) are associated with levels of contingency. This association, then, may direct the transition from one task to another when the level of contingency is reported to the task manager service <NUM>.

<FIG> illustrates another example of a task graph <NUM> similar to the task graph <NUM> illustrated in <FIG>. The task graph <NUM> includes selected task nodes <NUM> representing the selected tasks in a nominal sequence that are connected by edges <NUM> representing transitions between the selected tasks. The task graph includes or is associated with alternate task nodes <NUM> representing alternate tasks to be executed when a contingency event (e.g., local contingency event, global contingency event) occurs during execution of a selected task in the nominal sequence of selected tasks. In the illustrated task graph, the edges are associated with levels of contingency for level <NUM>, level <NUM> and level <NUM> contingencies, as well as a level <NUM> representing a nominal transition between task nodes in a sequence of task nodes. In some examples, one or more of the edges may be associated with multiple levels of contingency so that the same transition occurs when any of the multiple levels of contingency are detected.

Each task node <NUM> may be connected to one or more edges that represent transition from the task node (egress), and these one or more edges may cover all of the levels of contingency. For some tasks, however, the robot may be configured to ignore one or more levels of contingency. The task graph <NUM> may further include additional edges and nodes to cover these situations. As shown, for example, the task graph includes additional edges <NUM> associated with one or more levels of contingency, that connect task nodes to a ground node <NUM> representing "do nothing. " When a contingency event with one of these levels of contingency is detected, the task manager service <NUM> may interpret the contingency event as a default global contingency event that may be predefined for the respective task nodes.

Returning to <FIG>, again, the task manager service <NUM> may in some examples be configured to traverse the task graph <NUM> including the task nodes <NUM>, <NUM>, <NUM>. When a task node representing the task is visited, the task manager service is configured to call on the task library <NUM> of the tasks <NUM> that are executable to cause the robot to execute the maneuvers. In some of these examples, the task manager service is configured to transition from the task node on an edge that connects the task node to another of the task nodes representing the other of the tasks; and in some of these examples, the edge is associated with the level of contingency.

As also shown in <FIG>, the feasibility <NUM> service, which that may be implemented by one or more subsystems of the MMS <NUM>, such as the mission planning subsystem <NUM> or the mission execution subsystem <NUM>. The feasibility service may be configured to determine feasibility of a path in the task graph <NUM> from the predicted performance of the robot for the path, and outputting an indication of the feasibility of the path. The mission planning subsystem or the mission execution subsystem may continue uninterrupted when the path is feasible. When the path is infeasible, the indication of the feasibility may indicate the path is infeasible, which may cause any of a number of different responses. These responses may include the mission planning subsystem configured to replan the path, or the mission execution subsystem reacting to the indication as an event such as a contingency event. Additionally or alternatively, for example, the indication may prompt an operator to intervene.

More particularly, in some examples, the feasibility <NUM> service is configured to determine at least one of a state of the robot <NUM>, a status of the robot, or a state of the environment, some or all of which may be reflected in or determined from input data. In some examples, at least some of the input data may be provided by or determined from data provided by various sensors onboard the robot, the RMS <NUM> and/or the control station <NUM>, which may interface with the MMS <NUM> using the interface subsystem <NUM>. The state of the environment may in some examples be provided by or determined from data provided by the situational awareness subsystem <NUM> of the MMS.

The feasibility <NUM> service is configured to access the task graph <NUM> in which the mission is modeled, and determine a predicted performance of the robot <NUM> for a path such as a nominal path or an active in the task graph using the state of the robot, the status of the robot, and/or the state of the environment, using the task graph, and starting at an unexecuted one of the tasks. In some examples this includes the feasibility service configured to determine a prediction of at least one of a route traveled by the robot, a time taken by the robot, an end state of the robot, an end status of the robot, or a probability of success.

The predicted performance of the robot <NUM> is determined starting at an unexecuted one of the tasks. In this regard, the predicted performance and thus the feasibility of the path may be determined going forward in time from the unexecuted one of the tasks. Before execution of a mission, the unexecuted one of the tasks may be any of the tasks of the mission. As the mission is executed, the unexecuted one of the tasks may be any task subsequent to the task currently being executed, such as the next task in sequence. The path may be a nominal path or another path that may be an active path at the time. The feasibility <NUM> service may therefore repeatedly determine feasibility of the path particularly for those tasks of the mission that have not yet been executed.

The feasibility <NUM> service is configured to determine feasibility of the path from the predicted performance of the robot <NUM>, and output an indication of the feasibility of the path. In some examples, the feasibility service configured to determine the feasibility of the path based on a comparison of the predicted performance and a threshold performance. In some examples, this threshold performance defines a boundary between a feasible path and an infeasible path.

<FIG> more particularly illustrates a feasibility <NUM> service that in some examples may correspond to the feasibility <NUM> service. As shown, the feasibility <NUM> service includes one or more sub-services (each an individual service) such as a duplicate task manager <NUM> service, a duplicate task library <NUM> of duplicate tasks <NUM>, and a duplicate contingency monitor <NUM> service.

In some examples, the feasibility <NUM> service is configured to determine the predicted performance of the robot <NUM> using the duplicate task library <NUM> of duplicate tasks <NUM> executable to perform a simulation of the tasks. In some further examples, this includes the duplicate task manager <NUM> service configured to traverse the task graph <NUM> starting at a task node representing the unexecuted one of the tasks. And when the task node representing a task is visited, the duplicate task manager service is configured to call on the duplicate task library to execute a duplicate task to perform the simulation of the task, and output the predicted performance of the robot for the task.

In some examples, the duplicate contingency monitor <NUM> service is configured to monitor at least one of a model <NUM> of the robot <NUM> or a model <NUM> of the environment, and predict an event such as a contingency event during the simulation of the task by the duplicate task <NUM> of the duplicate task library <NUM>. The model <NUM> of the robot may depend on the state of the robot and/or the status of the robot, and the model <NUM> of the environment may depend on the state of the environment. In some of these examples, the duplicate task manager <NUM> service is configured to transition from the task node representing the task to another task node representing another task in response to the event. And the duplicate task manager service is configured to call on the duplicate task library to execute another duplicate task to perform the simulation of the other task, and output the predicted performance of the robot for the other task.

Briefly returning to <FIG>, the feasibility <NUM> service may therefore use simulation to predict performance of the robot <NUM>. In other examples, the feasibility service may use more deterministic predictions of performance of the robot. These predictions may include those deterministic from algorithms with low or lower computational requirements (e.g., lower than the simulation), although some predictions may require more complicated algorithms.

<FIG> more particularly illustrates another feasibility <NUM> service that in some examples may correspond to the feasibility <NUM> service. As shown, the feasibility <NUM> service includes one or more sub-services (each an individual service) such as a duplicate task manager <NUM> service and task predictors <NUM>.

In some examples, the feasibility <NUM> service is configured to determine the predicted performance of the robot <NUM> using the task predictors <NUM> that are executable to perform a prediction of performance and thereby determine the predicted performance of the robot for respective ones of the tasks. In particular, for example, the duplicate task manager <NUM> service is configured to traverse the task graph <NUM> starting at a task node representing the unexecuted one of the tasks. When the task node is visited, the duplicate task manager service is configured to execute a respective one of the task predictors to perform the prediction of performance and thereby determine the predicted performance of the robot for the unexecuted one of the tasks. In some examples, the task predictors perform predictions using the state of the robot, the status of the robot, and/or the state of the environment.

In some examples of the feasibility <NUM>, <NUM> service in either <FIG>, sequences of the edges <NUM>, <NUM>, <NUM> connecting respective ones of the task nodes <NUM>, <NUM>, <NUM> define paths in the task graph <NUM> (directed graph <NUM>). The duplicate task manager <NUM>, <NUM> service is configured to traverse the task graph on a nominal one of the paths (e.g., selected task nodes <NUM> connected by edges <NUM>). In some of these examples, the feasibility <NUM> service is configured to determine feasibility of the nominal one of the paths. Also in some of these examples, the duplicate task manager is further configured to determine an alternate one of the paths in response to the nominal one of the paths being determined infeasible. The indication of the feasibility of the path may then include the alternate one of the paths as a proposed feasible path, which may be traversed by the task manager <NUM> service.

<FIG>, <FIG> are flowcharts illustrating various steps in a method <NUM> of causing one or more robots <NUM> to execute a mission having one or more mission objectives, according to examples not currently claimed. As shown at block <NUM> of <FIG>, the method includes planning the mission including a sequence of selected tasks to be executed by the one or more robots to achieve a mission objective. The method includes determining a probability of success of the mission objective from selected tasks of the sequence of tasks, as shown at block <NUM>.

More particularly, determining the probability of success of the mission objective includes accessing a library <NUM> of probabilities of events that impact success of tasks of the one or more robots; and identifying, from the library, respective probabilities of those of the events that impact success of the selected tasks, as shown at blocks <NUM> and <NUM>. Determining the probability of success of the mission objective also includes determining probabilities of success of the selected tasks from the respective probabilities of those of the events that impact success of the selected tasks, as shown at block <NUM>. The probability of success of the mission objective is then determined from the probabilities of success of the selected tasks, as shown at block <NUM>.

Turning to <FIG>, for a first selected task in the sequence of selected tasks, determining the probabilities of success at block <NUM> includes determining a probability of success of the first selected task from the respective probabilities of those of the events that impact success of the first selected task, as shown at block 1010A. Then for a selected task that follows a first selected task in the sequence of selected tasks, a probability of success of the selected task is determined from the respective probabilities of those of the events that impact success of the selected task, as shown at block 1010B. The respective probabilities here are conditional given success of an immediately preceding selected task in the sequence of selected tasks.

Returning to <FIG>, the method <NUM> also includes determining a cost of the mission from the probability of success of the mission objective, as shown at block <NUM>. In some examples in which the mission has a plurality of mission objectives, the mission includes respective sequences of selected tasks for respective ones of the plurality of mission objectives. In some of these examples, probabilities of success of the plurality of mission objectives are determined at block <NUM>, and the cost of the mission is determined at block <NUM> from the probabilities of success of the plurality of mission objectives. In some further examples, the cost of the mission is determined at block <NUM> as a product of the probabilities of success of the plurality of mission objectives, or as a weighted mean of the probabilities of success of the plurality of mission objectives.

The method <NUM> further includes causing the one or more robots <NUM> to execute the mission when the cost of the mission meets a predetermined selection criterion, as shown at block <NUM>. In some examples, this includes constructing a task graph in which the mission is modeled, the task graph expressed as a directed graph and including selected task nodes representing the selected tasks that are connected by edges representing transitions between the selected tasks, as shown at block 1016A of <FIG>. Then as shown at block 1016B, mission data including the task graph is provided to the one or more robots, where the task graph is accessible onboard the one or more robots to cause the one or more robots to execute the mission.

<FIG>, <FIG> are flowcharts illustrating various steps in a method <NUM> of causing one or more robots <NUM> to execute a mission, according to examples not presently claimed but that provide a useful introduction to the currently claimed subject matter. The method includes identifying the mission including a nominal sequence of selected tasks that are executable to cause the one or more robots to execute maneuvers to achieve a mission objective, as shown at block <NUM>. The method includes determining a task graph <NUM> in which the mission is modeled, as shown at block <NUM>. The task graph is expressed as a directed graph <NUM> and includes selected task nodes <NUM> representing the selected tasks that are connected by edges <NUM> representing transitions between the selected tasks. The method also includes causing the one or more robots to execute the mission using the task graph and a task library <NUM> of tasks <NUM> including a selected task 618A executable to cause the one or more robots to execute a maneuver, as shown at block <NUM>.

In some examples, the selected task 618A in the nominal sequence of the selected tasks causes the one or more robots <NUM> to execute a nominal maneuver, or an alternate maneuver when a task-specific contingency event occurs during execution of the nominal maneuver. In some of these examples, the one or more robots are caused to execute the mission at block <NUM> using the task library <NUM> that includes the selected task 618A executable to cause the one or more robots to execute the nominal maneuver, or the alternate maneuver when the task-specific contingency event occurs.

In some examples, the mission further includes one or more alternate tasks to be executed when a contingency event (e.g., local contingency event) occurs during execution of the selected task 618A in the nominal sequence of selected tasks. In some of these examples, the task graph <NUM> determined at block <NUM> further includes one or more alternate task nodes <NUM> representing the one or more alternate tasks. The task graph may also include for the selected task node and each alternate task node representing each alternate task, an edge <NUM> connecting the selected task node to the alternate task node that represents a transition from the selected task to the alternate task.

In some examples, the mission is associated with one or more alternate tasks to be executed when a contingency event (e.g., global contingency event) occurs during execution of any one of the selected tasks in the nominal sequence of selected tasks. In some of these examples, determining the task graph <NUM> at block <NUM> includes determining an alternate task graph 608A including one or more alternate task nodes <NUM> representing the one or more alternate tasks, each selected task node associated with the alternate task graph.

In some examples, causing the one or more robots <NUM> to execute the mission at block <NUM> includes for a robot of the one or more robots, accessing mission data <NUM> including the task graph <NUM> in which the mission is modeled, as shown at block <NUM>. The task graph including the selected task nodes <NUM> is traversed, as shown at block <NUM>. And when a selected task node representing the selected task 618A is visited, the task library <NUM> is called on to execute the selected task and thereby cause the robot to execute the maneuver, as shown at block <NUM>.

Turning to <FIG>, in some examples, the mission further includes an alternate task to be executed when a contingency event (e.g., local contingency event) occurs during execution of the selected task, the task graph <NUM> further includes an alternate task node representing the alternate task, and the tasks <NUM> in the task library further include the alternate task 618B. In some of these examples, the method <NUM> further includes for the robot <NUM> of the one or more robot, detecting occurrence of the contingency event during execution of the selected task, as shown at block <NUM>. The method includes transitioning in the task graph from the selected task node to the alternate task node, and calling on the task library <NUM> to execute the alternate task and thereby cause the robot to execute another maneuver, as shown at blocks <NUM> and <NUM>.

Now to <FIG>, in some examples, the mission is associated with an alternate task to be executed when a contingency event (e.g., global contingency event) occurs during execution any one of the selected tasks, and the task graph <NUM> is associated an alternate task graph 608A including one or more alternate task nodes representing the one or more alternate tasks. The tasks <NUM> in the task library also include the one or more alternate tasks 618B. In some of these examples, the method <NUM> further includes for the robot <NUM> of the one or more robot, detecting occurrence of the contingency event during execution of any one of the selected tasks, as shown at block <NUM>. The method includes transitioning to the alternate task graph, and calling on the task library <NUM> to execute the one or more alternate tasks and thereby cause the robot to execute one or more other maneuvers, as shown at blocks <NUM> and <NUM>.

<FIG>, <FIG>, <FIG>, <FIG> and 12F are flowcharts illustrating various steps in a method <NUM> of causing one or more robots <NUM> to execute a mission in an environment, according to examples in accordance with the current claims. The method includes, for a robot of the one or more robots, accessing mission data <NUM> for the mission including or associated with tasks that are executable to cause the robot to execute maneuvers, and executing the tasks according to the mission data to cause the robot to execute the maneuvers, as shown at blocks <NUM> and <NUM> of <FIG>. The method includes, as the tasks are executed, monitoring at least one of the robot or the environment, and detecting an event based on the monitoring, during execution of a task of the tasks, as shown at blocks <NUM> and <NUM>. The method includes mapping the event to a level of contingency of a plurality of levels of contingency, as shown at block <NUM>. The method includes transitioning from the task to another of the tasks according to the level of contingency, and executing the other of the tasks, as shown at blocks <NUM> and <NUM>.

In some examples, the event is a local contingency event that is detectable during execution of only certain tasks. In some of these examples, detecting the event at block <NUM> includes detecting the local contingency event during execution of the task that is one of the certain tasks.

In some examples, the event is a global contingency event that is detectable during execution of any of the tasks. In some of these examples, detecting the event at block <NUM> includes detecting the global contingency event during execution of the task that is any one of the tasks.

In some examples, the plurality of levels of contingency are in order of priority. In some of these examples, detecting the event at block <NUM> includes detecting multiple events including the event, as shown at block 1208A of <FIG>. Mapping the event at block <NUM> includes mapping the multiple events to the respective levels of contingency, as shown at block 1210A. And transitioning from the task to the other of the tasks at block <NUM> includes transitioning from the task to another of the tasks according to the level of contingency that among the respective levels of contingency is higher in the order of priority, as shown at block 1212A.

In some examples, the method <NUM> further includes for the robot of the one or more robots, detecting another event based on the monitoring, as shown at block <NUM> of <FIG>. Also in some of these examples, the method includes mapping the other event to a non-maneuver action or procedure of the robot, and causing the robot to perform the non-maneuver action or procedure, as shown at blocks <NUM> and <NUM>.

The mission data <NUM> includes a task graph <NUM> in which the mission is modeled. The task graph is expressed as a directed graph <NUM> and includes task nodes <NUM>, <NUM>, <NUM> representing the tasks that are connected by edges <NUM>, <NUM>, <NUM> representing transitions between the tasks. Executing the tasks at block <NUM> includes traversing the task graph including the task nodes, as shown at block <NUM> of <FIG>. And the method <NUM> includes, when a task node representing the task is visited, calling on a task library <NUM> of the tasks <NUM> that are executable to cause the robot to execute the maneuvers, as shown at block <NUM>. The task library called on to execute the task and thereby cause the robot to execute a maneuver.

Transitioning from the task to the other of the tasks at block <NUM> includes transitioning from the task node <NUM>, <NUM><NUM><NUM>, <NUM> on an edge <NUM>, <NUM>, <NUM> that connects the task node to another of the task nodes representing the other of the tasks, as shown at block <NUM> of <FIG>. The edge <NUM>, <NUM>, <NUM> is associated with the level of contingency.

<FIG>, <FIG>, <FIG> are flowcharts illustrating various steps in a method <NUM> of causing one or more robots <NUM> to execute a mission in an environment, according to examples not currently claimed. The method includes, for a robot of the one or more robots, causing the robot to execute the mission that includes tasks that are executable to cause the robot to execute maneuvers, and determining at least one of a state of the robot, a status of the robot, or a state of the environment, as shown at blocks <NUM> and <NUM> of <FIG>.

The method <NUM> includes accessing a task graph <NUM> in which the mission is modeled, as shown at block <NUM>. The task graph expressed is as a directed graph and including task nodes representing the tasks that are connected by edges representing transitions between the tasks, sequences of edges connecting respective ones of the task nodes defining paths in the task graph. The method includes determining a predicted performance of the robot <NUM> for a path in the task graph using the at least one of the state of the robot, the status of the robot, or the state of the environment, using the task graph, and starting at an unexecuted one of the tasks, as shown at block <NUM>. And the method includes determining feasibility of the path from the predicted performance of the robot, and outputting an indication of the feasibility of the path, as shown at blocks <NUM> and <NUM>.

In some examples, determining the predicted performance at block <NUM> includes determining a prediction of at least one of a route traveled by the robot <NUM>, a time taken by the robot, an end state of the robot, an end status of the robot, or a probability of success.

In some examples, determining the feasibility of the path at block <NUM> includes determining the feasibility of the path based on a comparison of the predicted performance and a threshold performance.

In some examples, determining the predicted performance of the robot <NUM> at block <NUM> includes determining the predicted performance further using a duplicate task library <NUM> of duplicate tasks <NUM> executable to perform a simulation of the tasks. In some further examples, determining the predicted performance of the robot <NUM> includes traversing the task graph <NUM> starting at a task node representing the unexecuted one of the tasks, as shown at block <NUM> of <FIG>. And when the task node representing a task is visited, the duplicate task library is called on to execute a duplicate task to perform the simulation of the task, and output the predicted performance of the robot for the task, as shown at block <NUM>.

In some examples, determining the predicted performance at block <NUM> further includes monitoring at least one of a model <NUM> of the robot <NUM> or a model <NUM> of the environment, and predicting an event during the simulation of the task, as shown at blocks <NUM> and <NUM> of <FIG>. The method includes transitioning from the task node representing the task to another task node representing another task in response to the event, as shown at block <NUM>. And the method includes calling on the duplicate task library <NUM> to execute the another duplicate task <NUM> to perform the simulation of the other task, and output the predicted performance of the robot for the other task, as shown at block <NUM>.

In some examples, traversing the task graph at block <NUM> includes traversing the task graph <NUM> on a nominal one of the paths. Also in some of these examples, determining feasibility of the path at block <NUM> includes determining feasibility of the nominal one of the paths. And the method further includes determining an alternate one of the paths in response to the nominal one of the paths being determined infeasible as shown at block <NUM> of <FIG>. The indication of the feasibility of the path in some of these examples includes the alternate one of the paths as a proposed feasible path.

Briefly returning to <FIG>, in some examples, the predicted performance of the robot <NUM> is determined at block <NUM> further using task predictors that are executable to perform a prediction of performance and thereby determine the predicted performance of the robot for respective ones of the tasks. In some further examples, determining the predicted performance of the robot includes traversing the task graph <NUM> starting at a task node representing the unexecuted one of the tasks, as shown at block <NUM> of <FIG>. And when the task node is visited, a respective one of the task predictors is executed to perform the prediction of performance and thereby determine the predicted performance of the robot for the unexecuted one of the tasks, as shown at block <NUM>.

In some even further examples, and traversing the task graph <NUM> at block <NUM> includes traversing the task graph on a nominal one of the paths. In some of these examples, determining feasibility of the path at block <NUM> includes determining feasibility of the nominal one of the paths. The method <NUM> further includes determining an alternate one of the paths in response to the nominal one of the paths being determined infeasible, as shown at block <NUM> of <FIG>. Similar to above, the indication of the feasibility of the path in some of these examples includes the alternate one of the paths as a proposed feasible path.

The the MMS <NUM> and its subsystems including the interface subsystem <NUM>, situational awareness subsystem <NUM>, mission planning subsystem <NUM>, mission coordination subsystem <NUM> and mission execution subsystem <NUM> may be implemented by various means. Means for implementing the MMS and its subsystems may include hardware, alone or under direction of one or more computer programs from a computer-readable storage medium. In some examples, one or more apparatuses may be configured to function as or otherwise implement the MMS and its subsystems shown and described herein. In examples involving more than one apparatus, the respective apparatuses may be connected to or otherwise in communication with one another in a number of different manners, such as directly or indirectly via a wired or wireless network or the like.

<FIG> illustrates an exemplary apparatus <NUM> that may comprise, include or be embodied in one or more fixed or portable electronic devices. The apparatus may include one or more of each of a number of components such as, for example, processing circuitry <NUM> (e.g., processor unit) connected to a memory <NUM> (e.g., storage device).

The processing circuitry <NUM> may be composed of one or more processors alone or in combination with one or more memories. The processing circuitry may be any piece of computer hardware that is capable of processing information such as, for example, data, computer programs and/or other suitable electronic information. The processing circuitry is composed of a collection of electronic circuits some of which may be packaged as an integrated circuit or multiple interconnected integrated circuits (an integrated circuit at times more commonly referred to as a "chip"). The processing circuitry may be configured to execute computer programs, which may be stored onboard the processing circuitry or otherwise stored in the memory <NUM> (of the same or another apparatus).

The processing circuitry <NUM> may be a number of processors, a multi-core processor or some other type of processor, depending on the particular implementation. Further, the processing circuitry may be implemented using a number of heterogeneous processor systems in which a main processor is present with one or more secondary processors on a single chip. As another illustrative example, the processing circuitry may be a symmetric multi-processor system containing multiple processors of the same type. In yet another example, the processing circuitry may be embodied as or otherwise include one or more ASICs, FPGAs or the like. Thus, although the processing circuitry may be capable of executing a computer program to perform one or more functions, the processing circuitry of various examples may be capable of performing one or more functions without the aid of a computer program. In either instance, the processing circuitry may be appropriately programmed to perform functions or operations according to examples of the present disclosure.

The memory <NUM> may be any piece of computer hardware that is capable of storing information such as, for example, data, computer programs (e.g., computer-readable program code <NUM>) and/or other suitable information either on a temporary basis and/or a permanent basis. The memory may include volatile and/or non-volatile memory, and may be fixed or removable. Examples of suitable memory include random access memory (RAM), read-only memory (ROM), a hard drive, a flash memory, a thumb drive, a removable computer diskette, an optical disk, a magnetic tape or some combination of the above. Optical disks may include compact disk - read only memory (CD-ROM), compact disk - read/write (CD-R/W), DVD or the like. In various instances, the memory may be referred to as a computer-readable storage medium. The computer-readable storage medium is a non-transitory device capable of storing information, and is distinguishable from computer-readable transmission media such as electronic transitory signals capable of carrying information from one location to another. Computer-readable medium as described herein may refer to a computer-readable storage medium or computer-readable transmission medium.

In addition to the memory <NUM>, the processing circuitry <NUM> may also be connected to one or more interfaces for displaying, transmitting and/or receiving information. The interfaces may include a communications interface <NUM> (e.g., communications unit) and/or one or more user interfaces. The communications interface may be configured to transmit and/or receive information, such as to and/or from other apparatus(es), network(s) or the like. The communications interface may be configured to transmit and/or receive information by physical (wired) and/or wireless communications links. Examples of suitable communication interfaces include a network interface controller (NIC), wireless NIC (WNIC) or the like.

The user interfaces may include a display <NUM> and/or one or more user input interfaces <NUM> (e.g., input/output unit). The display may be configured to present or otherwise display information to a user, suitable examples of which include a liquid crystal display (LCD), light-emitting diode display (LED), plasma display panel (PDP) or the like. The user input interfaces may be wired or wireless, and may be configured to receive information from a user into the apparatus, such as for processing, storage and/or display. Suitable examples of user input interfaces include a microphone, image or video capture device, keyboard or keypad, joystick, touch-sensitive surface (separate from or integrated into a touchscreen), biometric sensor or the like. The user interfaces may further include one or more interfaces for communicating with peripherals such as printers, scanners or the like.

As indicated above, program code instructions may be stored in memory, and executed by processing circuitry that is thereby programmed, to implement functions of the systems, subsystems, tools and their respective elements described herein. As will be appreciated, any suitable program code instructions may be loaded onto a computer or other programmable apparatus from a computer-readable storage medium to produce a particular machine, such that the particular machine becomes a means for implementing the functions specified herein. These program code instructions may also be stored in a computer-readable storage medium that can direct a computer, a processing circuitry or other programmable apparatus to function in a particular manner to thereby generate a particular machine or particular article of manufacture. The instructions stored in the computer-readable storage medium may produce an article of manufacture, where the article of manufacture becomes a means for implementing functions described herein. The program code instructions may be retrieved from a computer-readable storage medium and loaded into a computer, processing circuitry or other programmable apparatus to configure the computer, processing circuitry or other programmable apparatus to execute operations to be performed on or by the computer, processing circuitry or other programmable apparatus.

Retrieval, loading and execution of the program code instructions may be performed sequentially such that one instruction is retrieved, loaded and executed at a time. In some examples, retrieval, loading and/or execution may be performed in parallel such that multiple instructions are retrieved, loaded, and/or executed together. Execution of the program code instructions may produce a computer-implemented process such that the instructions executed by the computer, processing circuitry or other programmable apparatus provide operations for implementing functions described herein.

Claim 1:
A method (<NUM>), optionally a computer-implemented method, of causing one or more robots (<NUM>) to execute a mission in an environment, the method comprising:
for a robot of the one or more robots, accessing (<NUM>) mission data for the mission including or associated with tasks (<NUM>) that are executable to cause the robot to execute maneuvers;
executing (<NUM>) the tasks according to the mission data to cause the robot to execute the maneuvers and, as the tasks are executed, monitoring (<NUM>) at least one of the robot or the environment;
detecting (<NUM>) an event based on the monitoring, during execution of a task of the tasks;
mapping (<NUM>) the event to a level of contingency of a plurality of levels of contingency;
transitioning (<NUM>) from the task to another of the tasks according to the level of contingency; and
executing (<NUM>) the other of the tasks; and
wherein:
the mission data includes a task graph (<NUM>) in which the mission is modeled, the task graph expressed as a directed graph (<NUM>) and including task nodes (<NUM>) representing the tasks that are connected by edges (<NUM>) representing transitions between the tasks;
executing the tasks includes traversing (<NUM>) the task graph including the task nodes;
when a task node representing the task is visited, calling (<NUM>) on a task library (<NUM>) of the tasks that are executable to cause the robot to execute the maneuvers, thereby causing the robot to execute a maneuver; and
transitioning from the task to the other of the tasks includes transitioning (<NUM>) from the task node on an edge (<NUM>, <NUM>, <NUM>) that connects the task node to another of the task nodes representing the other of the tasks, the edge associated with the level of contingency.