Patent Description:
Listening and responding to air traffic command messages can place significant cognitive stress on pilots, particularly in high-traffic areas (such as terminal areas) where the pilot must quickly and accurately parse numerous messages - many of which may not apply to that aircraft. This cognitive load can cause a pilot to make mistakes when processing and interpreting important air traffic voice messages and lead to costly delays (e.g., missed approach). The inherent risks and costs involved in air traffic routing and control make it imperative that air traffic commands are quickly, accurately, and safely processed and acted on. New systems and methods for reducing pilot workload can aid the advancement toward single pilot and autonomous flight operations.

<CIT> discloses a system and method for rendering an aircraft cockpit display for use with ATC conditional clearance instructions.

Hence, it is desirable to provide a system for automating the response to air traffic command messages. Furthermore, other desirable features and characteristics of the present invention will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.

The present invention relates to an air traffic command automation system on an aerial vehicle for automating the process of listening to, interpreting, adhering to, and responding to air traffic commands according to independent claim <NUM>.

The subject matter described herein discloses a system for automating an air traffic command interface in an aerial vehicle in a manner that allows the pilot or operator of the aerial vehicle to easily and intuitively monitor and, if necessary, change the aircraft's response. The system provided herein can allow for automated responses to air traffic voice messages to be performed in a demonstrably safe manner. The system provided herein can allow for the presentation of a proposed automated response to the flight crew to be presented in an intuitive manner (e.g., so that a pilot may be able to quickly and accurately interact with it). The system provided herein can be configured to interface with current aircraft control technology (e.g., to reduce the cost of installation).

The system provided herein can receive air traffic control (ATC) command messages in the form of voice messages or Controller-Pilot Data Link Communication (CPDLC) messages, interpret them, and generate a sequence of "actions" that are comparable to the actions that would be taken by a pilot (e.g., setting the altitude, changing speed, etc.) that an aircraft can undertake to accomplish tasks assigned via the ATC command message. By decomposing complex tasks into simple actions, apparatus, systems, techniques and articles provided herein can interface with an existing autopilot computer to provide automated control of an aircraft.

Decomposing complex tasks into simple actions may also allow for the intuitive presentation of the plan to the pilot for approval or monitoring. In generating the plan, apparatus, systems, techniques and articles provided herein can check the received command for coherence (e.g., a "descend" command from ATC should provide an altitude that is lower than the aircrafts current altitude), feasibility (e.g., a command to change the speed of the aircraft should provide a value that is within the operating speeds of the aircraft), and safety (e.g., simultaneous speed and altitude changes must not cause the aircraft to stall).

The system provided herein can provide an architecture that is both configurable and expandable. The system provided herein can be configured to present a generated plan of actions to the pilot for approval before enacting it, or it can be configured to automatically relay and enact all or part of the plan without the need for pilot approval (e.g., changes to internal states such as the ATIS code or altimeter setting could be set automatically, while changes to the physical state of the aircraft such as speed or altitude could require pilot approval). The decomposition of complex tasks into a sequence of simple actions provided by apparatus, systems, techniques and articles described herein can allow new tasks to be added simply by creating a new "recipe" for that task that uses already existing components - the coherence, feasibility, and safety checks incorporated in the simpler tasks are automatically included in the new complex task.

The system provided herein can provide a relatively low-cost solution for automating a significant portion of an aircraft's interactions with air traffic command messages. Doing so can reduce the cognitive load on pilots, providing faster and more reliable responses to ATC commands. This, in turn, can reduce the costs incurred by mistakes and delays in aircraft routing and control.

<FIG> is a block diagram depicting an example operating environment <NUM> for an example air traffic command automation system <NUM>. As a vehicle <NUM>, such as a manned aircraft or an unmanned aerial system (UAS) is traveling, the vehicle <NUM> may receive an air traffic command message (e.g., ATC voice command message / CPDLC message) <NUM> from a control entity, such as air traffic control <NUM>. Listening and responding to ATC command messages can place a significant cognitive stress on pilots or operators (e.g., in the case of a UAS), particularly in high-traffic areas (such as terminal areas) where the pilot/operator must quickly and accurately parse numerous messages - many of which may not apply to that aircraft. The cognitive load has the potential to cause costly delays or mistakes by the aircraft's pilot/operator in the processing and interpretation of important ATC command messages. The inherent risks and costs involved in air traffic routing and control make it imperative that ATC command messages are quickly, accurately, and safely processed and acted on.

To reduce pilot/operator workload, the air traffic command automation system <NUM> is provided in the vehicle <NUM>. The air traffic command automation system <NUM> is configured to automatically comply with and respond to air traffic commands in a quick, accurate and safe manner.

<FIG> is a diagram depicting an example air traffic command automation system <NUM> for automating the process of listening to, interpreting, adhering to, and responding to air traffic command messages. The example air traffic command automation system <NUM> is integrated into a vehicle such as an aircraft, to interface with voice or CPDLC services, the autopilot, the pilot's interface, and other software components (such as a detect and avoid system). The example air traffic command automation system <NUM> includes an automation executive <NUM> for listening to, interpreting, and responding to the air traffic command messages (e.g., receive CPDLC messages and process them to determine the appropriate plan of action for the aircraft) and a mission and safety executive <NUM> for implementing additional safety protocols before implementing tasks required by the air traffic command message (e.g., to interface with the flight crew and software systems on the aircraft to relay and, if appropriate, enact the plan).

The automation executive <NUM> may use an automated planning technique called Hierarchical Task Networks to decompose the incoming task(s) into a sequence of primitive actions. Each task may correspond to a possible CPDLC uplink message, and each action may correspond to a state change that can be made by the aircraft. A plan is then a mapping from an incoming task to one or more actions, in order; this mapping can make use of other simpler tasks in the decomposition of a complex task. Once a plan is generated for the incoming task(s), the actions in that plan are checked to ensure that their interactions meet specified safety conditions.

The mission and safety executive <NUM> is configured to take the plan that is output by the automation executive <NUM> and determine which actions in the plan should be enacted or passed to the pilot for approval. It also incorporates information from other software systems (e.g., Detect and Avoid) to ensure the safe flight of the aircraft, and passes appropriate inputs to the autopilot.

The example air traffic command automation system <NUM> includes a controller that is configured to implement the automation executive <NUM> and the mission and safety executive <NUM>. The controller includes at least one processor and a computer-readable storage device or media encoded with programming instructions for configuring the controller. The processor may be any custom-made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an auxiliary processor among several processors associated with the controller, a semiconductor-based microprocessor (in the form of a microchip or chip set), any combination thereof, or generally any device for executing instructions.

The computer readable storage device or media may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example. KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor is powered down. The computer-readable storage device or media may be implemented using any of a number of known memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or any other electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable programming instructions, used by the controller. In one example, each of the automation executive <NUM> and the mission and safety executive <NUM> includes one or more processors configured by programming instructions on non-transient computer readable media.

The example automation executive <NUM> is configured to receive an air traffic command message <NUM> from an entity other than the flight crew (e.g., ATC), wherein the air traffic command message includes one or more tasks for the vehicle to perform. In one example, the air traffic command message <NUM> is a controller-pilot data link communication (CPDLC) message from ATC. In another example the air traffic command message <NUM> is an ATC voice message.

The example automation executive <NUM> is configured to determine the intent of the air traffic command message <NUM>. In one example, to determine the intent, the example automation executive <NUM> is configured to identify the one or more tasks from a voice message. To identify the one or more tasks from the voice message, the example automation executive <NUM> may convert the voice message to text, for example, using voice recognition software, parse the text into a plurality of text sections, match text sections to a task type that is known to be contained in a CPDLC message, and adjust the task type based on the content of the text sections. In another example, to determine the intent, the example automation executive <NUM> is configured to identify the one or more tasks from a CPDLC message.

The example automation executive <NUM> is configured to generate a sequence of "actions" for each task that the aircraft can undertake. To generate the sequence of actions for each task, the example automation executive <NUM> may decompose each of the one or more tasks into one or more primitive actions, wherein the tasks are decomposed into individual actions that are comparable to the actions that would be taken by the pilot (e.g., setting the altitude, changing speed, etc.), which allows for the intuitive presentation of the plan to the pilot for approval or monitoring. An example of a task and corresponding primitive actions can be for the aircraft to undertake the task: "proceed direct to [position], and after passing [position] climb to [altitude]"; this task requires the aircraft to fly to the location specified by [position] and, after reaching that location, adjust its altitude to the specified value.

The example automation executive <NUM> is configured to check the coherence (e.g., a "descend" command from ATC should provide an altitude that is lower than the aircraft's current altitude), feasibility (e.g., a command to change the speed of the aircraft should provide a value that is within the operating speeds of the aircraft), and safety (e.g., simultaneous speed and altitude changes must not cause the aircraft to stall) of each action after generating a sequence of actions.

The example automation executive <NUM> is configured to generate a separate execution stream for different types of actions. The execution streams divorce independent actions from one another by treating their execution separately. This allows tasks that affect different states of the aircraft (e.g., a task that changes the altitude and one that changes the speed) to be planned for and executed independently, except where necessitated by the tasks themselves.

The example automation executive <NUM> is configured to relay its interpretation of the air traffic command message (retrieved tasks <NUM>) and/or the plan of actions to the air traffic command message originator (e.g., ATC), preview the plan with the flight crew, and request flight crew approval of the plan when necessary. To relay the interpretation of the air traffic command message and/or the plan of actions, the example automation executive <NUM> may reply to the ATC with scripted readback for the retrieved tasks <NUM> and/or series of actions. To preview the plan with the flight crew, and request flight crew approval of the plan when necessary, the example automation executive <NUM> may present the plan of actions to the flight crew, for example, via a graphical user interface (GUI) on a display device <NUM>.

The example automation executive <NUM> is configured to issue commands to execute the actions at one or more appropriate points during the mission after receiving flight crew approval of the plan of actions when necessary or automatically when flight crew approval is not required. The example automation executive <NUM> may generate execution event monitors for identifying events on which action execution is dependent (e.g., reaching a waypoint), evaluate event monitors based on the values of the vehicle states <NUM>, and issue a command to execute an action when an event (e.g., determined based on vehicle states <NUM>) on which the action is dependent occurs.

The example mission and safety executive <NUM> is configured to evaluate whether system states <NUM> indicate that action execution is appropriate when action execution is commanded by the automation executive <NUM> and to communicate to the flight crew that action execution has been commanded, for example, via the GUI.

The example mission and safety executive <NUM> is also configured to monitor external events, e.g., using an external events monitor <NUM>, to determine if any external event would dictate not performing a commanded action. For example, an external event may be crossing a geofence wherein geofencing constraints would cause the mission and safety executive <NUM> to override a commanded action to prevent flying into protected airspace. As another example, an external event may be a Detect-and-Avoid (DAA) alert from a DAA sensor that can detect one or more approaching aircraft that may be an obstacle to safely performing the commanded action. Examples of DAA sensors may include a radar system, an automatic dependent surveillance - broadcast (ADS-B) sensor, a traffic alert and collision avoidance system (TCAS), and others. ADS-B is a satellite-based navigation tool in which an aircraft determines its position and then broadcasts that information, enabling other nearby airplanes equipped with the same tool to know its location. TCAS keeps an electronic eye on the sky immediately surrounding an airplane. Should another airplane with a similar device fly too close, an alert will issue. The mission and safety executive <NUM> may choose to deviate from the plan created by the automation executive <NUM>, in order to react to external events.

The example mission and safety executive <NUM> is configured to cause the execution of an action when the system states <NUM> indicate that action execution is appropriate and when no external event is detected that dictates not performing the action. To cause the execution of the action, the example mission and safety executive <NUM> can prepare and send an appropriate input to vehicle equipment, such as the autopilot <NUM> to cause performance of the action. The example mission executive <NUM> is not only configured to check for external events before commanding new action, it is configured to override currently executing actions if an external event occurs while adhering to a given clearance.

A pilot's use of the air traffic command automation system <NUM> should be intuitive and require minimal effort - part of the value of the system. Once configured, the system <NUM> will allow the pilot to monitor the interpretation of incoming commands and (if so configured) approve the resulting plans. Notifications and warnings can be provided to the pilot via the GUI on the display device <NUM> in the cockpit.

The architecture is both configurable and expandable. The system can be configured to present the generated plan to the pilot for approval before enacting it, or it can be configured to automatically relay and enact all or part of the plan without the need for pilot approval (e.g., changes to internal states such as the ATIS code or altimeter setting could be set automatically, while changes to the physical state of the aircraft such as speed or altitude could require pilot approval). The decomposition of complex tasks into a sequence of simple actions allows new tasks to be added simply by creating a new "recipe" for that task that uses already existing components - the coherence, feasibility, and safety checks incorporated in the simpler tasks are automatically included in the new complex task.

<FIG> is a process flow chart depicting an example process <NUM> in an example automation executive and an example process <NUM> in an example mission and safety executive. The order of operation within the processes <NUM> and <NUM> are not limited to the sequential execution as illustrated in <FIG>, but may be performed in one or more varying orders as applicable and in accordance with the present disclosure.

The example process <NUM> includes determining the intent of an uplink message <NUM> (operation <NUM>). The message <NUM> is an uplink message from an entity other than the flight crew (e.g., ATC), wherein the message includes one or more tasks for a vehicle to perform. In one example, the message <NUM> is a controller-pilot data link communication (CPDLC) voice message from ATC. In another example, the message <NUM> is an ATC voice message. The intent is the one or more tasks for the aircraft to perform. In one example, determining the intent includes identifying one or more tasks from the uplink message (e.g., CPDLC or voice message). Identifying one or more tasks from the uplink message may include converting a voice message to text, for example, using voice recognition software, parsing the text into a plurality of text sections, matching text sections to a task type that is known to be contained in a CPDLC message, and adjusting the task type based on the content of the text sections.

The example process <NUM> includes decomposing the tasks into primitive actions (operation <NUM>). The tasks are decomposed into individual actions that are comparable to the actions that would be taken by the pilot (e.g., setting the altitude, changing speed, etc.), which allows for the intuitive presentation of the plan to the pilot for approval or monitoring.

The example process <NUM> includes checking the coherence and feasibility of actions (operation <NUM>). Checking the coherence may include determining if the actions are logical and consistent (e.g., a "descend" command from ATC should provide an altitude that is lower than the aircraft's current altitude). Checking the feasibility may include determining if the actions are possible to do easily or conveniently (e.g., a command to change the speed of the aircraft should provide a value that is within the operating speeds of the aircraft).

The example process <NUM> includes generating separate execution streams for decomposed tasks (operation <NUM>). The execution streams divorce independent actions from one another by treating their execution separately. This allows tasks that affect different states of the aircraft (e.g., a task that changes the altitude and one that changes the heading) to be planned for and executed independently, except where necessitated by the tasks themselves.

The example process <NUM> includes checking the safety of the interacting actions (operation <NUM>). As an example, simultaneous speed and altitude changes must not cause the aircraft to stall.

The example process <NUM> includes relaying the plan and obtaining pilot approval, when necessary (operation <NUM>). Relaying may involve relaying its interpretation <NUM> of the command message and/or the plan of actions to the command message originator (e.g., ATC). Obtaining pilot approval may involve previewing the plan with the flight crew, for example, via a GUI <NUM>, and requesting flight crew approval of the plan, via the GUI <NUM>, when necessary.

The example process <NUM> includes generating execution event monitors (operation <NUM>). Generating execution event monitors may include generating execution event monitors for identifying events on which action execution is dependent (e.g., reaching a waypoint).

The example process <NUM> includes evaluating event monitors and executing actions when the event monitors indicate action initiation is appropriate (operation <NUM>). Evaluating event monitors may include determining vehicle states <NUM>, and executing actions may include issuing a command to execute an action when an event (e.g., determined based on vehicle states <NUM>) on which the action is dependent occurs.

The example process <NUM> includes evaluating action execution (operation <NUM>). Evaluating execution may involve evaluating whether system states <NUM> indicate that action execution is appropriate after action execution has been commanded and communicating to the flight crew that action execution has been commanded, for example, via the GUI <NUM>.

The example process <NUM> includes detecting external events (operation <NUM>). Detecting external events may include monitoring for external events, e.g., to determine if any external event would dictate not performing a commanded action. For example, an external event may be a geofencing alert, from a geofence sensor <NUM>, wherein geofencing constraints would cause the overriding of a commanded action to prevent flying into protected airspace. As another example, an external event may be a Detect-and-Avoid (DAA) alert from a DAA sensor <NUM> that can detect one or more approaching aircraft that may be an obstacle to safely performing the commanded action. Examples of DAA sensors <NUM> may include a radar system, an automatic dependent surveillance - broadcast (ADS-B) sensor, a traffic alert and collision avoidance system (TCAS), and others.

The example process <NUM> includes processing UAS commands (operation <NUM>). Processing UAS commands may involve causing the execution of an action when the system states <NUM> indicate that action execution is appropriate and when no external event is detected that dictates not performing the action. Causing the execution of the action may further involve preparing and sending an appropriate input to vehicle equipment, such as the autopilot <NUM> to cause performance of the action.

Described herein are a system for automating a significant portion of an aircraft's interactions with air traffic command messages. The system provided herein can provide for coherence and feasibility checks during the planning process. The systems provided herein can provide for tasks to be decomposed into smaller, actionable pieces. The system provided herein can provide for safety, coherence, and feasibility checks to be included in any level of the decomposed task, and can include component-level checks. The system provided herein can provide that when tasks are defined using already defined tasks or actions as components, the safety checks become inherited. The system provided herein can provide for generating independent plans (e.g., sequences of primitive actions) for each "stream" of execution, e.g., a task that affects the vertical state of the aircraft may not affect a plan for the horizontal state of the aircraft. The system provided herein can provide for the execution of actions to be triggered by events (e.g., reaching a waypoint or a desired altitude). The system provided herein can provide for plan execution to be automatic or require pilot approval, e.g., based on a setting for level of autonomy and phase of flight. The system provided herein can provide a system with very minimal computational requirements. The system provided herein can provide a system wherein the framework is easy to extend to additional tasks.

The invention relates to an air traffic command automation system on an aerial vehicle for automating the process of listening to, interpreting, adhering to, and responding to air traffic commands according to appended claim <NUM>.

These aspects and other embodiments may include one or more of the following features. The command message may comprise a controller-pilot data link communication (CPDLC) message or a voice message from air traffic control (ATC). To determine the one or more tasks for the aircraft to perform the system may be configured to identify the one or more tasks from a CPDLC or voice message. To identify the one or more tasks from a voice message the system may be configured to convert the voice message to text using voice recognition software, parse the text into a plurality of text sections, match text sections to a task type that may be known to be contained in a CPDLC message, and adjust the task type based on the content of the text sections. To generate the sequence of actions for each task, the system may be configured to decompose each of the one or more tasks into one or more primitive actions that are comparable to the actions that would be taken by the pilot. To relay an interpretation of the command message to the command message originator the system may be configured to reply to ATC with scripted readback for the command message. To preview the plan with the flight crew, the system may be configured to present the plan of actions to the flight crew via a graphical user interface (GUI).

The system may be further configured to evaluate whether system states indicate that action execution is appropriate and to communicate to the flight crew, via a graphical user interface (GUI), that action execution has been commanded. The system may be further configured to monitor external events to determine if any external event would dictate not performing a commanded action. An external event may comprise a geofencing alert or a detect-and-avoid (DAA) alert indicating that one or more approaching aircraft have been detected that may be an obstacle to safely performing the commanded action. The system may be further configured to cause the execution of an action when the system states indicate that action execution is appropriate and when no external event is detected that dictates not performing the action. To cause the execution of the action, the system may be configured to prepare and send an appropriate input to an autopilot system on the vehicle to cause performance of the action.

Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention which is defined by the appended claims.

Claim 1:
An air traffic command automation system on an aerial vehicle for automating the process of listening to, interpreting, adhering to, and responding to air traffic commands, the system comprising one or more processors configured by programming instructions on non-transient computer readable media, the system configured to:
receive an air traffic command message directed to the aerial vehicle, from an entity other than flight crew on the aerial vehicle, that includes one or more tasks for the aerial vehicle to perform;
determine from the command message the one or more tasks for the aerial vehicle to perform;
generate a sequence of actions for each one of the one or more tasks that the aerial vehicle can undertake to accomplish the task, wherein each action corresponds to a state change that can be made by the aerial vehicle and wherein each action is mapped to one of the one or more tasks, the sequence of actions comprising at least simultaneous altitude and speed changes;
verify the coherence and feasibility of each one of the actions of the sequence of actions,
verify the safety of interacting actions of the sequence of actions,
wherein verification of the safety of interacting actions of the sequence of actions comprises checking that the simultaneous altitude and speed changes will not cause the aerial vehicle to stall;
relay an interpretation of the command message to the command message originator;
preview the verified sequence of actions with the flight crew; and
issue commands to execute actions at appropriate points during a mission, wherein the system is configured to generate execution event monitors for identifying events on which action execution is dependent, evaluate the event monitors, and issue a command to execute actions when an event on which the action is dependent is detected.