Patent Description:
While pilots are typically well-trained to handle emergency scenarios, no real-time cross-validation of a pilot's action is typically performed. Analysis is usually performed through debriefing after the emergency has been resolved. Further, no forecast or visualization of the action space and no prediction of the action space is typically performed. Thus, a pilot may not be certain regarding outcomes that may result from their actions.

<CIT>, in accordance with its abstract, states an action recommendation system uses reinforcement learning that provides a next action recommendation to a traffic controller to give to a vehicle pilot such as an aircraft pilot. The action recommendation system uses data of past human actions to create a reinforcement learning model and then uses the reinforcement learning model with current ABS-B data to provide the next action recommendation to the traffic controller. The action recommendation system may use an anisotropic reward function and may also include an expanding state space module that uses a non-uniform granularity of the state space.

<CIT>, in accordance with its abstract, states a method and system for autonomously operating an aircraft. The method comprises:- a pre-flight training step comprising: retrieving recorded surveillance data of a plurality of flights corresponding to at least one aircraft type and at least one route; inferring aircraft intent from the recorded surveillance data; computing reconstructed trajectories using the inferred aircraft intent; selecting a training dataset comprising aircraft intent and reconstructed trajectories of flights corresponding to a particular aircraft type and route; applying a machine learning algorithm on the training dataset to obtain a mapping function between aircraft states and actions; and- a real-time control step executed during a flight of an aircraft, the real-time control step comprising: repeatedly retrieving onboard sensor data; obtaining real-time aircraft states from the onboard sensor data; determining actions associated to the real-time aircraft states using the mapping function; and executing the selected actions on the aircraft. <NPL>), in accordance with its abstract, states online continuous reinforcement learning has shown results in flight control achieving near optimal control and the capability to adapt to changes in the system. However, no guarantees about safety and performance can be given, needed for use in general aviation. Furthermore, performance is often dependent on the tuning of the hyper parameters inside the system. As an initial step in providing guarantees about safety and performance, this paper presents Safe Incremental Dual Heuristic Programming (SIDHP). SIDHP combines the fast learning speed of Incremental Dual Heuristic Programming (IDHP) with a safety layer, able to keep the aircraft within a predetermined safe flight envelope during training.

Described herein is an artificial intelligence based autonomous pilot assistance agent. The pilot assistance agent, also referred to herein as an artificial intelligence power emergency pilot assistance system, is trained based on scenarios run in an aircraft simulator. The system may compute velocities, altitudes, and headings of an aircraft from a given origin and destination without human intervention. Based on the computations, the pilot assistance agent may provide suggestive assistance and guidance to a pilot by translating translated the computed velocities, altitudes, and headings into control actions that can be performed by the pilot during emergencies. During normal flying condition the system may act as a performance evaluation system. In either case, the pilot may remain in control of the aircraft. In an embodiment, an aircraft comprises an emergency pilot assistance system. The emergency pilot assistance system includes an artificial neural network configured to calculate reward (Q) values based on state-action vectors associated with an aircraft, where the state-action vectors include current state data associated with the aircraft and action data associated with the aircraft. The system further includes a user output device configured to provide an indication of an action to a user, where the action corresponds to an agent action that has a highest reward Q value as calculated by the artificial neural network. The artificial neural network is trained in two phases; the first phase includes training the model based on input from a pilot in a simulator and determining whether outcomes during emergency training scenarios are successful; and the second phase includes training the model based on automated scenarios without a pilot present.

In some examples, the highest reward Q value is associated with landing the aircraft at a predetermined destination or a calculated emergency destination in response to an emergency. In some examples, the state data include data matrices associated with the aircraft, the data matrices indicating a heading value, a position value, a system state value, an environmental condition value, a feedback value, a pilot action value, a system availability value, a roll value, a pitch value, a yaw value, a rate of change of roll value, a rate of change of pitch value, a rate of change of yaw value, a longitude value, a latitude value, a rate of change of position value, a rate of change of velocity value, or any combination thereof. In some examples, the action data corresponds to a change in heading, a change in velocity, a change in roll, a change in pitch, a change in yaw, a change in a rate of change of roll, a change in a rate of change of pitch, a change in a rate of change of yaw, change in a rate of change of position, a change in a rate of change of velocity, or any combination thereof. In some examples, the agent action is translated into an aircraft surface control action using an inverse aircraft model.

In some examples, the agent action is taken from a flight envelope including aircraft flight constraints, where the aircraft flight constraints include maps of acceleration and deceleration, rates of climb, rates of drop, velocity thresholds, roll change rate thresholds, pitch change rate thresholds, yaw change rate thresholds, roll thresholds, pitch thresholds, and yaw thresholds.

In some examples, the artificial neural network includes a deep Q network. In some examples, the user output device is incorporated into a cockpit of an aircraft. In some examples, the indication of the user action includes a visual indication, an audio indication, a written indication, or any combination thereof. In some examples, the artificial neural network is implemented at one or more processors, where the one or more processors are configured to determine the state data based on one or more aircraft systems, determine availability data associated with one or more aircraft systems, determine a safe landing zone based on the state data and based on the availability data, determine the action data based on the safe landing zone, the availability data, the state data, and stored constraint data, and generate the state-action vectors based on the state data and the action data. In some examples, the one or more processors are configured to determine heading and velocity data associated with the highest reward Q value and perform one or more inverse dynamics operations to translate the heading and velocity data into the agent action. In some examples, the one or more processors are configured to compare user input to the indicated action and generate a performance rating.

In some examples, the use output device is configured to warn the user when a user input differs from the action. In some examples, the one or more processors are configured to generate updated state-action vectors associated with the aircraft based on updated state data and updated action data, and calculate additional reward Q values based on the updated state-action vectors, where the user output device is configured to provide an additional indication of an additional action to the user, where the additional action corresponds to an updated agent action that has an updated highest reward Q value as calculated by the artificial neural network.

In some examples, a method for training an artificial neural network for an emergency pilot assistance system includes generating training data for a deep Q network by receiving state data associated with an aircraft and an environment of the aircraft from a simulator while a user is operating the simulator; receiving action data from the simulator associated with actions by the user, generating a set of state-action vectors based on the state data and the action data, and determining a reward Q value associated with the set of state-action vectors. The method further includes training a deep Q network based on the training data.

In some examples, the method includes generating additional training data for the deep Q network by receiving automated state data associated with the aircraft from a memory, the automated state data corresponding to an automated scenario, receiving automated action data from the memory, the automated action data associated with the automated scenario, generating an additional set of state-action vectors based on the automated state data and the automated action data, and determining an additional reward Q value associated with the additional set of state-action vectors. The method further includes training the deep Q network based on the additional training data. Optionally, the state data include data matrices associated with the aircraft, the data matrices indicating a heading value, a position value, a system state value, an environmental condition value, a feedback value, a pilot action value, a system availability value, a roll value, a pitch value, a yaw value, a rate of change of roll value, a rate of change of pitch value, a rate of change of yaw value, a longitude value, a latitude value, a rate of change of position value, a rate of change of velocity value, or any combination thereof. Optionally, the action data corresponds to a change in heading, a change in velocity, a change in roll, a change in pitch, a change in yaw, a change in a rate of change of roll, a change in a rate of change of pitch, a change in a rate of change of yaw, change in a rate of change of position, a change in a rate of change of velocity, or any combination thereof. Optionally, the action data is based on a flight envelope including aircraft flight constraints, wherein the aircraft flight constraints include maps of acceleration and deceleration, rates of climb, rates of drop, velocity thresholds, roll change rate thresholds, pitch change rate thresholds, yaw change rate thresholds, roll thresholds, pitch thresholds, and yaw thresholds.

In an example, an emergency pilot assistance method includes calculating reward Q values using a deep Q network, where the reward values are based on state-action vectors associated with an aircraft, and where the state-action vectors include state data associated with the aircraft and action data associated with the aircraft. The method further includes providing an indication of an action to a user at a user output device, where the action corresponds to an agent action that has a highest reward Q value as calculated by the deep Q network. In some examples, the highest reward Q value is associated with landing the aircraft at a predetermined destination or a calculated emergency destination in response to an emergency.

Described herein is a reinforcement learning based autonomous pilot assistance agent, also referred to herein as an artificial intelligence power emergency pilot assistance system, which is trained using an aircraft simulator and can perform the tasks of computing velocities, altitudes, and headings of an aircraft from a given origin and a destination without human intervention. The pilot assistance agent may be used to assist and guide a pilot during emergency situations. For example, the computed velocities, altitudes, and headings can be translated into control action that may be performed by the pilot to guide the aircraft to a safe landing zone.

The systems described herein may rely on a deep Q network to enable model free deep Q learning for obtaining complete reward-based mappings. The mappings may be used to determining a course of action during an emergency. As a brief overview of deep Q learning, as it is applied herein, during an emergency the system may determine a candidate goal (which for example may include determining a safe landing location). The system may also have access to a user policy, which may be based on aircraft flight constraints, a flight envelope, maps of acceleration and deceleration, rate of climb, and rate of drop. The user policy effectively describes the possible actions that may be taken at any given time within the aircraft. Based on these parameters, the system may iteratively map a sequence of possible actions to bring the aircraft to the candidate goal. If the sequence is successful in bringing the aircraft to the candidate goal (i.e., if the sequence will result in the aircraft landing safely at the safe landing location) then a high reward Q value (e.g., <NUM>) may be assigned. If the sequence is not successful then a low reward Q value (e.g., <NUM>) may be assigned. As each sequence may branch at each iteration the reward Q values may increase or decrease throughout the iterations depending on the likelihood of a safe landing at any given point in the sequence of actions.

The system may interact with an aircraft environment and pilot to select actions in a way that maximize future reward values. During the system calculations, because future states cannot be perfectly determined, a standard assumption that future rewards may be discounted by a set factor per time-step may be employed. A future discounted return Rt may be calculated as follows: <MAT> where T is the flight duration, t' is the current time step, t is the next time step in the iteration, γ is the discount factor, and rt' is the current discounted return. For the examples described herein, γ was set to <NUM>. However, other values are possible.

The desired action-value function Q*( s, a) may be defined as the best expected return achievable by following the policy based on a sequence, s, an action, a, Q*(s, a) may be derived based on the Bellman equation, which is known with respect to deep Q learning. For purposes of this disclosure, the relationship may be described as follows: if the optimal value Q*(s, a) of a sequence at the next time-step is known for all possible actions, then an optimizing strategy is to select an action that maximizes the expected value of r + YQ*(s', a'), where r is the discounted return and Υ is the discount factor.

The reinforcement learning algorithm as described above may be used to estimate the action-value function by using the Bellman equation as an iterative update. If fully performed, the algorithm would converge to an optimal action-value function. In practice, however, this approach may be impractical, because the action-value function would be estimated separately for each sequence, without any generalization. Thus, the computations would expand exponentially which would likely involve more processing power than is available. Instead, a function approximator may be used to estimate the action-value function, Q(s, a; θ) < Q(s, a). In the reinforcement learning field this is typically a linear function approximator. By relying on training data received during simulation, a deep Q network may be developed to approximate the optimal actions to achieve the greatest probability of a successful outcome.

<FIG> and <FIG> depict systems for training an artificial neural network for use with an emergency pilot assistance system. Training the artificial neural network takes place in two phases. A first phase includes training the model based on input from a pilot in a simulator and determinations of whether outcomes during emergency training scenarios are successful. During the second phase, the model is trained based on automated scenarios without a pilot present.

During the first phase, training of the artificial neural network may be performed along with training a pilot in a training simulator. The system may learn end-to-end mappings of aircraft flight paths (e.g., velocities, altitudes, and headings) from environmental observation and user input with the task reward, e.g., a safe landing, as a form of supervision. The reward may be calculated based on safely landing the aircraft at a desired location or at a nearby safe landing location. From the perspective of the system being trained, the pilot's actions are incorporated into a policy that also includes constraints such as a flight envelope, maps of acceleration and deceleration, a rate of climb, a rate of drop and others policy data for a safe flight. From the pilot's perspective, the system may behave like an adaptive interface that learns a personalized mapping from the pilot's commands, environments, goal space and flight constraint policy to action of flight path and its other parameters.

Referring to <FIG>, an example of a system <NUM> for training an artificial neural network in a first training phase is depicted. The system <NUM> may include a simulator <NUM> and a deep Q network <NUM>. It should be understood by persons of skill in the art, having the benefit of this disclosure, that the deep Q network <NUM> may be implemented as part of a broader artificial neural network as described further with reference to <FIG>. The simulator <NUM> simulates an aircraft <NUM> and an environment <NUM> of the aircraft during pilot training of a user <NUM>.

During operation, while the user <NUM> is performing training exercise in the simulator <NUM>, state data <NUM> associated with the aircraft <NUM> and with the environment <NUM> of the aircraft <NUM> may be collected from the simulator <NUM>. The state data <NUM> may indicate a current state of the aircraft <NUM> and the environment <NUM>. A portion of the state data <NUM> may also be based on system availability <NUM> of the aircraft <NUM>. For example, during an emergency one or more systems of the aircraft <NUM> may be inoperable or otherwise unavailable for use. These factors may be taken into account when generating the state data <NUM>. The state data <NUM> may also be based on aircraft performance operational constraints <NUM>, which may represent the limits of what a particular aircraft may do in a particular scenario being run at the simulator <NUM>.

Action data <NUM> may also be collected from the simulator <NUM>. The action data <NUM> may be derived from actions <NUM> taken by the user <NUM> during flight training. The action data <NUM> may also be based on a flight envelope <NUM>, representing the actions that may be taken with respect to a particular aircraft.

Based on the state data <NUM> and the action data <NUM>, training data <NUM> may be compiled. The training data <NUM> may include a set of state-action vectors <NUM> formed by combining the state data <NUM> and the action data <NUM> at incremental steps during the simulation. A reward Q value <NUM> may be determined based on an outcome associated with the set of state-action vectors <NUM> and based on the discounted return function described herein. The training data <NUM> may also include the reward Q value <NUM> and may be used as training data for the deep Q network <NUM>.

A challenge typically associate with training emergency assistance systems may be adapting standard deep reinforcement learning techniques that leverage continuous input from the actions <NUM> and make adjustments to the inputs based on a consequence of feedback associated with the actions <NUM>. By using human-in-the-loop deep Q-learning, as described herein, with a user <NUM> actively using the simulator <NUM>, the system <NUM> may learn an approximate state-action value function that computes expected future return values without computing each possible path in the state-action vectors <NUM> for an action given current environmental observation and the pilot's control input. Rather than finding a highest-value action, the deep Q network <NUM> may be trained to determine a closest high-value action to a user's input. This approach balances taking optimal actions with preserving a pilot's feedback control loop. This approach also enables the user <NUM> to directly modulate a level of assistance through a parameter α ε [<NUM>, <NUM>], which may set a threshold for tolerance for suboptimal actions.

Standard deep reinforcement learning algorithm may include a large number of interactions for a very long period in order to have sufficient training. Simulator training alone is likely to be insufficient because it may not be feasible to obtain enough data. During a second phase of training, pilot control input is replaced with automated scenario files having fixed control inputs from various origins to various destinations. The automated scenario files may cover more of the operating condition of an aircraft during these scenarios. This automated training approach may also be useful for covering extreme emergency conditions, which may be difficult to simulate with a pilot. In some cases, this training will enable the system to determine a safe course of action more reliably than a pilot by learning based on a full-spectrum of input from each scenario and learning based on scenarios that have not yet been anticipated by pilots.

The remaining portions of the second phase of training may be the same as described with reference to <FIG>. Deep Q-learning may be used to learn an approximate state-action value function that computes the expected future return of an action given the current environmental observation, policy constraint, and the automated scenario's input. Equipped with this value function, the system may execute the closest high-value action to the scenario's control input. The reward function for the agent may be a combination of known terms computed for every state, and a terminal reward provided by the user upon succeeding in landing the plane safely.

Referring to <FIG>, an example of a system <NUM> for training an artificial neural network in a second training phase is depicted. The system <NUM> may include a memory <NUM> and a deep Q network <NUM>. The memory <NUM> may store an automated scenario <NUM> associated with an aircraft <NUM>. In practice, many automated scenarios would be stored in the memory <NUM>.

The memory <NUM> may include memory devices such as random-access memory (RAM), read-only memory (ROM), magnetic disk memory, optical disk memory, flash memory, another type of memory capable of storing data and processor instructions, or the like, or combinations thereof. Further, the memory may be part of a processing device (not shown) such as a computing device.

During operation, automated state data <NUM> associated with the aircraft <NUM> and with the automated scenario <NUM> may be collected. In some examples, the collection may take the form of multiple automated scenario files. The automated state data <NUM> may indicate a current state of the aircraft <NUM> during the automated scenario <NUM>. A portion of the automated state data <NUM> may also be based on system availability <NUM> of the aircraft <NUM> and on aircraft performance operational constraints <NUM>, as described with reference to <FIG>. Automated action data <NUM> may also be derived from the automated scenario <NUM> and a flight envelope <NUM>, representing the actions that may be taken with respect to the aircraft <NUM>.

Based on the automated state data <NUM> and the automated action data <NUM>, additional training data <NUM> may be compiled. The additional training data <NUM> may include an additional set of state-action vectors <NUM> formed by combining the automated state data <NUM> and the automated action data <NUM>. An additional reward Q value <NUM> may be determined based on an outcome associated with the additional set of state-action vectors <NUM> and based on the discounted return function described herein. The additional training data <NUM> may include the additional reward Q value <NUM> and may be used to train the deep Q network <NUM>.

While <FIG> is described with respect to a single automated scenario <NUM>, in practice many scenarios may be stored in the memory <NUM> and may be used to generate the additional training data <NUM>. Because the additional training data <NUM> is not compiled based on real time situations, it may be generated much faster, thereby enabling sufficient training data to be generated to fully train the deep Q network <NUM>. By using both pilot simulator generated data and automated scenario data, the deep Q network <NUM> may be trained to learn realistic pilot responses for a complete set of emergency scenarios.

Referring to <FIG> and <FIG>, the deep Q network <NUM> may be implemented in an example of an emergency pilot assistance system <NUM> to assist a pilot during an emergency. As a brief overview, during an emergency, the system <NUM> may determine a state of an aircraft <NUM>. The state may relate to factors such as whether the aircraft is landing, approaching, or climbing. Other possible states may exist. Likewise, the system <NUM> may analyze the onboard system availability of the aircraft <NUM> to determine availability data <NUM>. The system availability may relate to potential engine failure, surface control failure, fuel availability, and structural integrity. Based on the system availability and aircraft situational condition, the system <NUM> may determine a safe landing zone and guide the pilot on maneuvers. Based on the current system state and aircraft current feedback the system <NUM> may estimate near-optimal trajectories to the safe landing destination. The system <NUM> may continuously evaluate the situation to guide the pilot to necessary action.

The system <NUM> may include, or otherwise be implemented at, an aircraft <NUM>. The system may also include one or more processors <NUM>, which may be implemented at the aircraft <NUM> or in some examples, may be distributed in a decentralized manner. The system <NUM> may also include an artificial neural network <NUM>. Portions of the system <NUM> may be implemented at the one or more processors <NUM>. However, for clarity different functional aspects of the system <NUM> may be depicted as separate from the processors <NUM>.

The aircraft <NUM> may include aircraft systems <NUM> and a cockpit <NUM>. The aircraft systems <NUM> may include mechanical systems, electrical systems, sensors, actuators, and the like. At least some of the aircraft system <NUM> may be able to determine the existence of an emergency <NUM>. The cockpit <NUM> may include a user output device <NUM>. The user output device <NUM> may include visual output systems, audio output systems, text output systems, and the like. The aircraft <NUM> may include additional systems to perform functions typically associated with aircraft, but which are omitted from <FIG> for clarity.

The one or more processors <NUM> may include a microcontroller, a central processing unit (CPU), a graphical processing unit (GPU), a digital signal processor (DSP), a peripheral interface controller (PIC), another type of microprocessor, and/or combinations thereof. Further, the one or more processors <NUM> may be implemented as integrated circuits, complementary metal-oxide-semiconductor (CMOS) field-effect-transistor (MOSFET) circuits, very-large-scale-integrated (VLSI) circuits, field-programmable gate arrays (FPGAs), application-specific integrated circuit (ASICs), combinations of logic gate circuitry, other types of digital or analog electrical design components, or combinations thereof.

The artificial neural network <NUM> may include the deep Q network <NUM> and may be trained as described herein. In particular, the artificial neural network may be trained to perform an approximation function to determine reward Q values associated with states and possible actions associated with the aircraft <NUM>. It should be understood by persons of skill in the art, having the benefit of this disclosure, that the artificial neural network <NUM> may be a broader network, of which the deep Q network <NUM> may be a part.

During operation, an emergency <NUM> may result from, or be detected by, one or more of the aircraft systems <NUM>. In response to the emergency <NUM>, the one or more processors <NUM> may determine state data <NUM> and action data <NUM> based on the aircraft systems <NUM>. For example, the state data <NUM> may include a matrix of aircraft heading, positions and velocity, current state, environmental condition, feedbacks, pilot action, aircraft system availability such as current value roll, pitch, yaw, rate of change of roll, pitch and yaw, longitude and latitude, rate of change of position, velocity, other state parameters associated with the aircraft <NUM>, or combinations thereof. The action data <NUM> may be based on heading and velocity such as the value of roll, pitch, yaw, rate of change of roll, pitch and yaw, rate of change of position, and velocity. State-action vectors <NUM> may be generated based on the state data <NUM> and the action data <NUM>.

The processors <NUM> may determine and/or compile availability data related to the aircraft systems <NUM>. For example, in an emergency <NUM>, some systems may not be available. A safe landing zone <NUM> may be determined based on the state data <NUM> and based on the availability data <NUM>. The safe landing zone <NUM> may be a predetermined destination <NUM> or, in some cases, an emergency destination <NUM> may be determined based on a location of the aircraft <NUM> and based on the availability data <NUM> and stored constraint data <NUM> associated with the aircraft <NUM>. The action data <NUM> may depend on the safe landing zone <NUM>, the availability data <NUM>, the state data <NUM>, and stored constraint data <NUM>.

The artificial neural network <NUM> may be used to determine headings and velocities data <NUM> that may be associated with calculated reward Q values <NUM>. The reward Q values <NUM> may be determined based on the state-action vectors <NUM> and may be associated with landing the aircraft <NUM> safely at the safe landing zone <NUM>. For example, the higher the reward Q values <NUM> are, the more likely a safe landing is to occur. From the headings and velocities data <NUM>, heading and velocity data <NUM> may be associated with a highest reward Q value <NUM> as determined by the artificial neural network <NUM>.

One or more inverse dynamics operations <NUM> may be performed to translate the heading and velocity data <NUM> into an agent action <NUM>. Further, in some examples, additional data from the headings and velocities data <NUM> may be translated into agent actions <NUM>. Each of the agent actions <NUM> may be associated with reward Q values <NUM>, which may correspond to the reward Q values <NUM>. The agent action <NUM> may be associated with a highest reward Q value <NUM> that corresponds to the highest reward Q value <NUM> of the heading and velocity data <NUM>. An inverse aircraft model <NUM> may be used to translate the agent action <NUM> into a surface control action <NUM> that may be usable as instructions to the user <NUM> to guide the aircraft <NUM>.

Within the cockpit <NUM>, the user output device <NUM> may provide an indication <NUM> of an action <NUM> to the user <NUM>. The action <NUM> may correspond to the agent action <NUM> and may also be, or may be derived from, the surface control action <NUM>. The indication <NUM> of the action <NUM> may include a visual indication <NUM>, an audio indication <NUM>, a written indication <NUM>, or any combination thereof. If the user <NUM> does not perform the action <NUM>, then the user output device <NUM> may generate a warning <NUM>. The user may perform actions using user input <NUM>, which may include flight controls and/or other controls associated with aircraft cockpits. In cases where, there is no emergency, the system <NUM> may nevertheless generate a performance rating <NUM> associated with a flight based on comparing the agent actions <NUM> generated by the artificial neural network <NUM> to the user input <NUM>.

It should be noted that the process described with respect to the system <NUM> is iterative and may be continually performed during a flight and/or during an in-flight emergency. Thus, agent actions may be continually fed to the output device <NUM> as the state-action vectors <NUM> change. Referring to <FIG>, this continual update is depicted. As the aircraft <NUM> changes its vectors and as the availability data <NUM> changes based on the aircraft systems <NUM> during the emergency <NUM>, updated state-action vector <NUM> may be generated. The updated state-action vectors <NUM> may include updated state data <NUM> and updated action data <NUM>.

The artificial neural network <NUM> may be used to generate updated headings and velocities data <NUM>, which may be associated with additional reward Q values <NUM>. The updated heading and velocity data <NUM> that is associated with a highest additional reward Q value <NUM> may be determined to safely guide the user <NUM> to land at the safe landing zone <NUM>. Based on the updated headings and velocities data <NUM>, updated agent actions <NUM> may be generated and associated with additional reward Q values <NUM>, which may correlate with the additional reward Q values of the updated headings and velocities data <NUM>. An updated agent action <NUM> may be associated with a highest additional reward Q value <NUM>, which may correlate with the highest additional reward Q value <NUM> of the updated heading and velocity data <NUM>. The updated agent action <NUM> may be used to generate an updated surface control action <NUM>.

The user output device <NUM> may be configured to provide an additional indication <NUM> of an additional action <NUM> to the user <NUM>. The additional indication <NUM> may include an additional visual indication <NUM>, an additional audio indication <NUM>, an additional written indication <NUM>, or any combination thereof. If the user <NUM> does not perform the additional action <NUM>, an additional warning <NUM> may be generated. As before, an updated performance rating <NUM> may be generated based on comparing the user input <NUM> to the updated agent actions <NUM>.

By providing indications of actions that a pilot can take to safely land an aircraft at a safe landing zone, the system <NUM> may reduce the workload on the pilot in case of an emergency. Further, the system <NUM> may warn pilot when the pilot's actions may lead to on such action which can lead into catastrophic failure. Also, even in cases where there is no emergency, the system <NUM> can, nevertheless, rate a pilot's performance for training purposes. Other advantages may exist.

Referring to <FIG>, an example of state data <NUM> is depicted. The state data <NUM> may include data matrices <NUM> associated with an aircraft, such as the aircraft <NUM>, the data matrices <NUM> may indicate a heading value <NUM>, a position value <NUM>, a system state value <NUM>, an environmental condition value <NUM>, a feedback value <NUM>, a pilot action value <NUM>, a system availability value <NUM>, a roll value <NUM>, a pitch value <NUM>, a yaw value <NUM>, a rate of change of roll value <NUM>, a rate of change of pitch value <NUM>, a rate of change of yaw value <NUM>, a longitude value <NUM>, a latitude value <NUM>, a rate of change of position value <NUM>, a rate of change of velocity value <NUM>, or any combination thereof. The state data <NUM> may correspond to the state data <NUM>, the automated state data <NUM>, the state data <NUM>, and/or the updated state data <NUM>.

Referring to <FIG>, an example of action data <NUM> is depicted. The action data <NUM> may include a change in heading <NUM>, a change in velocity <NUM>, a change in roll <NUM>, a change in pitch <NUM>, a change in yaw <NUM>, a change in a rate of change of roll <NUM>, a change in a rate of change of pitch <NUM>, a change in a rate of change of yaw <NUM>, change in a rate of change of position <NUM>, a change in a rate of change of velocity <NUM>, or any combination thereof. The action data <NUM> may correspond to the action data <NUM>, the automated action data <NUM>, the action data <NUM>, and/or the updated action data <NUM>.

Referring to <FIG> an example of a flight envelope <NUM> is depicted. The action data <NUM> may be based at least partially on the flight envelope <NUM>. The flight envelope <NUM> may include aircraft flight constraints <NUM>. The aircraft flight constraints <NUM> may include maps of acceleration and deceleration <NUM>, rates of climb <NUM>, rates of drop <NUM>, velocity thresholds <NUM>, roll change rate thresholds <NUM>, pitch change rate thresholds <NUM>, yaw change rate thresholds <NUM>, roll thresholds <NUM>, pitch thresholds <NUM>, and yaw thresholds <NUM>.

Referring to <FIG>, a flow chart depicting an example of a method <NUM> for training an artificial neural network is depicted. The method <NUM> includes generating training data for a deep Q network, at <NUM>. The training data <NUM> for the deep Q network <NUM> is generated by the simulator <NUM>.

Generating the training data may include receiving state data associated with an aircraft and an environment of the aircraft from a simulator while a user is operating the simulator, at <NUM>. For example, the state data <NUM> may be received from the simulator <NUM> while the user <NUM> is operating the simulator <NUM>.

Generating the training data may further include receiving action data from the simulator associated with actions by the user, at <NUM>. For example, the action data <NUM> may be received from the simulator <NUM>.

Generating the training data may also include generating a set of state-action vectors based on the state data and the action data, at <NUM>. For example, the set of state-action vectors <NUM> may be generated based on the state data <NUM> and the action data <NUM>.

Generating the training data may include determining a reward Q value <NUM> associated with the set of state-action vectors <NUM>, at <NUM>. For example, the reward Q value <NUM> may be determined by the system <NUM> and may be associated with the set of state-action vectors <NUM>.

The method <NUM> further includes training a deep Q network based on the training data, at <NUM>. For example, the deep Q network <NUM> is trained based on the training data <NUM>.

The method <NUM> also includes generating additional training data for the deep Q network, at <NUM>. For example, the additional training data <NUM> may be generated based on the automated scenario <NUM>, and additional automated scenarios during additional iterations.

Generating the additional training data may include receiving automated state data associated with the aircraft from a memory, the automated state data corresponding to an automated scenario, at <NUM>. For example, the automated state data <NUM> may be received from the memory <NUM>.

Generating the additional training data may further include receiving automated action data from the memory, the automated action data associated with the automated scenario, at <NUM>. For example, the automated action data <NUM> may be received from the memory <NUM>.

Generating the additional training data may also include generating an additional set of state-action vectors based on the automated state data and the automated action data, at <NUM>. For example, the additional set of state-action vectors <NUM> may be generated based on the automated state data <NUM> and the automated action data <NUM>.

Generating the additional training data may include determining an additional reward Q value associated with the additional set of state-action vectors, at <NUM>. For example, the additional reward Q value <NUM> may be generated and may be associated with the additional set of state-action vectors <NUM>.

The method <NUM> includes training the deep Q network based on the additional training data, at <NUM>. For example, the deep Q network <NUM> is trained based on the additional training data <NUM>.

Referring to <FIG>, a flow chart depicting an example of a method <NUM> for emergency pilot assistance is depicted. The method <NUM> may include calculating reward Q values using a deep Q network, wherein the reward values are based on state-action vectors associated with an aircraft, and wherein the state-action vectors include state data associated with the aircraft and action data associated with the aircraft, at <NUM>. For example, the reward Q values <NUM> may be calculated using the deep Q network <NUM> and may be based on the state-action vectors <NUM> associated with the aircraft <NUM>.

Claim 1:
An aircraft (<NUM>) comprising an emergency pilot assistance system (<NUM>), the emergency pilot assistance system (<NUM>) comprising:
an artificial neural network (<NUM>) configured to calculate reward (Q) values (<NUM>) based on state-action vectors (<NUM>) associated with the aircraft (<NUM>), wherein the state-action vectors (<NUM>) include state data (<NUM>, <NUM>) associated with the aircraft (<NUM>) and action data (<NUM>, <NUM>) associated with the aircraft (<NUM>); and
a user output device (<NUM>) configured to provide an indication (<NUM>) of an action (<NUM>) to a user (<NUM>), wherein the action (<NUM>) corresponds to an agent action (<NUM>) that has a highest reward Q value (<NUM>) as calculated by the artificial neural network (<NUM>),
wherein:
the artificial neural network (<NUM>) is trained in two phases;
the first phase includes training the artificial neural network (<NUM>) based on input from a pilot in a simulator and determining whether outcomes during emergency training scenarios are successful; and
the second phase includes training the artificial neural network (<NUM>) based on automated scenarios without a pilot present.