Patent Description:
A vehicle, such as an aircraft, may operate to travel along a predetermined route. Some contexts of operating the vehicle may rely at least in part on visual information. For example, a takeoff or landing phase of a route for an aircraft may use visual information to navigate the aircraft. A degraded visual environment may impair such navigation.

In the context of autonomous operation of a vehicle, lacking or unreliable sensor data caused by a degraded visual environment presents difficulties. For example, a controller of the vehicle may have difficulty identifying obstacles in the environment, or orienting the vehicle relative to a surface, such as a road surface or a ground surface. Degraded visual environments likewise disorient manual operators of vehicles. Autonomous and manual operators alike may need to make different decisions for operating the vehicle in a degraded visual environment depending on an operating context of the vehicle. Existing systems and methods for operating a vehicle in a degraded visual environment may struggle to effectively control the vehicle in these different operating contexts.

First results of LIDAR-aided helicopter approaches during NATO DVE-mitigation trials by <NPL>, in accordance with its abstract, states "Landing a helicopter on unprepared sites can quickly become a challenging task in operational scenarios. Especially when environmental factors reduce the available visual cues for the pilot, the risk of disorientation increases. Motivated to avoid accidents in degraded visual environments (DVE), the US Army program for DVE-Mitigation (DVE-M) with NATO participation supports international efforts in the development of systems for enhanced situational awareness during DVE. A workgroup of the German Aerospace Center (DLR) and HENSOLDT Sensors GmbH participated at flight trials of the European DVE-M campaign in Manching with DLR highly modified research helicopter. The state of a system under development was presented which combines eyes-out tunnel-in-the-sky symbology (SferiAssist®) with dynamic path updates based on a laser sensor. Details of four flights during the campaign in Manching performed in February <NUM> are given and discussed in terms of a technical analysis and results of pilot evaluations.

What is needed is a system for effectively operating a vehicle in a degraded visual environment.

There is described herein a method for controlling a vehicle in a degraded visual environment, comprising: identifying a degraded visual environment corresponding to a phase of a route followed by the vehicle; determining, based on the phase of the route, a first segment of a trajectory of the vehicle along which to search for a location with an improved navigation environment; causing the vehicle to follow the first segment until: (i) identifying the improved navigation environment, or (ii) reaching an end of the first segment without identifying the improved navigation environment; determining a second segment of the trajectory based on whether the improved navigation environment has been identified; and causing the vehicle to follow the second segment; wherein: identifying the degraded visual environment comprises: receiving sensor data obtained by one or more sensors on the vehicle; determining a difference between the sensor data and expected data; and identifying the degraded visual environment based on determining the difference between the sensor data and the expected data; and the one or more sensors comprise a Light Detection and Ranging (LIDAR) device, and determining the difference between the sensor data and the expected data comprises determining a difference between a number of returning light pulses represented by the sensor data and an expected number of returning light pulses.

A method for controlling a vehicle in a degraded visual environment is described. The method includes identifying a degraded visual environment corresponding to a phase of a route followed by the vehicle. The method includes determining, based on the phase of the route, a first segment of a trajectory of the vehicle along which to search for a location with an improved navigation environment. The method includes causing the vehicle to follow the first segment until: (i) identifying the improved navigation environment, or (ii) reaching an end of the first segment without identifying the improved navigation environment. The method includes determining a second segment of the trajectory based on whether the improved navigation environment has been identified. The method includes causing the vehicle to follow the second segment.

A system for controlling a vehicle in a degraded visual environment is described. The system includes a vehicle. The vehicle includes a computing device having a processor and memory storing instructions executable by the processor. The instructions are executable by the processor to identify a degraded visual environment corresponding to a phase of a route followed by the vehicle. The instructions are executable by the processor to determine, based on the phase of the route, a first segment of a trajectory of the vehicle along which to search for a location with an improved navigation environment. The instructions are executable by the processor to cause the vehicle to follow the first segment until: (i) identifying the improved navigation environment, or (ii) reaching an end of the first segment without identifying the improved navigation environment. The instructions are executable by the processor to determine a second segment of the trajectory based on whether the improved navigation environment has been identified. The instructions are executable by the processor to cause the vehicle to follow the second segment.

A non-transitory computer readable medium is described. The non-transitory computer readable medium has stored thereon instructions, that when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include identifying a degraded visual environment corresponding to a phase of a route followed by a vehicle. The functions include determining, based on the phase of the route, a first segment of a trajectory of the vehicle along which to search for a location with an improved navigation environment. The functions include causing the vehicle to follow the first segment until: (i) identifying the improved navigation environment, or (ii) reaching an end of the first segment without identifying the improved navigation environment. The functions include determining a second segment of the trajectory based on whether the improved navigation environment has been identified. The functions include causing the vehicle to follow the second segment.

The features, functions, and advantages that have been discussed can be achieved independently in various examples or may be combined in yet other examples. Further details of the examples can be seen with reference to the following description and drawings.

The novel features believed characteristic of the illustrative examples are set forth in the appended claims. The illustrative examples, however, as well as a preferred mode of use, further objectives and descriptions thereof, will best be understood by reference to the following detailed description of an illustrative example of the present disclosure when read in conjunction with the accompanying drawings, wherein:.

Disclosed examples will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all of the disclosed examples are shown. Indeed, several different examples may be described and should not be construed as limited to the examples set forth herein. Rather, these examples are described so that this disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

Within examples, systems and methods for operating a vehicle in a degraded visual environment (DVE) are described. As used herein, the term "degraded visual environment" refers to an environment lacking visual cues used for navigating a vehicle. For example, a DVE may result from one or more of rain, snow, fog, dust, sand, or smoke surrounding the vehicle. In DVE scenarios, a vehicle may receive sensor data that is either lacking or that misrepresents the environment or objects in the environment. Methods and systems are described that allow for automatic or manual navigation within the environment. For purposes of the following description, the terms "automatically" or "autonomously" may involve carrying out the functions programmatically based on sensor data, stored information, machine learning techniques, or the like, without user input and/or instructions. Within this context, functions may still be referred to as automatic or autonomous if they are prompted at some point by user actions.

As used herein, the term "path" refers to a continuous set of positions between two points, such as waypoints of a route. A "trajectory" refers to a velocity (speed and direction) taken while following a path. In some instances herein, a trajectory may be referred to in terms of its path for purposes of simplicity, but it should be understood that the trajectory additionally includes a velocity or velocities for traveling along the path. A "route" refers to at least one path defined by one or more sets of waypoints. A route may include a plurality of phases, each phase corresponding to at least one waypoint.

Within examples, a vehicle operates in different contexts that affect how to navigate within the DVE. For example, an aircraft follows a predetermined route including different phases that are defined by trajectories followed by the aircraft between various waypoints. The plurality of phases includes at least a takeoff phase, a landing phase, and a cruising phase. Depending on the phase of the route, the vehicle determines different trajectories. For example, in a takeoff phase, the vehicle may follow an upward trajectory to search for an improved navigation environment, while in a landing phase, the vehicle may follow a downward trajectory towards an alternate landing location. Accordingly, the methods and systems described herein provide an adaptive way of addressing a DVE experienced by a vehicle. In particular, examples described herein relate to determining trajectories for identifying improved visual conditions based on a context of experiencing the DVE.

The following examples generally depict aircraft implementations, it should be understood that the same systems and methods can be applied to other types of vehicles as well, such as land vehicles or water vehicles.

Turning now to the figures, <FIG> illustrates a block diagram of a system <NUM> that includes a vehicle <NUM>, according to an example. The vehicle <NUM> includes a computing device <NUM>, a navigation system <NUM>, a DVE detection system <NUM>, a trajectory generation system <NUM>, and steering/hover mechanisms <NUM>. Other devices, systems, devices, modules, software, data stores, and the like can also be included. Further, within examples, various components described with respect to <FIG> may be integrated into a singular computing device or system, separated into further discrete components, or otherwise rearranged to achieve similar functionality to that described herein. Further, various systems, devices, and mechanisms, can be implemented in either a software or hardware context.

The computing device <NUM> includes one or more processor(s) <NUM>, a memory <NUM>, instructions <NUM>, and a user interface <NUM>. The one or more processor(s) <NUM> is general-purpose processors or special purpose processors (e.g., digital signal processors, application specific integrated circuits, etc.). The one or more processor(s) <NUM> are configured to execute the instructions <NUM> (e.g., computer-readable program instructions) that are stored in the memory <NUM> to provide the functionality of computing device <NUM>, and related systems and methods described herein.

The memory <NUM> includes or takes the form of one or more computer-readable storage media that are read or accessed by the processor(s) <NUM>. The computer-readable storage media can include volatile and/or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with the processor(s) <NUM>. The memory <NUM> is considered non-transitory computer readable media. In some examples, the memory <NUM> can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other examples, the memory <NUM> can be implemented using two or more physical devices. The memory <NUM> thus is a non-transitory computer readable storage medium, and instructions <NUM> executable by the processor(s) <NUM> are stored on the memory <NUM>. The instructions <NUM> include computer executable code, and can be executed by the processor(s) <NUM> to achieve functionality described herein.

The user interface <NUM> includes a mouse, a keyboard, a touchscreen, a microphone, a gesture recognition system, a combination thereof, or any other means of receiving user input. In particular, the user interface <NUM> is configured to receive input from an operator (e.g., a pilot or a driver) of the vehicle <NUM>, or a remote technician of the vehicle <NUM>, for example. Examples described herein relate to autonomous operation of a vehicle. Accordingly, the user interface <NUM> may not be necessary to perform functionality described herein. Further, while the user interface is depicted as being a part of the vehicle <NUM>, it should be understood that the user interface <NUM> can be integrated in a separate device that is in communication with the vehicle <NUM>.

As shown in <FIG>, the computing device <NUM> is communicatively coupled to the navigation system <NUM>, the DVE detection system <NUM>, the trajectory generation system <NUM>, and the steering/hover mechanisms <NUM>. Though not depicted in <FIG>, each of these components of the vehicle <NUM> includes processor(s), memory, and instructions configured similarly to the one or more processor(s) <NUM>, the memory <NUM>, and the instructions <NUM> as described above, though each may include instructions executable to achieve a distinct functionality. Further, though these components are not depicted as being in direct communication (rather, they are shown being in communication via the computing device <NUM>), each of these components directly communicate with one another, or operate independently without receiving communications from one another.

The navigation system <NUM> includes a Global Positioning System (GPS) <NUM>, an Inertial Navigation System (INS) <NUM>, and an Inertial Measurement Unit (IMU) <NUM>. The navigation system <NUM> and/or one or more of its components is configured to determine a location, an orientation, and an altitude of the vehicle <NUM>. More particularly, one or more of the GPS <NUM>, the INS <NUM>, and the IMU <NUM> obtains sensor data indicative of a location, orientation/attitude, and altitude (though not depicted, an altimeter, for example, may also be included) of the vehicle <NUM>. This information, either in the form of unprocessed sensor data, or as targeted information indicative of the location, orientation, attitude, and altitude of the vehicle <NUM>, is transmitted to the computing device <NUM>. For example, the navigation system <NUM> transmits a simplified representation of the sensor data. As described below, the computing device <NUM> and/or one or more additional components of the vehicle <NUM> use this information in generating and selecting a flight path or trajectory for the vehicle <NUM>.

The DVE detection system <NUM> includes a Light Detection and Ranging (LIDAR) device <NUM>, an image capture device <NUM> (e.g., a camera, a light sensor array, or another imaging device or system), and a radar device <NUM>. Other devices may be included as well. The DVE detection system <NUM> and/or one or more of its components, is configured for obtaining sensor data indicative of an environment of the vehicle <NUM>. For example, one or more of the LIDAR device, image capture device, and the radar device <NUM> periodically scans an area surrounding the vehicle <NUM>, such as an area corresponding to a planned flight path, to obtain data indicative of aspects of the environment. The sensor data includes three-dimensional (3D) point cloud data, image data, or other data that indicates whether a DVE is present (e.g., detecting a number of objects exceeding an expected number). This information, either in the form of unprocessed sensor data, or as targeted information indicative of the environment surrounding the vehicle <NUM>, is transmitted to the computing device <NUM>. For example, the DVE detection system transmits a representation of the sensor data, a measure of confidence in the obtained data (e.g., an indication that data is noisy, or that one or more sensors have conflicting data), or an indication that a DVE has been detected. As described below, the computing device <NUM> and/or one or more additional components of the vehicle <NUM> may use this information in generating and selecting a trajectory for the vehicle <NUM>.

The trajectory generation system <NUM> includes a trajectory decision module <NUM>. The generation system receives information from the navigation system <NUM> and from the computing device <NUM> to determine a trajectory including a flight path for the aircraft to follow. The trajectory generation system <NUM> may be a standalone computing device, or alternatively be included as part of the computing device <NUM>, the navigation system <NUM>, or any other component of the vehicle <NUM>. Further details relating to the trajectory generation system <NUM> and the trajectory decision module <NUM> are described below with respect to <FIG>.

The steering/hover mechanisms <NUM> include one or more rotors, thrusters, stabilizers, ailerons, elevators, control surfaces, or other controllable operational devices of the vehicle <NUM>. The computing device <NUM> sends control signals to the steering/hover mechanisms <NUM> in order to effectuate navigation, guidance, and/or control of the vehicle <NUM> on a determined trajectory and/or flight path.

<FIG> illustrates a flowchart of a trajectory decision module <NUM> of the vehicle <NUM>, according to an example. In particular, <FIG> shows a logical diagram describing a simplified method for determining a trajectory using the trajectory decision module <NUM>. The trajectory decision module <NUM> can be a software program having instructions stored in memory on the trajectory generation system <NUM> and executable to perform functionality shown in <FIG>. The flowchart representing aspects of this functionality includes blocks <NUM>-<NUM>.

At block <NUM>, functions include detecting a DVE. This may involve comparing sensor data obtained by one or more sensors on the vehicle <NUM> (e.g., one or more sensors of the DVE detection system <NUM>) to expected data, comparing data processing outputs to an expected data processing output, comparing first sensor data from a first sensor to second sensor data from a second sensor, or failing to identify one or more objects or surfaces in the environment using sensor data. Additional details of detecting the DVE are provided below with respect to <FIG>.

At block <NUM>, functions include determining a phase of a route of the vehicle <NUM>. The phase might be a starting phase, an ending phase, or a traveling phase. For example, the starting phase may correspond to a takeoff phase of an aircraft, the ending phase may correspond to a landing phase of the aircraft, and the traveling phase may correspond to a cruising phase of the aircraft. Within examples, the computing device <NUM> may determine the phase based on one or more predetermined waypoints of the route compared to a position of the vehicle <NUM>. Within examples, block <NUM> can be performed prior to or concurrently with block <NUM>. Other blocks in the trajectory decision module <NUM> are performed based on which phase is determined at block <NUM>.

Depending on the determined phase of the route of the vehicle <NUM>, a first segment of a trajectory along which to search for an improved navigation environment. Further details of determining the first segment of the trajectory are described below with respect to <FIG>, and <FIG>.

In the context of a takeoff of the vehicle <NUM>, at block <NUM>, functions include determining whether an improved navigation environment is detected while traveling along the first segment of the trajectory. As used herein, the term "improved navigation environment" refers to determining a threshold increase in similarity between the sensor data and the expected data (e.g., a threshold increase in correlation), an improved data processing output (e.g., a number of detected obstacles falling below a threshold number), a threshold increase in data resolution, a threshold increase in data similarity (e.g., based on comparing data from two or more sensors), or an increased confidence in detecting an object or surface (e.g., using statistical metrics such as a confidence interval, correlation, or <NUM> - p-value) relative to the DVE or relative to one or more benchmarks associated with an improved navigation environment. The improved navigation environment is detected using sensor data from one or more sensors in DVE detection system <NUM>. Responsive to detecting an improved navigation environment, at block <NUM>, functions include determining a new trajectory for the vehicle <NUM>. For example, the new trajectory may start at a position at which the improved navigation environment is detected, and end at a previously defined waypoint on the route of the vehicle <NUM>. In other examples, the new trajectory may include an ascending hover that ends at a predetermined altitude, and which then transitions to a cruising phase of the vehicle <NUM>. Responsive to not detecting the improved navigation environment, at block <NUM>, functions include determining a return trajectory to a starting position of the vehicle <NUM>. For example, the return trajectory may start at an end point of the first segment of the trajectory, and end at a takeoff point of the vehicle <NUM>. The new trajectory or returning to the takeoff point serves as a second segment following the first segment of the trajectory.

In the context of landing the vehicle <NUM>, at block <NUM>, functions include determining whether an improved navigation environment is detected while traveling along the first segment of the trajectory. Responsive to detecting an improved navigation environment, at block <NUM>, functions include following a planned trajectory. For example, the first segment of the trajectory includes an end point at which the vehicle <NUM> determines if an improved navigation environment is detected, and the planned trajectory of block <NUM> may be determined at the same time (or at substantially the same time) as the first segment. If the improved navigation environment is detected, the vehicle <NUM> continues along the planned trajectory. The planned trajectory serves as a second segment following the first segment of the trajectory. Responsive to not detecting the improved navigation environment, at block <NUM>, functions include a providing a decision prompt of the vehicle <NUM>. For example, the decision prompt may include a prompt to switch from an autonomous mode of operation of the vehicle <NUM> to a manual mode of operation of the vehicle <NUM>, a prompt to determine a second segment of the trajectory that deviates from the planned trajectory of block <NUM>, or a prompt to follow the planned trajectory of block <NUM> even without detecting an improved navigation environment. In this manner, the trajectory decision module allows for adaptive contextual control of the vehicle <NUM> after detecting a DVE.

<FIG> illustrates the vehicle <NUM> in a takeoff phase <NUM> of a route with a clear visual environment, according to an example. In particular, <FIG> shows an example scenario in which the vehicle <NUM> does not detect a DVE. In the takeoff phase <NUM>, the vehicle initially rests on a ground surface <NUM>, and begins to approach a predetermined hover point <NUM> at which the predetermined takeoff trajectory <NUM> begins. From a takeoff point <NUM>, the vehicle <NUM> follows a translation path <NUM> in which its position is translated to reach a hover point <NUM>. Though the translation path <NUM> is depicted as including a vertical segment and a horizontal segment, it should be understood that the translation path includes more or fewer than two segments. The hover point <NUM> corresponds to a takeoff waypoint for the vehicle <NUM>. At or before the hover point <NUM>, the vehicle <NUM> rotates to align with a direction of a flight path for the takeoff phase. At or before the hover point, the DVE detection system <NUM> scans in a direction of a takeoff trajectory <NUM>, or more generally in an environment surrounding the vehicle <NUM>, to determine whether a DVE is detected. In some examples, including the example depicted in <FIG>, the environment does not include a DVE, so the vehicle <NUM> follows the takeoff trajectory <NUM>.

<FIG> illustrates the vehicle <NUM> in the takeoff phase <NUM> of a route with a degraded visual environment <NUM>, according to an example. The vehicle <NUM> is depicted as a helicopter, and in the example scenario depicted in <FIG> the degraded visual environment <NUM> may result from dust or sand being kicked up into the air by rotors of the vehicle <NUM>, from fog, rain, snow, or other factors that limit visibility. One or more sensors on vehicle <NUM> scan the environment while vehicle <NUM> is on the ground surface <NUM> or while the vehicle travels towards the hover point <NUM>. In this and other example scenarios, the vehicle <NUM> detects the DVE. Detecting a DVE is described in further detail below with respect to <FIG>.

Responsive to detecting the DVE, the vehicle <NUM> (e.g., trajectory generation system <NUM> of vehicle <NUM>) determines a first segment of a trajectory for handling the DVE, and a second segment for handling the DVE. The first segment and the second segment may be alterations of the translation path <NUM> and the takeoff trajectory <NUM>. For example, rather than approaching the hover point <NUM> using the translation path <NUM>, the vehicle <NUM> may instead follow an upward trajectory while searching for an improved navigation environment. Further details of this process are described below with respect to <FIG>, <FIG>, and <FIG>.

<FIG> illustrates the vehicle <NUM> following a first segment <NUM> of a trajectory according to an example. The first segment <NUM> of the trajectory follows a path between a first waypoint <NUM> and a second waypoint <NUM>. The first waypoint <NUM> corresponds to a point at which the DVE is detected, and the second waypoint <NUM> corresponds to a predetermined altitude, such as a maximum hover height of the vehicle <NUM>. As shown in <FIG>, the vehicle <NUM> travels upward along the trajectory to search for an improved navigation environment. While searching for the improved navigation environment, the one or more sensors on the vehicle <NUM> may continually scan for signs of increased visibility. Detecting an improved navigation environment is described in further detail below with respect to <FIG>.

<FIG> illustrates the vehicle <NUM> following a second segment <NUM> of the trajectory, according to an example. In particular, <FIG> shows the second segment <NUM> after failing to detect an improved navigation environment while following the first segment <NUM> of the trajectory. As shown in <FIG>, the vehicle <NUM> remains within the degraded visual environment <NUM> after reaching the second waypoint <NUM>. After reaching the second waypoint <NUM> of the first segment <NUM>, the vehicle <NUM> determines a return trajectory that starts at the second waypoint <NUM>, and ends at the ground surface <NUM>. The vehicle <NUM> waits for a predetermined period of time prior to making a second attempt at searching for an improved navigation environment. For example, the predetermined time is a default threshold, such as <NUM> minutes, or a threshold associated with a particular type of DVE. For example, if the DVE is determined to be associated with sand or dust, the threshold may be <NUM> minutes, and if the DVE is determined to be associated with snow or fog, the threshold may be longer. Other thresholds can be used for different types of DVEs. In these examples, the vehicle <NUM> also determines a type of DVE using the sensor data, such as by determining that the sensor data closely matches a data signature of a particular type of DVE.

Though <FIG> illustrates the vehicle <NUM> taking a direct path along the second segment <NUM> to the ground surface <NUM>, it should be understood that other paths are possible for the second segment <NUM>. For example, the second segment <NUM> may lead to a last known location without a DVE. In these examples, the vehicle <NUM> may rely on GPS, IMU or other data in order to navigate while ignoring sensor data used to detect the DVE, such as image data or LIDAR data.

<FIG> illustrates the vehicle <NUM> following an alternative second segment <NUM> of the trajectory, according to an example. In some examples, including the example scenario depicted in <FIG>, the vehicle <NUM> detects an improved navigation environment <NUM> prior to reaching the second waypoint <NUM> of the first segment <NUM>. The alternative second segment <NUM> starts at a point <NUM> at which the improved navigation environment <NUM> is detected, and ends at a predetermined waypoint <NUM> that is included in the route of the vehicle <NUM>. For example, predetermined waypoint <NUM> is associated with an end to the takeoff phase of the route.

<FIG> show the vehicle <NUM> in multiple example scenarios associated with a helicopter. It should be understood that similar scenarios may exist for other types of aircraft and rotorcraft, such as a vertical take-off and landing (VTOL) aircraft and short take-off and landing aircraft (STOL). Further, it should be understood that there are alternative contexts for determining trajectories in the presence of a DVE for still other types of aircraft, such as a fixed wing aircraft, and for other types of vehicles, such as land vehicles or marine vehicles.

<FIG> illustrates the vehicle <NUM> in a landing phase <NUM> of a route with a clear visual environment, according to an example. In particular, <FIG> depicts a scenario in which the vehicle <NUM> does not detect a DVE. In the landing phase <NUM>, the vehicle <NUM> follows a trajectory defined by a path <NUM> that starts at a first waypoint <NUM> and ends at a second waypoint <NUM>. The second waypoint <NUM> corresponds to a hover point at which the vehicle maneuvers to reaching a landing location <NUM> on the ground surface <NUM>. Because the vehicle <NUM> does not detect a DVE in the scenario depicted in <FIG>, the vehicle <NUM> continues to travel normally along the route during the landing phase <NUM>.

<FIG> illustrates the vehicle <NUM> in the landing phase <NUM> of a route with a degraded visual environment <NUM>, according to an example. In particular, the scenario depicted in <FIG> shows the vehicle <NUM> having detected a remote DVE. Detecting the DVE in this context may include failing to detect an object or the ground surface <NUM> from an altitude at which such details are expected to be identifiable by one or more sensors on the vehicle <NUM>, detecting an unexpected aspect of the environment, such as detecting more than a threshold/expected number of obstacles in the environment, or receiving a report of a DVE from another vehicle or a remote device. For example, a weather report may indicate that a target landing location of the route is surrounded by heavy fog, rain, a sand storm, or other conditions associated with a DVE. Further details of determining a DVE are described below with respect to <FIG>.

<FIG> illustrates the vehicle <NUM> determining first segment of a trajectory according to an example. In particular, <FIG> depicts a scenario in which the vehicle <NUM> determines an alternative landing location <NUM> that corresponds to an area <NUM> with no identified DVE. The first segment of the trajectory is defined by a path <NUM> that starts at a first waypoint <NUM> and ends at a second waypoint <NUM> associated with the alternative landing location <NUM>. The vehicle <NUM> may include a database of landing locations or have access to a database of landing locations that includes landing ratings for each landing location. For example, landing ratings indicate how many times the vehicle <NUM> or a plurality of vehicles have landed at a given landing location, the reliability of a ground surface at the landing location, typical wind conditions at the landing location, and additional or alternative factors. These factors are scored and weighted to form a landing rating, perhaps using a cost function or another cost-benefit calculation.

Determining the alternative landing location <NUM> is part of determining a trajectory along which to search for an improved navigation environment. For example, in the landing phase <NUM>, determining the first segment may include a decision prompt to an autonomous or manual operator indicating available landing locations and corresponding landing ratings, distance, time, and other considerations in order to determine whether to deviate from the route. The decision prompt may coincide with detecting the degraded visual environment <NUM>, and may either include a query to an autonomous controller of the vehicle <NUM> (e.g., computing device <NUM>) or a request presented on a user interface to a manual operator (e.g., the user interface <NUM>) In view of these considerations, the first segment follows a trajectory towards the alternative landing location <NUM> or another alternative location, or the first segment follows a predetermined trajectory associated with the route (e.g., along a trajectory that follows path <NUM>).

<FIG> illustrates the vehicle <NUM> determining a second segment <NUM> of the trajectory in the degraded visual environment <NUM>, according to an example. In particular, <FIG> depicts a scenario in which the vehicle <NUM> progresses to the landing location <NUM> despite detecting the degraded visual environment <NUM>. In this context, determining a first segment corresponds to following a predetermined route associated with the path <NUM>. While following the path <NUM>, the vehicle <NUM> continues to scan for an improved navigation environment, such as an alternative landing location within an area <NUM> surrounding the landing location <NUM>. In some examples, the first segment diverges from the predetermined route while the vehicle <NUM> searches for an improved navigation environment.

In the scenario depicted in <FIG>, the degraded visual environment <NUM> has not dissipated by the time the vehicle <NUM> approaches the second waypoint <NUM> while following a first segment <NUM> of the trajectory. In this scenario, the vehicle <NUM> enters the degraded visual environment <NUM> responsive to one or more decision prompts, and may involve switching from an autonomous mode of operation to a manual mode of operation, or switching off or ignoring sensor data from one or more sensors (e.g., an RGB camera and/or a LIDAR device) while navigating the vehicle <NUM>. For example, within the degraded visual environment <NUM>, the vehicle relies on GPS data, radar data, or other non-visual data rather than visual data such as RGB camera data or LIDAR data.

After reaching an end of the first segment <NUM> of the trajectory at the second waypoint <NUM>, the vehicle <NUM> follows a second segment <NUM> of the trajectory until reaching the ground surface <NUM>. As shown in <FIG>, the second segment <NUM> might not exactly correspond with the landing location <NUM> due to less visual information being available to the vehicle <NUM> while navigating in the degraded visual environment <NUM>. For example, directly after, or nearly directly after, reaching the second waypoint <NUM>, the vehicle descends towards the ground surface <NUM>.

<FIG> illustrates the vehicle <NUM> determining a second segment <NUM> of the trajectory in a clear environment, according to an example. In particular, <FIG> depicts a scenario in which the vehicle <NUM> follows a first segment <NUM> of a trajectory along the path <NUM> corresponding to an improved navigation environment <NUM>. While following the first segment <NUM>, the vehicle <NUM> monitors sensor data or incoming reports to confirm that the improved navigation environment <NUM> persists, and may re-asses the first segment <NUM> if a DVE is encountered. Accordingly, in this context, the vehicle <NUM> is considered to be "searching" for the improved navigation environment <NUM> at least because an environment surrounding the alternative landing location <NUM> might not be observable at the first waypoint <NUM> (shown in <FIG>). As shown in <FIG>, the second segment <NUM> of the trajectory begins at the second waypoint <NUM> and ends at the alternative landing location <NUM>. In this manner, determining a first segment and second segment of a trajectory during a landing phase <NUM> of a route differs depending on context and based on one or more decision prompts of the vehicle <NUM>, allowing for adaptive trajectories during the landing phase <NUM>.

In alternative examples, the vehicle <NUM> identifies a DVE while approaching a landing location, such as the alternative landing location <NUM>. For example, in examples where the vehicle <NUM> is a helicopter, rotor downwash may disturb dust or sand at the landing location, thereby obscuring relevant visual cues and causing the DVE. In these examples, after detecting the DVE, the vehicle <NUM> selectively ignores the sensor data from one or more sensors after detecting the DVE, and continue along a predetermined trajectory, perhaps relying on other sensor data, such as GPS data, or previously determined information. For example, in <FIG>, the vehicle <NUM> detects the DVE while following the second segment <NUM>, but continue towards the alternative landing location <NUM>. As in the example in <FIG>, the second segment <NUM> might not exactly correspond with the alternative landing location <NUM> due to less sensor data being available to the vehicle <NUM> while navigating in the DVE.

<FIG> illustrates a user interface <NUM>, according to an example. In particular, Figure shows the user interface <NUM> in the context of providing a decision prompt <NUM> after detecting a DVE. In this scenario, the user interface includes the decision prompt <NUM>, a map <NUM>, a plurality of landing ratings <NUM> corresponding to alternative landing locations, and a representation of the DVE <NUM>. Other information, such as priority of landing at a target landing location of the route, fuel status, time cost of re-routing, and other factors may be presented in the user interface <NUM>. This allows a manual operator of the vehicle <NUM> to assess the costs and benefits of following an alternative trajectory. Similar information is considered programmatically by an autonomous controller of the vehicle <NUM> in order to determine whether to generate and follow an alternative trajectory to a different landing location.

Though the decision prompt <NUM> relates to updating a route of a vehicle <NUM>, other decision prompts may be presented to an operator of the vehicle. For example, after the first segment of a trajectory ends in a DVE, a secondary prompt to switch from an autonomous mode of operation to a manual mode of operation may be presented, or a prompt to switch off or ignore certain sensor data may be presented. In this manner, multiple decision prompts may be provided to an operator to allow a robust system of handling a detected DVE.

<FIG> illustrates a flowchart of a method <NUM> of controlling a vehicle in a degraded visual environment, according to an example. Method <NUM> shown in <FIG> presents an example of a method that could be used with the vehicle <NUM> shown in <FIG>, or with components of vehicle <NUM>, such as the computing device <NUM> described with respect to <FIG>. Further, devices or systems may be used or configured to perform logical functions presented in <FIG>. In some instances, components of the devices and/or systems may be configured to perform the functions such that the components are actually configured and structured (with hardware and/or software) to enable such performance. In other examples, components of the devices and/or systems may be arranged to be adapted to, capable of, or suited for performing the functions, such as when operated in a specific manner. Method <NUM> may include one or more operations, functions, or actions as illustrated by one or more of blocks <NUM>-<NUM>. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and/or in a different order than those described herein. Also, the various blocks may be combined into fewer blocks, divided into additional blocks, and/or removed based upon the desired implementation.

It should be understood that for this and other processes and methods disclosed herein, flowcharts show functionality and operation of one possible examples. In this regard, each block or portions of each block represent a module, a segment, or a portion of program code, which includes one or more instructions executable by a processor for implementing specific logical functions or steps in the process. The program code may be stored on any type of computer readable medium or data storage, for example, such as a storage device including a disk or hard drive. Further, the program code can be encoded on a computer-readable storage media in a machine-readable format, or on other non-transitory media or articles of manufacture. The computer readable medium may include non-transitory computer readable medium or memory, for example, such as computer-readable media that stores data for short periods of time like register memory, processor cache and Random Access Memory (RAM). The computer readable medium may also include non-transitory media, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. The computer readable medium may be considered a tangible computer readable storage medium, for example.

In addition, each block or portions of each block in <FIG>, and within other processes and methods disclosed herein, may represent circuitry that is wired to perform the specific logical functions in the process. Alternative implementations are included within the scope of the examples of the present disclosure in which functions may be executed out of order from that shown or discussed, including concurrent, substantially concurrent or in reverse or substantially order, depending on the functionality involved, as would be understood by those reasonably skilled in the art.

At block <NUM>, the method <NUM> includes identifying a degraded visual environment <NUM> corresponding to a phase of a route followed by the vehicle <NUM>. For example, identifying the degraded visual environment <NUM> may be performed by the DVE detection system <NUM>, as described above with respect to <FIG>.

At block <NUM>, the method <NUM> includes determining, based on the phase of the route, a first segment <NUM> of a trajectory of the vehicle <NUM> along which to search for a location with an improved navigation environment <NUM>. For example, in a takeoff phase <NUM>, the first segment <NUM> includes an upwards trajectory, and in a landing phase <NUM>, the first segment may include a trajectory towards a selected landing location.

At block <NUM>, the method <NUM> includes causing the vehicle <NUM> to follow the first segment <NUM> until: (i) identifying the improved navigation environment <NUM>, or (ii) reaching an end of the first segment <NUM> without identifying the improved navigation environment <NUM>. In the takeoff phase <NUM>, this involves searching for the improved navigation environment <NUM> until reaching a predetermined altitude (e.g., a maximum hovering altitude of the vehicle <NUM>). In the landing phase <NUM>, this involves searching for confirmation that an alternative landing location (e.g., the alternative landing location <NUM>) corresponds to an improved navigation environment (e.g., the improved navigation environment <NUM>).

Within examples, detecting an improved navigation environment includes comparing sensor data to one or benchmarks for the sensor data, such as comparing first sensor data from a first sensor (e.g., a LIDAR) to second sensor data from a second sensor (e.g., a radar), and confirming that the first sensor data and the second sensor data represent the environment in a similar manner (e.g., within a threshold level of variation, such as with a correlation of. <NUM> or greater). In further examples, detecting the improved navigation environment involves determining a change in data processing outputs using the sensor data. For example, this involves confirming that a surface (e.g., ground surface <NUM>) or object is identifiable using sensor data, perhaps after previously failing to identify the surface or obj ect. In other examples, determining a change in data processing outputs includes determining a change in a number of detected obstacles using the sensor data. For example, if the DVE is determined based on an excess number of obstacles are detected in the environment (e.g., a detected number of obstacles exceeding a threshold number), then a reduction in the number of detected obstacles (e.g., a reduced number that is less than the threshold number) indicates an improved navigation environment. Within examples, determining the improves visual environment can be based on a type of DVE or a metric used for identifying the DVE. Other metrics of increased visibility relative to the DVE are possible.

At block <NUM>, the method <NUM> includes determining a second segment (e.g., the alternative second segment <NUM>) of the trajectory based on whether the improved navigation environment <NUM> has been identified. For example, in the takeoff phase <NUM>, this involves determining whether the improved navigation environment <NUM> has been identified before reaching the predetermined altitude. In the landing phase <NUM>, this involves determining whether to follow a planned trajectory (e.g., the planned route of block <NUM>) or considering a decision prompt in order to determine the second segment (e.g., the second segment <NUM>).

At block <NUM>, the method <NUM> includes causing the vehicle to follow the second segment. Determining the first segment and the second segment in this manner allows the vehicle <NUM> to adaptively and reliably navigate within the degraded visual environment <NUM>.

Within examples, identifying the degraded visual environment includes receiving sensor data obtained by one or more sensors (e.g., one or more sensors of the DVE detection system <NUM>) on the vehicle <NUM>, determining a difference between the sensor data and expected data, and identifying the degraded visual environment based on determining the difference between the sensor data and the expected data. For example, the expected data includes a data signature for each type of sensor that indicates a clear visual environment.

Within examples, identifying the degraded visual environment includes receiving sensor data obtained by one or more sensors (e.g., one or more sensors of the DVE detection system <NUM>) on the vehicle <NUM>, determining a data processing output (e.g., detecting one or more obstacles using the sensor data), comparing the data processing output to an expected data processing output (e.g., a threshold number of detected obstacles, such as a <NUM> detected obstacles), and identifying the degraded visual environment based on comparing the data processing output to the expected data processing output. For example, if the vehicle <NUM> appears to be surrounded by a number of obstacles that exceeds the threshold number based on the sensor data, then a degraded visual environment can be identified. These steps reveal a degraded visual environment because some types of DVEs can cause visual sensors to provide data that appears to show many obstacles in the environment that are not actually present in the environment. Rather than allowing falsely detected obstacles to impact navigation of the vehicle <NUM>, the vehicle can identify the DVE based on a data processing output, and react accordingly, such as by searching for an improved navigation environment within predetermined constraints.

Within examples, the one or more sensors includes a Light Detection and Ranging (LIDAR) device, and determining the difference between the sensor data and the expected data includes determining a difference between a number of returning light pulses represented by the sensor data and an expected number of returning light pulses. For example, a threshold percentage (e.g., <NUM>%) of returning light pulses may be used for the comparison, and a threshold deviation (e.g., more than ± <NUM>%) of returning light pulses indicates the degraded visual environment <NUM>.

Within examples, the one or more sensors includes a camera (e.g., an RGB camera), and determining the difference between the sensor data and the expected data includes determining a lack of variation in pixel intensities relative to the expected data. For example, a histogram of the pixel intensities may not follow an expected distribution, and instead have mostly similar pixel intensities that indicate scattered or ambient lighting associated with the degraded visual environment <NUM>.

Within examples, identifying the degraded visual environment <NUM> includes receiving sensor data obtained by one or more sensors on the vehicle <NUM>, searching for an identifiable surface (e.g., the ground surface <NUM>) from the sensor data, and identifying the degraded visual environment based on the search. For example, this involves failing to identify the ground surface <NUM> from a predetermined distance (e.g., <NUM> feet (<NUM>)) may be associated with the degraded visual environment.

Within examples receiving sensor data obtained by one or more sensors on the vehicle <NUM>, and identifying the degraded visual environment <NUM> based on determining that the sensor data matches a data signature corresponding to a type of degraded visual environment. For example, a histogram of pixel intensities of a camera match a histogram a given source of a DVE (e.g., dust, fog, etc.), or a distribution of detected distances from a LIDAR device match a distribution of distances of a given source of a DVE. Within examples, the data signature comprises a noise signature associated with the one or more sensors. For example, data associated with a DVE may manifest as Gaussian noise, salt-and-pepper noise, shot noise, or another type of noise signature.

Within examples, identifying the degraded visual environment <NUM> includes receiving an indication of an area <NUM> having the degraded visual environment <NUM>, and determining that a waypoint (e.g., the second waypoint <NUM>) associated with the first segment of the trajectory falls within the area <NUM>. For example, the indication of the area <NUM> may be a weather report or visibility report received from another vehicle or another computing device, such as an air traffic controller.

Within examples, identifying the degraded visual environment <NUM> environment includes receiving sensor data obtained by one or more sensors on the vehicle, <NUM> wherein the sensor data comprises first sensor data from a first sensor (e.g., an RGB camera) and second sensor data from a second sensor (e.g., a radar), and identifying the degraded visual environment based on a comparison between the first sensor data and the second sensor data. For example, the second sensor data indicates an object or surface that is not indicated by the first sensor data, or the second sensor data may render into a representation of the environment that does not match a representation of the environment rendered from the first sensor data.

Within examples, the first sensor data may be associated with a first category of sensors. For example the first category of sensors may be a visual sensor that converts light rays into electronic signals and outputs a 2D pixel array representation of a measured 3D space. In this context, the term "visual" relates to light being readable/detectable by the sensor, and the term "light" includes the visible spectrum as well as the infrared spectrum. Further, the second sensor data may be associated with a second category of sensors. The second category of sensors may be a non-visual sensor that does not resolve detected light into a 2D pixel array representation of a measured 3D space. This may include sound wave detectors, magnetic sensors, pressure sensors, temperature sensors, or other sensors that do not correspond to the first category of sensors. In other examples, the first sensor data and the second sensor data may correspond to two different sensors, which provides redundancy in detecting a DVE, detecting an improved navigation environment after detecting the DVE, verifying DVE detections between the first and second sensors data, identifying failure of one of the first and second sensors based on a comparison between the first and second sensor data, or enhancing the accuracy of DVE detection using two different characteristics of the sensor data from the first and second sensors. Some examples of combinations of sensors include an electro-optical (EO) sensor and a short wave infrared (SWIR) sensor, a synthetic aperture radar (SAR) sensor and a LIDAR sensor, a SAR sensor and an EO sensor, a SAR sensor and a SWIR sensor, two EO sensors, two LIDAR sensors. Some sensors have differing ability to view an environment when the vehicle <NUM> is in a DVE. For example, a long wave infrared (LWIR) sensor have better ability to see through the DVE than a LIDAR sensor, while the LIDAR sensor has a higher resolution that the LWIR sensor, and these can be paired together when determining the second segment of the trajectory. Similarly, a LWIR sensor can be paired with an EO sensor, or with an SWIR sensor. Still further, different sensors having relatively high ability to view the environment during the DVE but having relatively low resolutions can be paired together. For example, a SAR sensor can be paired with a LWIR sensor. Other combinations of sensors that provide the first sensor data and the second sensor data are possible.

Within examples, the vehicle <NUM> includes an aircraft and the phase of the route is a takeoff phase <NUM>. Within these examples, determining the first segment <NUM> of the trajectory includes determining an upward trajectory that starts at a takeoff location and ends at a predetermined waypoint. Within these examples, determining the second segment <NUM> of the trajectory based on whether the improved navigation environment has been identified includes determining that the improved navigation environment has not been identified while following the first segment <NUM> of the trajectory, and responsive to determining that the improved navigation environment has not been identified while following the first segment <NUM> of the trajectory, determining a downward trajectory that starts at the predetermined waypoint and ends at the takeoff location. For example, this corresponds to the examples illustrated in <FIG> and <FIG>.

Within examples, the vehicle <NUM> includes an aircraft and the phase of the route is a landing phase <NUM>. In these examples, determining the first segment of the trajectory includes determining a downward trajectory that starts at a current position of the aircraft and ends at a landing waypoint disposed above a ground surface <NUM>.

Within examples, the vehicle <NUM> includes an aircraft navigating towards a landing location <NUM>, and the phase of the route is a landing phase <NUM>. In these examples, the method <NUM> further includes determining a landing rating for the landing location <NUM> based on previous landings of one or more aircrafts at the landing location. For example the aircraft includes or has access to a database of landing locations and landing ratings corresponding to the landing locations, and determining the landing ratings includes retrieving a landing rating for the landing location <NUM>. Determining the first segment of the trajectory can include setting an alternative landing location (e.g., the alternative landing location <NUM>) based on (i) the landing rating being less than a threshold landing rating (e.g., less than <NUM> out of <NUM>), and (ii) the degraded visual environment <NUM>. In these examples, determining the first segment <NUM> of a trajectory includes determining a trajectory that starts at a current position of the aircraft and ends at a second waypoint <NUM> associated with the alternative landing location <NUM>.

Within examples, the method <NUM> further includes, responsive to determining the degraded visual environment, providing a prompt (e.g., the decision prompt <NUM>) by way of a user interface <NUM> of the vehicle <NUM>. In these examples, the prompt relates to (i) the degraded visual environment <NUM>, and (ii) an option to set the first segment <NUM> of the trajectory and the second segment <NUM> of the trajectory.

Within examples, the method <NUM> further includes, responsive to determining the degraded visual environment <NUM>, switching from an autonomous mode of operation of the vehicle <NUM> to a manual mode of operation of the vehicle <NUM>.

The described systems and methods described herein provide functionality that enables autonomous takeoff and landing of a vehicle in a degraded visual environment. Determining segments of a trajectory of the vehicle depending on context and the degraded visual environment allows for a robust, consistent, and adaptive manner of handling different types of visual states, and provides both autonomous and manual operators ways of dynamically selecting and updating routes of the vehicle.

Though the contexts of operating vehicles provided herein generally depict or describe an aircraft, such as a helicopter, it should be understood that similar contexts may arise for other types of vehicles. For example, a land vehicle may also encounter a DVE, and may handle its route differently depending on a current phase (e.g., a starting phase or an ending phase) of the route. Accordingly, the above-described embodiments are not limited to those involving an aircraft, but can more generally be applied to vehicles.

By the term "substantially," "similarity," and "about" used herein, it is meant that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including for example, tolerances, measurement error, measurement accuracy limitations and other factors known to skill in the art, may occur in amounts that do not preclude the effect the characteristic was intended to provide.

Different examples of the system(s), device(s), and method(s) disclosed herein include a variety of components, features, and functionalities. It should be understood that the various examples of the system(s), device(s), and method(s) disclosed herein may include any of the components, features, and functionalities of any of the other examples of the system(s), device(s), and method(s) disclosed herein in any combination or any sub-combination, and all of such possibilities are intended to be within the scope of the disclosure.

Claim 1:
A method (<NUM>) for controlling a vehicle (<NUM>) in a degraded visual environment (<NUM>), comprising:
identifying (<NUM>) a degraded visual environment (<NUM>) corresponding to a phase of a route followed by the vehicle (<NUM>);
determining(<NUM>), based on the phase of the route, a first segment (<NUM>) of a trajectory of the vehicle (<NUM>) along which to search for a location with an improved navigation environment (<NUM>);
causing (<NUM>) the vehicle (<NUM>) to follow the first segment (<NUM>) until:
(i) identifying the improved navigation environment (<NUM>), or
(ii) reaching an end of the first segment (<NUM>) without identifying the improved navigation environment (<NUM>);
determining (<NUM>) a second segment (<NUM>) of the trajectory based on whether the improved navigation environment (<NUM>) has been identified; and
causing (<NUM>) the vehicle (<NUM>) to follow the second segment (<NUM>); wherein:
identifying (<NUM>) the degraded visual environment (<NUM>) comprises:
receiving sensor data obtained by one or more sensors on the vehicle (<NUM>);
determining a difference between the sensor data and expected data; and
identifying the degraded visual environment (<NUM>) based on determining the difference between the sensor data and the expected data; and
the one or more sensors comprise a Light Detection and Ranging (LIDAR) device (<NUM>), and determining the difference between the sensor data and the expected data comprises determining a difference between a number of returning light pulses represented by the sensor data and an expected number of returning light pulses.