Patent Description:
Aerial vehicles, such as vertical takeoff and landing (VTOL) aircraft, can be navigated close to obstacles to complete aerial tasks. It is challenging and hazardous for pilots to navigate such aerial vehicles close to obstacles, however doing so may be unavoidable.

Pilots of aerial vehicles may be tasked with flying close to one or more obstacles, navigating hazardous pathways, or performing takeoff or landing procedures while positioned close to potential obstacles. These pilots take great care to prevent the aerial vehicle from colliding with obstacles in their environment when performing these hazardous tasks. Because pilots are typically attending to other tasks while flying aerial vehicles, maintaining control over an aerial vehicle while navigating close to environmental obstacles may be unmanageable or generally unsafe. <CIT> describes a system for controlling an unmanned aerial vehicle within an environment, the system comprising: one or more sensors carried on the unmanned aerial vehicle and configured to receive sensor data of the environment; and one or more processors individually or collectively configured to: determine, based on the sensor data, an environmental complexity factor representative of an obstacle density for the environment, determine, based on the environmental complexity factor, one or more operating rules for the unmanned aerial vehicle, receive a signal indicating a desired movement of the unmanned aerial vehicle, and cause the unmanned aerial vehicle to move in accordance with the signal while complying with the one or more operating rules. <CIT> describes a method of vehicle collision avoidance, comprising: receiving sensor data comprising one or more points; determining a distance and a direction from a vehicle to the one or more points; determining a velocity constraint for each of the one or more points based on the determined distance and direction from the vehicle to each of the one or more points; determining one or more velocity solutions that satisfy the determined velocity constraints based on a vehicle navigation instruction; selecting a velocity solution from the determined one or more velocity solutions based on the vehicle navigation instruction; and adjusting a vehicle velocity based on the selected velocity solution. The present invention is specified in the claims.

The systems and methods of this technical solution provide an automatic control mode for an aerial vehicle that engages when a pilot navigates close to obstacles in the environment. The systems and methods described herein leverage existing sensor data positioned on the aircraft, and in some implementations existing world models of the environment, to detect objects that are proximate to the aircraft. If an object is detected, the systems and methods described herein can create a speed constraint for the aircraft in the direction of the object, which prevents the pilot from colliding with the obstacle while allowing the pilot to otherwise freely navigate the aerial vehicle as intended. The speed constraint, and the constrained direction, can be dynamically updated as the aerial vehicle is piloted through the environment. The present techniques provide improvements to the safety and stability of aerial vehicles while navigating near obstacles.

At least one aspect of the present disclosure is directed to a method for controlling an aerial vehicle to avoid obstacles. The method can be performed, for example, by one or more processors coupled to memory of the aerial vehicle. The method can include detecting, based on a world model generated from sensor data captured by one or more sensors positioned on the aerial vehicle during flight, an obstacle for the aerial vehicle. The method can include triggering an augmented manual control mode responsive to a speed of the aerial vehicle being less than a predetermined threshold and detecting the obstacle. The method can include setting, responsive to triggering the augmented manual control mode, a speed constraint for the aerial vehicle in a direction of the obstacle based on a distance between the aerial vehicle and the obstacle. The method can include receiving, via an inceptor of the aerial vehicle, an instruction to navigate the aerial vehicle in the direction at a second speed greater than the speed constraint. The method can include adjusting the instruction to replace the second speed with the speed constraint, to cause the aerial vehicle to navigate in the direction at the speed constraint instead of the second speed.

In some implementations, the one or more sensors can include one or more of a light detection and ranging (LiDAR) sensor, a radar sensor, or a camera. In some implementations, the obstacle can be an along-track obstacle. In some implementations, the method can include detecting a cross-track obstacle for the aerial vehicle in a second direction that is at least partially opposite of the direction. In some implementations, the method can include adjusting a second instruction received via the inceptor of the aerial vehicle to limit a third speed for the aerial vehicle in the second direction, to guide the aerial vehicle between the along-track obstacle and the cross-track obstacle.

In some implementations, adjusting the instruction to navigate the aerial vehicle is further based on one or more attributes of the aerial vehicle. In some implementations, the method can include determining that the distance is less than a predetermined distance threshold. In some implementations, the method can include setting the speed constraint to prevent movement in the direction responsive to determining that the distance is less than the predetermined distance threshold. In some implementations, the method can include providing an alert to a pilot of the aerial vehicle responsive to setting the speed constraint. In some implementations, the alert can be at least one of haptic feedback, an audio alert, or a visual alert presented on a display of the aerial vehicle. In some implementations, the method can include adjusting an inceptor constraint to limit physical movement of the inceptor based on the speed constraint.

In some implementations, the method can include setting, responsive to triggering the augmented manual control mode, an attitude constraint for the aerial vehicle based on the direction to the obstacle and the distance. In some implementations, the method can include adjusting, responsive to receiving a second instruction to change an attitude of the aerial vehicle, the second instruction to replace the change of the attitude with the attitude constraint. In some implementations, the method can include setting, responsive to triggering the augmented manual control mode, a vertical acceleration constraint for the aerial vehicle based on the direction to the obstacle and the distance. In some implementations, the method can include adjusting, responsive to receiving a second instruction to change a vertical acceleration of the aerial vehicle, the second instruction to replace the change of the vertical acceleration with the vertical acceleration constraint. In some implementations, setting the speed constraint for the aerial vehicle can include setting a respective speed constraint for each of an x-axis direction, a y-axis direction, and a z-axis direction.

At least one other aspect of the present disclosure is directed to a system for controlling an aerial vehicle to avoid obstacles. The system can include, for example, one or more processors coupled to memory of the aerial vehicle. The system can detect, based on a world model generated from sensor data captured by one or more sensors positioned on the aerial vehicle during flight, an obstacle for the aerial vehicle. The system can trigger an augmented manual control mode responsive to a speed of the aerial vehicle being less than a predetermined threshold and detecting the obstacle. The system can set, responsive to triggering the augmented manual control mode, a speed constraint for the aerial vehicle in a direction of the obstacle based on a distance between the aerial vehicle and the obstacle. The system can receive, via an inceptor of the aerial vehicle, an instruction to navigate the aerial vehicle in the direction at a second speed greater than the speed constraint. The system can adjust the instruction to replace the second speed with the speed constraint, to cause the aerial vehicle to navigate in the direction at the speed constraint instead of the second speed.

In some implementations, the one or more sensors can include one or more of a LiDAR sensor, a radar sensor, or a camera. In some implementations, the obstacle can be an along-track obstacle. In some implementations, the system can detect a cross-track obstacle for the aerial vehicle in a second direction that is at least partially opposite the direction. In some implementations, the system can adjust a second instruction received via the inceptor of the aerial vehicle to limit a third speed for the aerial vehicle in the second direction, to guide the aerial vehicle between the along-track obstacle and the cross-track obstacle. In some implementations, the system can adjust the instruction to navigate the aerial vehicle further based on one or more attributes of the aerial vehicle.

In some implementations, the system can determine that the distance is less than a predetermined distance threshold. In some implementations, the system can set the speed constraint to prevent movement in the direction responsive to determining that the distance is less than the predetermined distance threshold. In some implementations, the system can provide an alert to a pilot of the aerial vehicle responsive to setting the speed constraint, wherein the alert is at least one of haptic feedback, an audio alert, or a visual alert presented on a display of the aerial vehicle. In some implementations, the system can adjust an inceptor constraint to limit physical movement of the inceptor based on the speed constraint.

In some implementations, the system can set, responsive to triggering the augmented manual control mode, an attitude constraint for the aerial vehicle based on the direction to the obstacle and the distance. In some implementations, the system can adjust, responsive to receiving a second instruction to change an attitude of the aerial vehicle, the second instruction to replace the change of the attitude with the attitude constraint. In some implementations, the system can generate the world model based on the sensor data captured by the one or more sensors positioned on the aerial vehicle. In some implementations, the system can set the speed constraint for the aerial vehicle by setting a respective speed constraint for each of an x-axis direction, a y-axis direction, and a z-axis direction.

Aspects can be combined and it will be readily appreciated that features described in the context of one aspect of the invention can be combined with other aspects. Aspects can be implemented in any convenient form. For example, by appropriate computer programs, which may be carried on appropriate carrier media (computer readable media), which may be tangible carrier media (e.g. disks or other non-transitory storage media) or intangible carrier media (e.g. communication signals). Aspects may also be implemented using suitable apparatus, which may take the form of programmable computers running computer programs arranged to implement the aspects. As used in the specification and in the claims, the singular form of "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context indicates otherwise. The illustrative implementations described in the detailed description, drawings, and claims are not meant to be limiting. Other implementations may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and make part of this disclosure.

For purposes of reading the description of the various embodiments below, the following descriptions of the sections of the specification and their respective contents may be helpful:.

The systems, apparatuses, and methods described herein allow for augmented manual control modes for aerial vehicles, which allow pilots to safely navigate aerial vehicles close to obstacles. In many types of aerial missions or tasks, a pilot may operate an aerial vehicle in close proximity to obstacles in the environment. The techniques described herein can automatically impose constraints on manual input provided by pilots when an aerial vehicle is close to obstacles.

Obstacles can be detected by leveraging existing sensor systems that are positioned on the aerial vehicles. Such on-aircraft sensor systems can include light detection and ranging (LiDAR), visible light, infrared light sensors, accelerometers, gyroscopes, magnetometers, or any other sensor that is capable of detecting objects or a status of the aerial vehicle in the environment. Using these sensors, information from the surrounding environment can be continually retained and updated as part of a comprehensive world model within the memory of the aerial vehicle. The systems and methods described herein can then use this world model to determine the distance and direction from the aerial vehicle to proximate objects.

For example, the systems and methods described herein can automatically assign a speed constraint for a particular direction based on the position of a detected obstacle relative to the position of the aerial vehicle. When a pilot attempts to navigate the aerial vehicle in the direction of the obstacle at a speed that exceeds the constraint, the systems and methods described herein can automatically adjust the instruction to conform to the constraint. Constraints other than speed, such as attitude (e.g., roll, pitch, and yaw) constraints or vertical acceleration constraints may also be implemented. By constraining the movement of the aerial vehicle in the direction of obstacles, the systems and methods described herein allow pilots to navigate freely in other directions while protecting the aerial vehicle in the event of a potentially hazardous input.

The motion constraints imposed on the aerial vehicle can be updated over time, allowing pilots to slowly and safely approach obstacles without resulting in a collision. This allows for the completion of missions or tasks in which the aerial vehicle is in close proximity to objects in the environment without completely constraining manual movement input, and without requiring the pilot to continuously perform the high-workload task of managing vehicle position with respect to obstacles while engaging in other activities. The systems and methods described herein therefore provide improvements to conventional aerial vehicle control systems by providing an augmented manual control mode, which can allow aerial vehicles to be safely navigated close to obstacles in an environment without restricting other movement commands. These and other improvements are detailed herein.

Referring now to <FIG>, illustrated is a block diagram of an example system <NUM> for controlling an aerial vehicle to avoid obstacles, in accordance with one or more implementations. In brief overview, the system <NUM> can include at least one aerial vehicle <NUM> and one or more obstacles <NUM>. The aerial vehicle <NUM> can include at least one vehicle management system <NUM>, at least one memory <NUM>, one or more sensors <NUM>, one or more display devices <NUM>, and one or more vehicle control devices <NUM>. The memory <NUM> can store at least one world model <NUM>, obstacle data <NUM>, and one or more constraints <NUM>. The vehicle management system <NUM>, the memory <NUM>, the sensors <NUM>, the display devices <NUM>, and the vehicle control devices <NUM> may be connected or communicatively coupled with one another. The vehicle management system <NUM>, the memory <NUM>, the sensors <NUM>, the display devices <NUM>, and the vehicle control devices <NUM> may be implemented at least in part using one or more of the components of the computer system <NUM> described in connection with Section B.

The aerial vehicle <NUM> can be any type of aerial vehicle, such as a helicopter, a drone, a vertical take-off and landing (VTOL) aircraft, or any other type of aerial vehicle. The aerial vehicle <NUM> can be operated by one or more pilots, for example, using the vehicle control devices <NUM>. The aerial vehicle <NUM> can be dispatched to address one or more remote issues by completing one or more assigned tasks, which may be performed in close proximity to environmental obstacles. Such assigned tasks can include, for example, combat tasks or taking off or landing in precarious environments, such as oil rigs. The aerial vehicle <NUM> may be equipped with any type of device or system that may be used to complete a task or mission that has been assigned to the aerial vehicle. The aerial vehicle may include one or more communications interfaces (not pictured), via which the aerial vehicle <NUM> or the vehicle management system <NUM> may transmit information, including any of the information used in the techniques described herein. For example, the aerial vehicle <NUM> may communicate the obstacle data <NUM> or the constraints <NUM> to one or more remote devices (not pictured), which may include other aerial vehicles or computing systems in communication with the aerial vehicle <NUM>.

The sensors <NUM> can be mounted on the interior or the exterior of the aerial vehicle <NUM>. Non-limiting examples of the sensors <NUM> include LiDAR sensors, visible light sensors (e.g., cameras, video capture devices, etc.), infrared light sensors, accelerometers, gyroscopes, magnetometers, elevation sensors, pressure sensors, temperature sensors, force sensors, proximity sensors, radar sensors, angle-of-attack sensors, global positioning system (GPS) sensors, thermal infrared cameras, and thermal imaging cameras, among others. Sensor information can be retrieved from the sensors <NUM> by the vehicle management system <NUM>. In some implementations, one or more of the sensors <NUM> can provide sensor data periodically (e.g., in a batch transmission, etc.) to the vehicle management system <NUM>. In some implementations, one or more of the sensors <NUM> can provide sensor data to the vehicle management system <NUM> upon receiving a corresponding request for sensor data from the vehicle management system <NUM>.

The display devices <NUM> can include any type of device capable of presenting information to a pilot (not pictured) that operates the aerial vehicle <NUM>. The display devices <NUM> can be positioned within the aerial vehicle <NUM> such that they can be viewed while the pilot is operating the aerial vehicle <NUM>. The display devices <NUM> can include devices that present specific sensor information, such as speed, direction, velocity, or location. The display devices <NUM> can also be general display devices that can present information used in the techniques described herein, such as the obstacle data <NUM> or the constraints <NUM>. The display devices <NUM> can include an electronic flight instrument system (EFIS), which can include one or more primary flight displays, one or more multi-function displays, or other displays. Types of display devices can include liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, bi-stable displays (e.g., e-ink, etc.), among others. As described herein, the vehicle management system <NUM> may present alerts or other information related to the constraints <NUM> or an indication an augmented manual control mode being activated.

The vehicle control devices <NUM> can be any type of control device that is capable of maneuvering the aerial vehicle <NUM>. Some non-limiting examples of the vehicle control devices <NUM> can include inceptors, cyclic controls, collective levers, pedals, throttles, VTOL unmanned aerial vehicle (UAV) controls, or any other type of aerial vehicle control device. The vehicle control devices <NUM> can be operated by a pilot of the aerial vehicle to send navigation or maneuver instructions to the vehicle management system, which can change the speed, attitude, or vertical acceleration of the aerial vehicle <NUM> according to the constraints <NUM>. In implementations where the aerial vehicle is a UAV, the vehicle control devices <NUM> (and in some implementations, components of the vehicle management system <NUM>) may be remote from the aerial vehicle <NUM>, and may communicate flight commands to the aerial vehicle <NUM> to maneuver the aerial vehicle <NUM> according to pilot instructions. The vehicle control devices <NUM> may include haptic feedback devices that indicate one or more alerts to the pilot, such as alerts that an augmented manual control mode has been triggered, or alerts that the maneuverability of the vehicle has been limited according to one or more of the constraints <NUM>. The vehicle control devices <NUM> can include configurable (e.g., configurable by the vehicle management system <NUM>, etc.) actuators that prevent physical movement of the vehicle control devices <NUM> according to the constraints <NUM>. For example, the vehicle control devices <NUM> may include actuators that prevent physical movement of the vehicle control devices <NUM> in directions, or at magnitudes, that would cause the aerial vehicle <NUM> to exceed one or more of the constraints <NUM>.

The memory <NUM> can be a computer-readable memory that can store or maintain any of the information described herein. The memory can maintain one or more data structures, which may contain, index, or otherwise store each of the values, pluralities, sets, variables, vectors, numbers, or thresholds described herein. The memory <NUM> can be accessed using one or more memory addresses, index values, or identifiers of any item, structure, or region maintained in the memory <NUM>. The memory <NUM> can be accessed by the vehicle management system <NUM>. In some implementations, other computing devices present on the aerial vehicle <NUM> can access the memory <NUM>. The vehicle management system <NUM> can store the results of any or all computations in one or more data structures in the memory <NUM>, such that the data structures are indexed or identified with appropriate values for later access.

The memory <NUM> can store a world model <NUM>, for example, in one or more data structures. The vehicle management system <NUM> can retrieve sensor data from the sensors <NUM> to generate the world model <NUM>. The world model <NUM> can be a collection of related sensor information that models the environment surrounding the aerial vehicle. An example plot of a distance field of obstacles, which may be part of the world model <NUM>, is shown in <FIG>. The sensor information in the world model <NUM> can be stored in association with one or more timestamps, identifying the time the sensors <NUM> captured the particular sensor information. The world model <NUM> can be used by the vehicle management system <NUM> to detect the distance between the aerial vehicle <NUM> and one or more obstacles <NUM>, as well as a direction from the aerial vehicle <NUM> to the one or more obstacles <NUM>. The world model <NUM> can include a corresponding record of one or more obstacles in the environment surrounding the aerial vehicle <NUM>. The world model <NUM> may also include collection of different collections of sensor data that represent the environment surrounding the aerial vehicle, including height maps, temperature maps, red-green-blue (RGB) channels (e.g., from visible light cameras), terrain information, weather information, or visibility information, among any other type of sensor data. The world model <NUM> can be iteratively updated by the vehicle management system <NUM> as additional sensor information is captured and processed by the vehicle management system <NUM>. One or more portions of the world map <NUM> may be presented to the pilot via the display devices <NUM>.

The memory <NUM> can store obstacle data <NUM>, for example, in one or more data structures. The obstacle data may be stored by the vehicle management system <NUM> upon detecting an obstacle <NUM> based on the sensor data in the world model <NUM>. The obstacle data <NUM> can be a corresponding record of information used in the techniques described herein for each obstacle. For example, the obstacle data <NUM> can include a position of an obstacle <NUM>, a direction that, if traveled by the aerial vehicle <NUM>, would eventually result in a collision between the obstacle <NUM> and the aerial vehicle <NUM>, a distance between the aerial vehicle <NUM> and the obstacle <NUM>, a classification of the obstacle data <NUM>, or any other information about an obstacle <NUM> that may be derived from the data produced by the sensors <NUM>. In some implementations, the obstacle data <NUM> may include velocity or acceleration data for one or more obstacles <NUM>, for example, if the obstacle <NUM> is a moving object. Corresponding obstacle data <NUM> may be stored and updated for each obstacle <NUM> detected from the world model <NUM>.

The memory <NUM> can store one or more constraints <NUM>, for example, in one or more data structures. The constraints <NUM> may be any type of constraint that may be applied to one or more maneuvers performed by the aerial vehicle <NUM> during its operation. One example of a constraint <NUM> includes a speed constraint, which may constrain the speed of the aerial vehicle <NUM> in a particular direction, as determined by the vehicle management system <NUM>. Another type of constraint <NUM> may be an attitude (e.g., roll, pitch, or yaw, etc.) constraint (or an attitude rate constraint), which constrains rotational motion (which through operation of the aerial vehicle, may translate to translational motion in the environment) of the aerial vehicle <NUM>. Another example of a constraint is a vertical acceleration or vertical movement constraint. Such constraints may limit vertical movement speed of the aerial vehicle <NUM>. One or more constraints <NUM> may be implemented at the same time to limit multiple aspects of the movement of the aerial vehicle <NUM>. For example multiple speed constraints may be active if the aerial vehicle is flying between two obstacles <NUM>. Similarly, in some implementations, both an attitude and a speed constraint may be implemented at the same time. Other combinations of multiple constraints <NUM> are also contemplated, and any combination of constraints <NUM> may be generated, updated, or used, as described herein. The vehicle management system <NUM> may create, update, and remove the constraints <NUM> to limit movement of the aerial vehicle <NUM>, as described herein. An example plot showing a range of attitude constraints for the aerial vehicle for various angles and distances is shown in <FIG>.

The obstacle <NUM> can be any type of obstacle that may be encountered or observed when piloting an aerial vehicle. Obstacles <NUM> may be static, or non-moving, obstacles, such as terrain, buildings, or other stationary objects. In some implementations, one or more of the obstacles may be moving obstacles, such as other aerial vehicles, or other dynamic, or moving, obstacles. Properties or characteristics of the obstacles <NUM> may be captured by the sensors <NUM>, and stored as part of corresponding obstacle data <NUM> in the memory <NUM>.

The vehicle management system <NUM> can include at least one processor and a memory (e.g., a processing circuit). The memory can store processor-executable instructions that, when executed by processor, cause the processor to perform one or more of the operations described herein. The processor may include a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., or combinations thereof. The memory may include, but is not limited to, electronic, optical, magnetic, or any other storage or transmission device capable of providing the processor with program instructions. The memory may further include a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ASIC, FPGA, read-only memory (ROM), random-access memory (RAM), electrically erasable programmable ROM (EEPROM), erasable programmable ROM (EPROM), flash memory, optical media, or any other suitable memory from which the processor can read instructions. The instructions may include code from any suitable computer programming language. The vehicle management system <NUM> can communicate with any of the components of the aerial vehicle <NUM>, including the memory <NUM>, the sensors <NUM>, the display devices <NUM>, and one or more communication interfaces (not pictured), to perform the operations described herein. The vehicle management system <NUM> can include any or all of the components and perform any or all of the functions of the computer system <NUM> described in connection with <FIG>.

Referring now to the operations of the vehicle management system <NUM>, the vehicle management system <NUM> can detect, based on the world model <NUM> generated from sensor data captured by one or more sensors <NUM> positioned on the aerial vehicle <NUM> during flight, an obstacle <NUM> for the aerial vehicle <NUM>. The sensor data captured by the sensors <NUM> may be used to generate the world model <NUM>, which may include generating one or more distance fields, such as the example distance field represented in the plot shown in <FIG>. A distance field can be a multidimensional field where a value at each point is equal to a distance from a nearest obstacle <NUM>. Gradients in a distance field may be used to determine a direction to the nearest obstacle <NUM> relative to the aerial vehicle <NUM>. Using the sensor data, such as data captured from proximity sensors, cameras, or LiDAR sensors (e.g., the sensors <NUM>), the vehicle management system <NUM> can detect one or more obstacles <NUM> in the environment as the aerial vehicle <NUM> is operated. In some implementations, multiple objects may be detected. In such implementations, multiple directions between each of the obstacles <NUM> relative to the obstacles <NUM> may be determined. For example, using the position of the aerial vehicle <NUM> in the world model, and the determined position of the aerial vehicle <NUM> in the world model, the vehicle management system <NUM> can calculate a direction of a vector in three-dimensional space (e.g., using Euclidean vector functions) from the aerial vehicle <NUM> to the position of the object. In some implementations, the vehicle management system <NUM> can calculate the direction based on one or more gradients in a distance field. The distance from the aerial vehicle <NUM> to the detected objects can be calculated using similar operations. The direction to the obstacle <NUM> and the distance (e.g., Euclidean distance or distance using other coordinate systems) to the obstacle <NUM> can be stored as part of the obstacle data <NUM> for that obstacle <NUM>.

The vehicle management system <NUM> can trigger an augmented manual control mode responsive to a speed of the aerial vehicle being less than a predetermined threshold and detecting the obstacle <NUM>. The techniques described herein may operate when the aerial vehicle <NUM> is navigating at less than a predetermined speed. The predetermined speed may be based on the capabilities (e.g., attributes) of the aerial vehicle <NUM> (e.g., top speed, maneuverability, turn coordination speed limit, etc.). To determine whether the speed of the aerial vehicle <NUM> is less than the threshold, the vehicle management system <NUM> can access one or more vehicle sensor readings that provide vehicle status information, such as aerial vehicle <NUM> acceleration and aerial vehicle <NUM> speed, among others. The vehicle management system <NUM> can compare the current speed of the aerial vehicle <NUM> to the predetermined threshold. If the speed of the aerial vehicle <NUM> is less than the predetermined threshold, the method can proceed to trigger the augmented manual control mode. Otherwise, the augmented manual control mode can remain deactivated.

The augmented manual control mode can be a configuration setting of the vehicle management system <NUM> that indicates that the vehicle management system <NUM> can perform the techniques described herein to limit the maneuverability of the aerial vehicle <NUM> while operating close to obstacles <NUM> in the environment. When the augmented manual mode is triggered (or otherwise initiated) by the vehicle management system <NUM>, commands that are input by the pilot are adjusted, or augmented, based on the distance of the aerial vehicle <NUM> to obstacles in the environment. This augmented manual control mode allows for the pilot to maintain control over the aerial vehicle <NUM>, even when navigating close to obstacles <NUM>. However, the inputs provided by the pilot may be modified according to constraints <NUM> established in the direction of any detected obstacles <NUM>. To trigger the augmented manual control mode, the vehicle management system <NUM> can modify a data structure in the memory of the vehicle management system <NUM> that indicates the augmented control mode is enabled. The augmented manual control mode, if triggered, can cause the vehicle management system <NUM> to perform the further steps of the method <NUM>. As described herein, the augmented manual control mode can limit pilot-input changes to speed, attitude (e.g., roll, pitch, yaw), attitude rate (e.g., roll rate, pitch rate, or yaw rate), or vertical acceleration of the aerial vehicle <NUM> in the direction of obstacles <NUM> in the environment, according to the constraints <NUM>. In some implementations, if the speed of the vehicle exceeds the predetermined threshold after triggering the augmented manual control mode, the vehicle management system <NUM> can deactivate the augmented manual control mode by modifying the data structure in the memory of the vehicle management system <NUM>. Once the augmented manual control mode has been triggered, the vehicle management system <NUM> can perform further operations described herein to limit the maneuverability of the aerial vehicle according to the constraints <NUM>.

The vehicle management system <NUM> can set, responsive to triggering the augmented manual control mode, a constraint <NUM> for the aerial vehicle <NUM> (e.g., a speed constraint, an attitude constraint, attitude rate constraint, a vertical acceleration constraint, etc.) in a direction of the obstacle <NUM> based on a distance between the aerial vehicle <NUM> and the obstacle <NUM>. The direction and distance (e.g., Euclidean distance or angular distance) to the obstacle <NUM> can be retrieved from the corresponding obstacle data <NUM>. The speed constraint <NUM> may be stored in one or more data structures in the memory of the vehicle management system <NUM> (e.g., the constraints <NUM>, etc.). As described herein, the vehicle management system <NUM> can detect both a distance between detected objects in the environment and the aerial vehicle <NUM> and a direction that, if traveled by the aerial vehicle <NUM>, would result in a collision between the aerial vehicle <NUM> and the obstacle <NUM>. The vehicle management system <NUM> may detect multiple obstacles <NUM> in the environment from the world model, and subsequently set multiple constraints <NUM> (e.g., one or more constraints <NUM> for each obstacle <NUM>) for the aerial vehicle <NUM>.

The constraints <NUM> may be set by identifying one or more control limits for movement of the aerial vehicle <NUM>, and subsequently setting a constraint <NUM> for those control limits. For example, if the constraint <NUM> is a speed constraint <NUM>, the vehicle management system <NUM> can set corresponding speed constraints <NUM> for the aerial vehicle <NUM> in the direction of the obstacle <NUM>. To set a constraint <NUM> for the aerial vehicle <NUM>, the vehicle management system <NUM> can apply a function, such as an exponential function or another type of function, to the distance between the aerial vehicle <NUM> and the obstacle <NUM>. This function can be a function having an output that decreases exponentially as the distance between the obstacle <NUM> and the aerial vehicle <NUM> also decreases. Similarly, the function may also use the current orientation (e.g., angle or attitude of the aerial vehicle <NUM>) as input, such that the output of the function also decreases as the angle to the obstacle <NUM> decreases.

An example plot of such a function for a maximum roll/pitch constraint <NUM> for the aerial vehicle <NUM> is shown in <FIG>. As shown in <FIG>, when the angle of the aerial vehicle <NUM> to the obstacle <NUM> or the distance between the aerial vehicle <NUM> to the obstacle <NUM> decreases, the maximum allowable limit (e.g., the constraint <NUM>) also decreases in value. Although this plot is shown for a pitch/roll constraint <NUM>, it should be understood that such constraints <NUM> may be generated using similar functions with similar limits for the speed, attitude, attitude rate, vertical acceleration, or component vectors thereof. The parameters (e.g., weights, coefficients, etc.) of the function used to calculate the constraint <NUM> may be selected based on capabilities (e.g., attributes) of the aerial vehicle <NUM>. For example, different aerial vehicles with different maneuvers, turn coordination speed limits, or abilities may utilize different functions, which may be predetermined or selected to optimize the safety of the aerial vehicle <NUM> while navigating close to obstacles <NUM> in the environment. The vehicle management system <NUM> can set the constraint <NUM> for the speed, attitude, attitude rate, or vertical acceleration of the aerial vehicle <NUM> by providing the distance and attitude of the aerial vehicle <NUM> into the function to generate a constraint <NUM> value. Then, the aerial vehicle <NUM> can update one or more data structures in the memory of the vehicle management system <NUM> with the constraint <NUM> value, which can be stored in association with the corresponding direction. Then, when the aerial vehicle <NUM> receives a command to navigate in the direction (or partially along the direction), the vehicle management system <NUM> can utilize the constraint <NUM> to limit the movement of the aerial vehicle <NUM> according to the constraint <NUM> to maintain a safe distance between the aerial vehicle <NUM> and the detected obstacle <NUM>.

In some implementations, the vehicle management system <NUM> can decompose the direction vector to the obstacle <NUM> from the aerial vehicle <NUM> into component (e.g. x, y, and z vectors) vectors, and set a corresponding speed constraint <NUM> for the aerial vehicle <NUM> in the direction of each component vector. The direction from the aerial vehicle <NUM> to the obstacle <NUM> can be represented in an inertial coordinate frame (e.g., along track, cross-track, etc.). However, the constraints <NUM> for the aerial vehicle <NUM> are represented in the body frame of reference (e.g., the frame of reference relative to the aerial vehicle). Therefore, in some implementations, the vehicle management system <NUM> can transform the inertial frame of reference to the vehicle frame of reference to calculate the proper component constraint <NUM> for each corresponding body axis of the aerial vehicle <NUM>. Similar decomposition techniques may be utilized for the attitude of the aerial vehicle <NUM>, by decomposing the direction into corresponding roll, pitch, and yaw rotation of the aerial vehicle <NUM>. Corresponding constraints <NUM> can then be set for each of the component roll, pitch, and yaw rotations, according to the constraint <NUM> value calculated based on the distance or angle of the aerial vehicle <NUM> to the obstacle <NUM>. In some implementations, the aerial vehicle <NUM> may set a stop constraint <NUM> for an obstacle <NUM> if the vehicle management system <NUM> determines that the aerial vehicle <NUM> is within a predetermined distance of the obstacle <NUM>. For example, if the aerial vehicle <NUM> is within the predetermined distance of the obstacle <NUM>, the vehicle management system <NUM> can set a constraint <NUM> to prevent movement of the aerial vehicle <NUM> in the direction of the obstacle <NUM>.

In some implementations, upon setting a constraint <NUM> for the aerial vehicle <NUM>, the vehicle management system <NUM> can provide an alert to the pilot. The alert can be provided using any suitable alert device, such as an audio alert device, a visual alert device, or another device present in the aerial vehicle <NUM>, such as the display device <NUM>. In some implementations, the alert may be haptic feedback created by sending a signal to an actuator in the vehicle control devices <NUM> of the aerial vehicle <NUM>. The intensity of the haptic feedback, or the alerts described herein, may be proportional to the amount by which the movement of the aerial vehicle <NUM> is constrained. The alerts may be provided to the pilot when instructions are adjusted by the vehicle management system <NUM> or when one or more constraints <NUM> are set.

The vehicle management system <NUM> can receive, via the aerial vehicle control devices <NUM> (e.g., an inceptor, etc.) of the aerial vehicle <NUM>, an instruction to navigate the aerial vehicle <NUM> in the direction to the obstacle <NUM> at a second speed greater than the speed constraint. As the vehicle management system <NUM> updates constraints for the aerial vehicle <NUM>, the pilot of the aerial vehicle <NUM> can send instructions to navigate the aerial vehicle <NUM> close to obstacles in the environment. As described herein, the vehicle control devices <NUM> can be used by a pilot to send instructions to navigate the aerial vehicle <NUM> at different speeds, different attitudes, different attitude rate, or different vertical accelerations. The instructions provided by the vehicle control devices can be received by the vehicle management system <NUM> via one or more communications interfaces, and subsequently applied to the constraints <NUM>. For example, if the vehicle management system <NUM> determines that a constraint <NUM> has been set for a particular direction toward which the pilot is attempting to maneuver the aerial vehicle <NUM>, the vehicle management system <NUM> can proceed to adjust the instruction according to the corresponding constraint <NUM>. If no constraints <NUM> are set, or if the aerial vehicle <NUM> is navigated in a direction that is not associated with a constraint <NUM>, the vehicle management system <NUM> can maneuver the aerial vehicle <NUM> according to the instruction without adjustment. These techniques therefore allow the pilot to maintain manual control over the aerial vehicle <NUM>, while having the flexibility to safely navigate the aerial vehicle <NUM> close to obstacles without significant effort.

The vehicle management system <NUM> can adjust the instruction to replace the second speed with the speed constraint <NUM>. This can cause the aerial vehicle <NUM> to navigate in the direction toward the obstacle <NUM> at the speed constraint <NUM> instead of the second speed. As described herein, the vehicle control devices <NUM> of the aerial vehicle <NUM> can be operated by a pilot to provide one or more maneuver instructions to the aerial vehicle <NUM>. Some examples of the maneuver instructions include a change to a speed of the aerial vehicle <NUM> in a specified direction, a change in orientation (e.g., attitude) of the aerial vehicle <NUM>, change in attitude rate, or a change in vertical acceleration of the aerial vehicle <NUM>. The instructions can be provided to the vehicle management system <NUM>, which can determine whether the instruction should be modified according to one or more constraints <NUM>. The vehicle management system <NUM> can access one or more data structures in the memory of the vehicle management system <NUM> to determine whether any constraints <NUM> are set. If a constraint <NUM> is set, the vehicle management system <NUM> can compare the instructions received from the vehicle control devices to the limitations imposed in the constraint <NUM>. If the vehicle management system <NUM> determines that instruction exceeds the constraint <NUM> (e.g., whether a speed instruction in a particular direction exceeds the speed constraint <NUM> in that direction, etc.), the vehicle management system <NUM> can replace the speed in the instruction along the direction with the speed constraint <NUM>.

In some implementations, the vehicle management system <NUM> can limit instructions according to more than one constraint <NUM>. The vehicle management system <NUM> can adjust constraints <NUM> in multiple directions, for example, to navigate the aerial vehicle <NUM> between two detected obstacles <NUM>. Similarly, the vehicle management system <NUM> can simultaneously adjust instructions to limit both rotation changes (e.g., attitude changes), attitude rate, and speed changes in the same instruction. In some implementations, the vehicle management system <NUM> can generate and provide one or more signals to a vehicle control device, such as an inceptor of the aerial vehicle <NUM>, that limit physical movement of the inceptor based on the speed constraint <NUM>. The signal can be provided to an actuator of the vehicle control device that limits the movement the vehicle control device in the direction of the constraint <NUM>. The constraints <NUM> for changes in the attitude, or attitude rate of the aerial vehicle <NUM> or vertical acceleration of the vehicle can be used by the vehicle management system <NUM> to adjust maneuver instructions, by replacing the corresponding attitude or vertical acceleration changes with corresponding constraints <NUM> when the instruction exceeds the value specified in the constraint <NUM>. In some implementations, the vehicle management system <NUM> can replace decomposed portions of movement instructions, such as x, y, or z axis movement, or roll, yaw, or pitch rotation, with corresponding decomposed constraint <NUM> values, as described herein. When constraints <NUM> are updated by the vehicle management system <NUM> as the aerial vehicle <NUM> navigates the environment, the vehicle management system <NUM> can update the amounts by which instructions are adjusted, allowing the aerial vehicle <NUM> to be safely navigated by a pilot near obstacles <NUM> while conducting aerial tasks.

Referring to <FIG>, and others, illustrated is a data flow diagram <NUM> including the vehicle management system <NUM> of the aerial vehicle <NUM> described in connection with <FIG>, in accordance with one or more implementations. As shown in the data flow diagram <NUM>, the vehicle management system <NUM> can receive sensor data (shown as "Raw Sensor Data") from the sensors <NUM>, as described herein. This sensor data may be captured from the obstacles <NUM> in the environment that is external to the aerial vehicle <NUM>. The vehicle management system <NUM> can generate the world model <NUM> by performing sensor fusion logic <NUM>, which may be implemented to generate one or more distance fields, such as the distance fields shown in <FIG>. The sensor fusion logic <NUM> can be software or a combination of hardware and software that can generate the world model <NUM>, and store the world model <NUM> in the memory <NUM> (not pictured) of the aerial vehicle <NUM>.

The obstacle and terrain position data (e.g., the obstacle data <NUM>, etc.) can then be provided as input to one or more motion planning algorithms <NUM> executing on the vehicle management system <NUM>. The motion planning algorithms <NUM> can be algorithms that, at least in part, carry out the techniques described herein. The motion planning algorithms <NUM> can include software or a combination of hardware and software that can set one or more of the constraints <NUM>, as described herein. The motion planning algorithms <NUM> may also receive one or more input commands (e.g., the stick commands) from the vehicle control devices <NUM>, which may in some implementations be used when setting the constraints <NUM>. The constraints <NUM> may be implemented as part of the control laws <NUM> to limit the amount by which the instructions from the vehicle control devices <NUM> can maneuver the aerial vehicle <NUM>. The control laws <NUM> may also receive information about the vehicle state, which may include position, velocity, acceleration, and attitude data from the vehicle state sensors <NUM>. The vehicle state sensors <NUM> can be one or more of the sensors <NUM> that provide information about the state of the aerial vehicle <NUM>.

The control laws <NUM> can apply the constraints <NUM> to the instructions received from the vehicle control devices <NUM> to generate one or more commands for the vehicle effectors <NUM>. The vehicle effectors can be any type of device that is capable of changing the position, velocity, acceleration, or attitude of the aerial vehicle <NUM>. The commands, which may include adjustments based on the constraints <NUM> as described herein, can cause the aerial vehicle <NUM> to be navigated close to obstacles <NUM> without requiring large amounts of pilot workload. Doing so can preserve the mission intent of pilots for operations or tasks such as terrain masking, while gradually easing the aerial vehicle <NUM> into a desired position near obstacles, rather than swerving the aerial vehicle away, as in other approaches.

<FIG> illustrate example diagrams showing various constraints that can be applied to the motion of an aerial vehicle, in accordance with one or more implementations. <FIG> shows arrows 305A and 305B which represent the allowable motion of the aerial vehicle <NUM> according to constraints set based on detected obstacles <NUM>. As shown, the arrow 305A, which is in the direction of the obstacle <NUM>, represents the limited movement of the aerial vehicle <NUM> in the direction of the obstacle <NUM> as the aerial vehicle approaches the obstacle. In contrast, the arrow 305B, which is not in the direction of any obstacle, is much larger than the arrow 305A, representing unconstrained motion limits of the aerial vehicle away from obstacles <NUM>. <FIG> shows that when the aerial vehicle <NUM> is within the minimum allowed distance toward an obstacle <NUM>, the motion of the aerial vehicle <NUM> in the direction of the obstacle is prevented. Although not explicitly shown in <FIG>, it should be understood that motion in directions that are away from obstacles <NUM> are not constrained.

<FIG> illustrate example plots of a distance field of obstacles 400A and control axis limits 400B, respectively, in accordance with one or more implementations. <FIG> shows an example distance field, which may be included in the world model <NUM> described in connection with <FIG>. In these figures, the pink rectangles are obstacles, and the color bars show a range from the minimum to the maximum distance to the pink obstacles. The distance field can be a two-dimensional or three-dimensional field of values, where each value in the field is equal to a distance to the closest obstacle. Gradients in the distance field can be used to determine the directions to the closest obstacle from any point in the distance field. <FIG> shows an example representation of a function that calculates constraints for a maximum pitch/roll of an aerial vehicle. The function can receive both distance and angle to a detected obstacle as input, and can provide a corresponding output value which is equal to the maximum allowable roll/pitch of the aerial vehicle. As shown, as the distance or angle to the obstacle decreases, the allowable range of maximum pitch/roll values also decreases to zero. Likewise, as the distance or angel to the obstacle increases, the allowable range of maximum pitch roll values for the aerial vehicle also increases. The function can be a non-linear function, such that the motion of the aerial vehicle slows down exponentially as the aerial vehicle approaches the obstacle.

Referring now to <FIG>, depicted is an illustrative flow diagram of a method <NUM> for controlling an aerial vehicle to avoid obstacles. The method <NUM> can be executed, performed, or otherwise carried out by the vehicle management system <NUM><NUM>, the computer system <NUM> described herein in conjunction with <FIG>, or any other computing devices described herein. The method can include detecting an obstacle for an aerial vehicle based on a world model generated from sensor data (STEP <NUM>), determining whether a speed of the aerial vehicle is less than a predetermined threshold (STEP <NUM>), triggering an augmented manual control mode (STEP <NUM>), setting a constraint for the aerial vehicle in the direction of the obstacle (STEP <NUM>), receiving an instruction to navigate the aerial vehicle at a second speed (STEP <NUM>), and adjusting the instruction to replace the second speed with the speed constraint (STEP <NUM>).

At STEP <NUM>, the vehicle management system can detect, based on a world model generated from sensor data captured by one or more sensors positioned on the aerial vehicle during flight, an obstacle for the aerial vehicle. As described herein, the one or more sensors can include one or more of a LiDAR sensor, a radar sensor, or a camera. The sensor data captured by the sensors may be used to generate a world model, which may include generating one or more distance fields, such as the example distance field represented in the plot shown in <FIG>. A distance field can be a multidimensional field where a value at each point is equal to a distance from a particular obstacle. Gradients in a distance field may be used to determine a direction to the obstacle relative to the aerial vehicle. Using the sensor data, such as proximity sensors, cameras, or LiDAR sensors, the vehicle detects one or more obstacles in the environment as the aerial vehicle is operated. In some implementations, multiple objects may be detected. In such implementations, multiple directions between each of the obstacles relative to the obstacles may be determined. For example, using the position of the aerial vehicle in the world model, and the determined position of the aerial vehicle in the world model, the vehicle management system can calculate a direction of a vector in three-dimensional space (e.g., using Euclidean vector functions) from the aerial vehicle to the position of the object. In some implementations, the vehicle management system can calculate the direction based on one or more gradients in a distance field. The distance from the aerial vehicle to the detected objects can be calculated using similar operations.

At STEP <NUM>, the vehicle management system can determine whether a speed of the aerial vehicle is less than a predetermined threshold. The techniques described herein may operate when the aerial vehicle is navigating at less than a predetermined speed. The predetermined speed may be based on the capabilities of the aerial vehicle (e.g., the top speed, maneuverability, turn coordination speed limit, etc.). To determine whether the speed of the aerial vehicle is less than the threshold, the vehicle management system can access one or more vehicle sensor readings that provide vehicle status information, such as aerial vehicle acceleration and aerial vehicle speed, among others. The vehicle management system can compare the current speed of the aerial vehicle to the predetermined threshold. If the speed of the aerial vehicle is less than the predetermined threshold, the method can proceed to STEP <NUM>. Otherwise, if the speed of the aerial vehicle is not less than the predetermined threshold, the method can return to STEP <NUM> of the method.

At STEP <NUM>, the vehicle management system can trigger an augmented manual control mode responsive to the speed of the aerial vehicle being less than the predetermined threshold and detecting the obstacle. The augmented manual control mode can be a configuration setting of the vehicle management system that indicates that the vehicle management system can perform the techniques described herein to limit the maneuverability of the aerial vehicle while operating close to obstacles in the environment. To trigger the augmented manual control mode, the vehicle management system can modify a data structure in the memory of the vehicle management system that indicates the augmented control mode is enabled. The augmented manual control mode, if triggered, can cause the vehicle management system to perform the further steps of the method <NUM>. As described herein, the augmented manual control mode can limit pilot-input changes to speed, attitude (e.g., roll, pitch, yaw), attitude rate, or vertical acceleration of the aerial vehicle in the direction of obstacles in the environment (e.g., the constraints <NUM>). In some implementations, if the speed of the vehicle exceeds the predetermined threshold, the vehicle management system can deactivate the augmented manual control mode by modifying the data structure in the memory of the vehicle management system. In some implementations, the augmented manual control mode may be enabled during any operation of the aerial vehicle, and does not need to be triggered or activated. Once the augmented manual control mode is enabled, the vehicle management system can perform STEP <NUM> of the method <NUM>. In some implementations, the vehicle management system may perform further steps of the method <NUM> without triggering the augmented manual control mode, for example, if the augmented manual control mode may be active by default, regardless of speed.

At STEP <NUM>, the vehicle management system can set, responsive to triggering the augmented manual control mode, a speed constraint (e.g., a constraint <NUM>) for the aerial vehicle in a direction of the obstacle based on a distance between the aerial vehicle and the obstacle. The speed constraint may be stored in one or more data structures in the memory of the vehicle management system (e.g., the constraints <NUM>, etc.). As described herein, the vehicle management system can detect both a distance between detected objects in the environment and the aerial vehicle and a direction that, if traveled by the aerial vehicle, would result in a collision between the aerial vehicle and the obstacle. The vehicle management system may detect multiple obstacles in the environment from the world model, and subsequently set multiple constraints (e.g., one or more constraints for each obstacle) for the aerial vehicle.

The constraints may be set by identifying one or more control limits for movement of the aerial vehicle, and subsequently setting a constraint for those control limits. For example, if the constraint is a speed constraint, the vehicle management system can set corresponding speed constraints for the aerial vehicle in the direction of the obstacle. To set a constraint for the aerial vehicle, the vehicle management system can apply a function, such as an exponential function or another type of function, to the distance between the aerial vehicle and the obstacle. This function can be a function having an output that decreases exponentially as the distance between the obstacle and the aerial vehicle also decreases. Similarly, the function may also use the current orientation (e.g., angle or attitude of the aerial vehicle) as input, such that the output of the function also decreases as the angle to the obstacle decreases.

An example plot of such a function for a maximum roll/pitch constraint for the aerial vehicle is shown in <FIG>. As shown in <FIG>, when the angle of the aerial vehicle to the obstacle or the distance between the aerial vehicle to the obstacle decreases, the maximum allowable limit (e.g., the constraint) also decreases in value. Although this plot is shown for a pitch/roll constraint, it should be understood that such constraints may be generated using similar functions with similar limits for the speed, attitude, attitude rate, vertical acceleration, or component vectors thereof. The parameters (e.g., weights, coefficients, etc.) of the function used to calculate the constraint may be selected based on capabilities of the aerial vehicle. For example, different aerial vehicles with different maneuvers or abilities may utilize different functions, which may be predetermined or selected to optimize the safety of the aerial vehicle while navigating close to obstacles in the environment. The vehicle management system can set the constraint for the speed, attitude, attitude rate, or vertical acceleration of the aerial vehicle by providing the distance and attitude of the aerial vehicle into the function to generate a constraint value. Then, the aerial vehicle can update one or more data structures in the memory of the vehicle management system with the constraint value, which can be stored in association with the corresponding direction. Then, when the aerial vehicle receives a command to navigate in the direction (or partially along the direction), the vehicle management system can utilize the constraint to limit the movement of the aerial vehicle according to the constraint to maintain a safe distance between the aerial vehicle and the detected obstacle.

In some implementations, the vehicle management system can decompose the direction vector to the obstacle from the aerial vehicle into component (e.g. x, y, and z vectors) vectors, and set a corresponding speed constraint for the aerial vehicle in the direction of each component vector. The direction from the aerial vehicle to the obstacle can be represented in an inertial coordinate frame (e.g., along track, cross-track, etc.). However, the constraints for the aerial vehicle are represented in the body frame of reference (e.g., the frame of reference relative to the aerial vehicle). Therefore, in some implementations, the vehicle management system can transform the inertial frame of reference to the vehicle frame of reference to calculate the proper component constraint for each corresponding body axis of the aerial vehicle. Similar decomposition techniques may be utilized for the attitude of the aerial vehicle, by decomposing the direction into corresponding roll, pitch, and yaw rotation of the aerial vehicle. Corresponding constraints can then be set for each of the component roll, pitch, and yaw rotations, according to the constraint value calculated based on the distance or angle of the aerial vehicle to the obstacle. In some implementations, the aerial vehicle may set a stop constraint for an obstacle if the vehicle management system determines that the aerial vehicle is within a predetermined distance of the obstacle. For example, if the aerial vehicle is within the predetermined distance of the obstacle, the vehicle management system can set a constraint to prevent movement of the aerial vehicle in the direction of the obstacle.

In some implementations, upon setting a constraint for the aerial vehicle, the vehicle management system can provide an alert to the pilot. The alert can be provided using any suitable alert device, such as an audio alert device, a visual alert device, or another device present in the aerial vehicle, such as a display device (e.g., the display device <NUM>). In some implementations, the alert may be haptic feedback created by sending a signal to an actuator in the vehicle control devices of the aerial vehicle. The intensity of the haptic feedback, or the alerts described herein, may be proportional to the amount by which the movement of the aerial vehicle is constrained. The alerts may be provided to the pilot when instructions are adjusted by the vehicle management system or when one or more constraints are set.

At STEP <NUM>, the vehicle management system can receive, via an inceptor (e.g., a vehicle control device <NUM>) of the aerial vehicle, an instruction to navigate the aerial vehicle in the direction at a second speed greater than the speed constraint. As the vehicle management system updates constraints for the aerial vehicle, the pilot of the aerial vehicle can send instructions to navigate the aerial vehicle close to obstacles in the environment. As described herein, the vehicle control devices can be used by a pilot to send instructions to navigate the aerial vehicle at different speeds, different attitudes, or different vertical accelerations. The instructions provided by the vehicle control devices can be received by the vehicle management system via one or more communications interfaces, and subsequently applied to the constraints. For example, if the vehicle management system determines that a constraint has been set for a particular direction toward which the pilot is attempting to maneuver the aerial vehicle, the vehicle management system can proceed to execute STEP <NUM> to adjust the instruction according to the corresponding constraint. If no constraints are set, or if the aerial vehicle is navigated in a direction that is not associated with a constraint, the vehicle management system can maneuver the aerial vehicle according to the instruction without adjustment. These techniques therefore allow the pilot to maintain manual control over the aerial vehicle, while having the flexibility to safely navigate the aerial vehicle close to obstacles without significant effort.

At STEP <NUM>, the vehicle management system can adjust the instruction to replace the second speed with the speed constraint, to cause the aerial vehicle to navigate in the direction at the speed constraint instead of the second speed. As described herein, the vehicle control devices (e.g., the vehicle control devices <NUM>) of the aerial vehicle can be operated by a pilot to provide one or more maneuver instructions to the aerial vehicle. Some examples of the maneuver instructions include a change to a speed of the aerial vehicle in a specified direction, a change in orientation (e.g., attitude) of the aerial vehicle, a change in attitude rate, or a change in vertical acceleration of the aerial vehicle. The instructions can be provided to the vehicle management system, which can determine whether the instruction should be modified according to one or more constraints. The vehicle management system can access one or more data structures in the memory of the vehicle management system to determine whether any constraints are set. If a constraint is set, the vehicle management system can compare the instructions received from the vehicle control devices to the limitations imposed in the constraint. If the vehicle management system determines that instruction exceeds the constraint (e.g., whether a speed instruction in a particular direction exceeds the speed constraint in that direction, etc.), the vehicle management system can replace the speed in the instruction along the direction with the speed constraint.

In some implementations, the vehicle management system can limit instructions according to more than one constraint. The vehicle management system can adjust constraints in multiple directions, for example, to navigate the aerial vehicle between two detected obstacles. Similarly, the vehicle management system can simultaneously adjust instructions to limit both rotation changes (e.g., attitude changes) and speed changes in the same instruction. In some implementations, the vehicle management system can generate and provide one or more signals to a vehicle control device, such as an inceptor of the aerial vehicle, that limit physical movement of the inceptor based on the speed constraint. The signal can be provided to an actuator of the vehicle control device that limits the movement the vehicle control device in the direction of the constraint. Constraints to changes in the attitude of the aerial vehicle or vertical acceleration of the vehicle can be used by the vehicle management system to adjust maneuver instructions, by replacing the corresponding attitude or vertical acceleration changes with corresponding constraint values when the instruction exceeds the value specified in the constraint. In some implementations, the vehicle management system can replace decomposed portions of movement instructions, such as x, y, or z axis movement, or roll, yaw, or pitch rotation, with corresponding decomposed constraint values, as described herein. When constraints are updated by the vehicle management system as the aerial vehicle navigates the environment, the vehicle management system can update the amounts by which instructions are adjusted, allowing the aerial vehicle to be safely navigated by a pilot near obstacles while conducting aerial tasks.

Referring now to <FIG>, depicted is a block diagram of an example computer system <NUM>. The computer system or computing device <NUM> can include or be used to implement the system <NUM> or the system <NUM>, or its components such as the vehicle management system <NUM>. The computing system <NUM> includes at least one bus <NUM> or other communication component for communicating information and at least one processor <NUM> or processing circuit coupled to the bus <NUM> for processing information. The computing system <NUM> can also include one or more processors <NUM> or processing circuits coupled to the bus <NUM> for processing information. The computing system <NUM> also includes at least one main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <NUM> for storing information, and instructions to be executed by the processor <NUM>. The computing system <NUM> may further include at least one read only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor <NUM>. A storage device <NUM>, such as a solid state device, magnetic disk, or optical disk, can be coupled to the bus <NUM> to persistently store information and instructions.

The computing system <NUM> may be coupled via the bus <NUM> to a display <NUM>, such as a liquid crystal display, or active matrix display, for displaying information to a user such as an administrator of the data processing system. An input device <NUM>, such as a keyboard or voice interface may be coupled to the bus <NUM> for communicating information and commands to the processor <NUM>. The input device <NUM> can include a touch screen display <NUM>. The input device <NUM> can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The display <NUM> can be part of the display devices <NUM>, or other components of <FIG>.

The processes, systems, and methods described herein can be implemented by the computing system <NUM> in response to the processor <NUM> executing an arrangement of instructions contained in main memory <NUM>. One or more processors in a multi-processing arrangement may also be employed to execute the instructions contained in main memory <NUM>.

Some of the description herein emphasizes the structural independence of the aspects of the system components and illustrates one grouping of operations and responsibilities of these system components. Other groupings that execute similar overall operations are understood to be within the scope of the present application. Modules can be implemented in hardware or as computer instructions on a non-transient computer readable storage medium, and modules can be distributed across various hardware- or computer-based components.

The systems described above can provide multiple ones of any or each of those components, and these components can be provided on either a standalone system or on multiple instantiation in a distributed system. In addition, the systems and methods described above can be provided as one or more computer-readable programs or executable instructions embodied on or in one or more articles of manufacture. The article of manufacture can be cloud storage, a hard disk, a CD-ROM, a flash memory card, a PROM, a RAM, a ROM, or a magnetic tape. In general, the computer-readable programs can be implemented in any programming language, such as LISP, PERL, C, C++, C#, PROLOG, or in any byte code language such as JAVA. The software programs or executable instructions can be stored on or in one or more articles of manufacture as object code.

Example and non-limiting module implementation elements include sensors providing any value determined herein, sensors providing any value that is a precursor to a value determined herein, datalink or network hardware including communication chips, oscillating crystals, communication links, cables, twisted pair wiring, coaxial wiring, shielded wiring, transmitters, receivers, or transceivers, logic circuits, hard-wired logic circuits, reconfigurable logic circuits in a particular non-transient state configured according to the module specification, any actuator including at least an electrical, hydraulic, or pneumatic actuator, a solenoid, an op-amp, analog control elements (springs, filters, integrators, adders, dividers, gain elements), or digital control elements.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices include cloud storage).

The terms "computing device," "component," or "data processing apparatus" or the like encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array), a GPU, or an ASIC (application specific integrated circuit). The apparatus and execution environment can realize various different computing model infrastructures, such as web services and/or distributed computing infrastructures.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).

The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), a GPU, or an ASIC (application specific integrated circuit). Devices suitable for storing computer program instructions and data can include nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements, and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof herein is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively.

Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "an implementation," "some implementations," "one implementation" or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. For example, a reference to "at least one of 'A' and 'B'" can include only "A," only "B," as well as both "A" and "B. " Such references used in conjunction with "comprising" or other open terminology can include additional items.

Modifications of described elements and acts such as variations in sizes, dimensions, structures, shapes and proportions of the various elements, values of parameters, mounting arrangements, use of materials, colors, and orientations can occur without materially departing from the teachings and advantages of the subject matter disclosed herein. For example, elements shown as integrally formed can be constructed of multiple parts or elements, the position of elements can be reversed or otherwise varied, and the nature or number of discrete elements or positions can be altered or varied.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what can be claimed, but rather as descriptions of features specific to particular embodiments of particular aspects. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged into multiple software products.

Claim 1:
A method for controlling an aerial vehicle (<NUM>) to avoid obstacles, comprising:
detecting (<NUM>), by one or more processors coupled to memory of the aerial vehicle (<NUM>), based on a world model (<NUM>) generated from sensor data captured by one or more sensors (<NUM>) positioned on the aerial vehicle (<NUM>) during flight, an obstacle (<NUM>) for the aerial vehicle (<NUM>);
the method being characterised by the following steps:
triggering (<NUM>, <NUM>), by the one or more processors, an augmented manual control mode responsive to a speed of the aerial vehicle (<NUM>) being less than a predetermined threshold and detecting the obstacle (<NUM>);
setting (<NUM>), by the one or more processors responsive to triggering the augmented manual control mode, a speed constraint for the aerial vehicle (<NUM>) in a direction of the obstacle (<NUM>) based on a distance between the aerial vehicle (<NUM>) and the obstacle (<NUM>);
receiving (<NUM>), by the one or more processors via an inceptor of the aerial vehicle (<NUM>), an instruction to navigate the aerial vehicle (<NUM>) in the direction of the obstacle (<NUM>) at a second speed greater than the speed constraint; and
adjusting (<NUM>), by the one or more processors, the instruction to replace the second speed with the speed constraint, to cause the aerial vehicle (<NUM>) to navigate in the direction of the obstacle (<NUM>) at the speed constraint instead of the second speed.