Adjustable object avoidance proximity threshold based on predictability of the environment

Various embodiments include methods, devices, and robotic vehicle processing devices implementing such methods for automatically adjusting the minimum distance that a robotic vehicle is permitted to approach an object by a collision avoidance system (the “proximity threshold”) to compensate for unpredictability in environmental or other conditions that may compromise control or navigation of the robotic vehicle, and/or to accommodate movement unpredictability of the object. Some embodiments enable dynamic adjustments to the proximity threshold to compensate for changes in environmental and other conditions. Some embodiments include path planning that takes into account unpredictability in environmental or other conditions plus movement unpredictability of objects in the environment.

BACKGROUND

Robotic vehicles, such as aerial robotic vehicles or “drones,” are often used for a variety of applications, such as surveillance, photography, and/or cargo delivery. Many robotic vehicles use obstacle avoidance systems that work in conjunction with vehicle control systems to avoid hitting people, property, and objects. For example, once a robotic vehicle detects a nearby object, the obstacle avoidance system executing within the controller may prevent the robotic vehicle from approaching the object closer than some minimum distance (which is referred to herein as a “proximity threshold”). The proximity threshold is typically a fixed distance. In the case of aerial robotic vehicles that includes propeller blades, the proximity threshold may be a few feet to avoid damage and/or injury from contact with the spinning propeller blades.

SUMMARY

Various embodiments include devices, systems, and methods for operating a robotic vehicle that take into account conditions that may impact the ability of the robotic vehicle to avoid colliding with objects. Various embodiments may include a processor of the robotic vehicle monitoring environmental or other conditions affecting predictability of control or navigation of the robotic vehicle, and in response to determining that control or navigation of the robotic vehicle is or could be compromised by the environmental or other conditions, adjusting a proximity threshold used in a collision avoidance system consistent with an effect on the control or navigation of the robotic vehicle of the environmental or other conditions. Some embodiments may further include monitoring the environmental or other conditions affecting control or navigation of the robotic vehicle for change, and returning the proximity threshold to a default value in response to determining that control or navigation of the robotic vehicle are no longer compromised by the environmental or other conditions.

In some embodiments, monitoring environmental or other conditions affecting predictability of control or navigation of the robotic vehicle may include operating the robotic vehicle to remain in a set position or follow a defined path, monitoring positions of the robotic vehicle to detect deviations from the set position or follow a defined path, and determining a degree of control or navigation unpredictability based on observed deviations from the set position or follow a defined path.

Some embodiments may further include obtaining sensor data from one or more sensors configured to detect one or more objects in a vicinity of the robotic vehicle, determining, based on the sensor data, whether one or more objects in the vicinity of the robotic vehicle pose an obstacle or potential obstacle to the robotic vehicle, and in response to determining that one or more objects in the vicinity of the robotic vehicle pose an obstacle or potential obstacle to the robotic vehicle: determining a classification of an object posing an obstacle or potential obstacle to the robotic vehicle; further adjusting the proximity threshold based on the classification of the object posing an obstacle or potential obstacle to the robotic vehicle; and controlling the robotic vehicle using the further adjusted proximity threshold for collision avoidance. In some embodiments, further adjusting the proximity threshold based on the classification of the object posing an obstacle or potential obstacle to the robotic vehicle may include increasing the proximity threshold adjusted for environmental or other conditions affecting control or navigation of the robotic vehicle by an amount corresponding to the classification of the object and unpredictability of the object. In some embodiments, determining a classification of the object posing an obstacle or potential obstacle to the robotic vehicle may include determining whether the object is animate object or inanimate object, and further adjusting the proximity threshold based on the classification of the object may include one of increasing the adjusted proximity threshold in response to the classification of the object being animate or decreasing the proximity threshold in response to the classification of the object being inanimate. In some embodiments, adjusting the proximity threshold setting in the collision avoidance system based on the classification of the object may include determining a degree of movement unpredictability of the object corresponding to the determined classification of the object. In such embodiments, determining the degree of movement unpredictability of the object corresponding to the determined classification of the object may include accessing a data structure in memory for the degree of movement unpredictability correlated to the classification of the object. Some embodiments may further include returning the proximity threshold setting in the collision avoidance system to the proximity threshold adjusted consistent with the effect on the control or navigation of the robotic vehicle of the environmental or other conditions in response to determining that there are no objects in the vicinity of the robotic vehicle posing an obstacle or potential obstacle to the robotic vehicle.

In some embodiments, determining a classification of an object posing an obstacle or potential obstacle to the robotic vehicle may include determining a classification for all objects posing an obstacle or potential obstacle to the robotic vehicle. Such embodiments may further include generating a map of all objects posing an obstacle or potential obstacle to the robotic vehicle in which the proximity threshold adjusted consistent with the effect on the control or navigation of the robotic vehicle of the environmental or other conditions and a further distance corresponding to each object based on the object's classification are added as an exclusion perimeter around the object's volume, determining a detour that remains outside the exclusion perimeter of all detected obstacles, and controlling the robotic vehicle to execute the detour.

Some embodiments may include a processor implemented method for operating a robotic vehicle that includes obtaining data from one or more sensors configured to detect one or more objects in a vicinity of the robotic vehicle, determining, based on the sensor data, whether one or more objects in the vicinity of the robotic vehicle pose an obstacle or potential obstacle to the robotic vehicle, and in response to determining that one or more objects in the vicinity of the robotic vehicle pose an obstacle or potential obstacle to the robotic vehicle: determining a classification of an object posing an obstacle or potential obstacle to the robotic vehicle; adjusting a proximity threshold of a collision avoidance system to accommodate movement unpredictability of the object based on the classification of the object posing an obstacle or potential obstacle to the robotic vehicle, and controlling the robotic vehicle using the adjusted proximity threshold for collision avoidance.

Further embodiments include a robotic vehicle having a processor configured with processor-executable instructions to perform operations of any of the methods summarized above. Further embodiments include a processing device for use in a robotic vehicle configured to perform operations of any of the methods summarized above. Further embodiments include a non-transitory processor-readable media having stored thereon processor-executable instructions configured to cause a processor of a robotic vehicle to perform operations of any of the methods summarized above.

DETAILED DESCRIPTION

Various embodiments include methods and robotic vehicle processing devices implementing such methods for automatically adjusting the minimum distance that a robotic vehicle is permitted to approach an object by a collision avoidance system based upon the unpredictability in the environment affecting the ability of the robotic vehicle to maneuver and navigate accurately and/or the nature and unpredictability of nearby objects. Adjusting the minimum approach distance or proximity threshold used in a collision avoidance system based on unpredictability in the environment and/or of nearby obstacles or objects enables robotic vehicles to operate with greater flexibility than is feasible using a fixed proximity threshold, while remaining a safe distance away from objects. In some embodiments, the threshold distance used by a collision avoidance system for avoiding objects may be adjusted by an amount or factor to ensure collisions are avoided despite impacts on navigational and/or control precision caused by environmental or other conditions. In some embodiments, the threshold distance used by a collision avoidance system for avoiding objects may be adjusted by an amount or factor that accommodates unpredictability in object's movements determined based on a classification of the object. In some embodiments, the threshold distance used by a collision avoidance system for avoiding objects may be based on a threshold distance appropriate for the object or type of object plus an adjustment that accommodates unpredictability in the object's movements determined based on a classification of the object. In some embodiments, the threshold distance may be based on a threshold distance appropriate for the object or type of object plus an adjustment that accommodates unpredictability in object's movements determined with both based on a classification of the object, with an additional adjustment by an amount or factor to ensure collisions are avoided despite impacts on navigational and/or control precision caused by environmental or other conditions.

In some embodiments, a processor of the robotic vehicle may determine a degree to which environmental or other conditions may compromise the ability of the robotic vehicle to maneuver and/or navigate with precision. Examples of unpredictable or low certainty environmental conditions that may compromise maneuvering and/or navigational precision include wind (particularly gusty winds) for aerial vehicles, precipitation, fog (which can limit visibility for navigation cameras), waves and current for waterborne vehicles, ice and snow for land based vehicles, and the like. Such environmental conditions can change rapidly or affect the control of a robotic vehicle faster than a control system can respond, and thus can decrease the precision by which the robotic vehicle can maneuver and maintain navigational/positional control. For example, if an aerial robotic vehicle maneuvering to avoid a collision (e.g., under control of a collision avoidance system) is hit by a gust of wind, the robotic vehicle could be pushed closer to an object than the proximity threshold before the collision avoidance system can react and redirect the robotic vehicle. Examples of other conditions that may compromise the ability of the robotic vehicle to maneuver and/or navigate include damage to propellers, wheels or wings or control elements of the vehicle, structural damage, loose or shifting payloads, obstructions on or damage to navigation cameras, damage to radar or lidar sensors, interruptions or degradation of signals from navigation satellites, and the like. For example, if an aerial robotic vehicle maneuvering to avoid a collision (e.g., under control of a collision avoidance system) experiences a shift in payload, the robotic vehicle could drift closer to an object than the proximity threshold before the collision avoidance system can react and redirect the robotic vehicle. In some embodiments, the processor may determine the degree to which a number of environmental or other conditions combine to compromise maneuvering and/or navigational precision. In some embodiments, the processor may determine environmental or other conditions from observing the environment and/or operations of the robotic vehicle. In some embodiments, the processor may determine environmental or other conditions from external sources (e.g., a weather service).

In some embodiments, the classification may be whether the object or obstacle is animate (and thus mobile) or inanimate (and thus immobile). In some embodiments, the threshold distance appropriate for the object classification may depend upon the vulnerability or importance the object or obstacle, such as whether the object or obstacle is classified as a human, an animal, a structure, an automobile, artwork, glassware, etc. The adjustment to the proximity threshold to accommodate unpredictability of the object or obstacle may vary depending upon the type or classification of the object or obstacle, such as assigning a larger proximity threshold to children, which may behave unpredictably in the presence of a robotic vehicle, than to adults, which can be expected to behave in a more predictable manner. For example, dogs may be assigned a relatively large adjustment to the proximity threshold to account the fact that dogs often behave unpredictably, including chasing after robotic vehicles. The adjustment to the proximity threshold may accommodate both the degree to which movements may be unpredictable and a speed or magnitude of unpredictable movements. For example, objects (e.g., animals) capable of fast movements (e.g., dogs, birds, etc.) may be assigned a larger adjustment to proximity thresholds than objects that move relatively slowly (e.g., people, cows, gates, etc.). A default proximity threshold and adjustment for unpredictability may be used for objects for which a classification is not determined.

In some embodiments, a processor of a robotic vehicle may determine a classification for an object that is being approached, and adjust the proximity threshold used by the collision avoidance system based on that object's classification plus an adjustment for unpredictability of environmental or other conditions. In some embodiments, the processor of the robotic vehicle may classify all detected objects, determine an exclusion perimeter for each object based on each object's unpredictability plus an adjustment for unpredictability in environmental or other conditions that might impact navigational control of the robotic vehicle, and determine a detour flight path to remain outside the exclusion perimeter of all detected objects.

As used herein, the terms “robotic vehicle” and “drone” refer to one of various types of vehicles including an onboard computing device configured to provide some autonomous or semi-autonomous capabilities. Examples of robotic vehicles include but are not limited to: robotic vehicles, such as an unmanned aerial vehicle (UAV); ground vehicles (e.g., an autonomous or semi-autonomous car, a vacuum robot, etc.); water-based vehicles (i.e., vehicles configured for operation on the surface of the water or under water); space-based vehicles (e.g., a spacecraft or space probe); and/or some combination thereof. In some embodiments, the robotic vehicle may be manned. In other embodiments, the robotic vehicle may be unmanned. In embodiments in which the robotic vehicle is autonomous, the robotic vehicle may include an onboard computing device configured to maneuver and/or navigate the robotic vehicle without remote operating instructions (i.e., autonomously), such as from a human operator (e.g., via a remote computing device). In embodiments in which the robotic vehicle is semi-autonomous, the robotic vehicle may include an onboard computing device configured to receive some information or instructions, such as from a human operator (e.g., via a remote computing device), and autonomously maneuver and/or navigate the robotic vehicle consistent with the received information or instructions. In some implementations, the robotic vehicle may be an aerial vehicle (unmanned or manned), which may be a rotorcraft or winged aircraft. For example, a rotorcraft (also referred to as a multirotor or multicopter) may include a plurality of propulsion units (e.g., rotors/propellers) that provide propulsion and/or lifting forces for the robotic vehicle. Specific non-limiting examples of rotorcraft include tricopters (three rotors), quadcopters (four rotors), hexacopters (six rotors), and octocopters (eight rotors). However, a rotorcraft may include any number of rotors.

The term “obstacle” is used herein to refer to an object that a robotic vehicle must maneuver around to avoid a collision.

The term “proximity threshold” is used herein to refer to a minimum distance between an object and a robotic vehicle that a collision avoidance system will permit before controlling the robotic vehicle to stop or change a direction of travel away from the object. Similarly, the term “exclusion perimeter” is used herein to refer to a distance around an obstacle that a robotic vehicle should avoid to ensure that the robotic vehicle remains outside the proximity threshold. The term “adjusted proximity threshold” is used herein to refer to the minimum distance between an object and the robotic vehicle that the collision avoidance system will permit including adjustments that account for degradation of maneuvering and/or navigational precision that may be caused by environmental or other conditions, and/or for unpredictability in the movements or position of nearby objects.

Operations of a robotic vehicle200within an environment10that includes various trees31,32,33according to various embodiments are illustrated inFIG. 1A. When the robotic vehicle200approaches objects (e.g., trees31-33), a processor within the robotic vehicle200may process data received from onboard sensors (e.g., a camera, radar, lidar, etc.) to determine a type or classification of nearby objects. In the example illustrated inFIG. 1A, the processor of the robotic vehicle200may identify the nearby objects31-33as trees or non-animate objects. In some embodiments, the processor may determine that nearby objects31-33are not classified as sensitive or valuable objects, particularly if the objects do not fit a given or preloaded classification. In the illustrated example in which the nearby objects31-33are trees (i.e., not fragile or valuable), the processor may not adjust the proximity threshold60a, and thus the collision avoidance system implemented in the processor may remain at a default value. In some embodiments, the default value of the proximity threshold60amay depend upon whether the robotic vehicle200is or is not equipped with propeller guards. With the proximity threshold60aset at the default value, the robotic vehicle200is able to maneuver or follow user control commands to fly between the detected objects, such as along a path between trees32and33.

Additionally, the processor of the robotic vehicle200may monitor environmental or other conditions regarding their unpredictability and potential impact on the ability of the robotic vehicle to maneuver and maintain precise navigational control. For example, if the processor determines that the avionics system has issued sudden maneuver instructions to maintain position (e.g., in a hover) or follow a navigational route, the processor may determine that environmental conditions are impacting positional control of the robotic vehicle, and increase the proximity threshold to accommodate increased errors in positional control. As another example, the processor may receive information from broadcast weather forecasts regarding wind conditions, and increase the proximity threshold to accommodate errors in positional control that may arise from high winds and gusty conditions.

As a result of adjusting the proximity threshold to accommodate uncertainty in the navigational control the robotic vehicle200, the adjusted proximity threshold60bused by the collision avoidance system may be larger than the default proximity threshold60awhen there is no or low uncertainty in environmental conditions. With a larger adjusted proximity threshold60b, the robotic vehicle200will avoid obstacles such as the trees31-33by a larger distance.

Referring to the example illustrated inFIG. 1B, having transited the trees31-33, the robotic vehicle200may approach a person40walking a dog50. With reference toFIGS. 1A-1C, upon detecting these objects, the processor of the robotic vehicle200may process sensor data (e.g., camera image data) and classify the objects as a human and animal dog. Based upon this classification, the processor may adjust the proximity threshold used by the collision avoidance system from the default proximity threshold60ato a proximity threshold consistent with or corresponding to a human (proximity threshold64a) and a dog (proximity threshold62a). The larger proximity thresholds62a,64bused by the collision avoidance system ensure that the robotic vehicle200will give the person40and dog50a wider berth to accommodate unpredictability in their movements (as well as their increased vulnerability to collisions compared to inanimate objects). Further, the proximity threshold64aused by the collision avoidance system for avoiding the person40may be different (e.g., smaller) than the proximity threshold62aused by the collision avoidance system for avoiding the dog50because the dog is more unpredictable and can move faster than the person.

Additionally, the processor of the robotic vehicle200may monitor environmental or other conditions regarding their unpredictability and potential impact on the ability to maneuver and maintain precise navigational control of the robotic vehicle, and further adjust proximity thresholds accordingly. For example, if the processor notes that the avionics system has exhibited difficulty maintaining maneuver or navigation control of the robotic vehicle, the processor may increase the proximity threshold to accommodate such control errors. As another example, the processor may receive information from broadcast weather forecasts regarding wind conditions, and increase the proximity threshold to accommodate errors in positional control that may arise from high winds and gusty conditions.

Increasing the proximity threshold to accommodate unpredictability in environmental or other conditions may result in an adjusted proximity threshold64bfor a person40and a different adjusted proximity threshold62bfor a dog50. In the example illustrated inFIG. 1Bin which the person40and the dog50are together, the processor of the robotic vehicle200may implement the largest of the adjusted proximity thresholds (i.e., the adjusted proximity threshold62bfor the dog50) in the collision avoidance system.

Thus, various embodiments enable the robotic vehicle collision avoidance system to dynamically adjust how close the robotic vehicle200is permitted to approach various objects while operating in an environment in order to ensure the robotic vehicle does not approach any object closer than a minimum distance (e.g., a default proximity threshold) despite unpredictability in environmental and other conditions and unpredictability in movements of various objects.

FIG. 1Cillustrates a further embodiment in which the proximity threshold for various objects detected by robotic vehicle under observed environmental or other conditions may be used in planning a path to avoid approaching any object closer than an appropriate proximity threshold despite unpredictability in environmental and other conditions and unpredictability in movements of various objects. With reference toFIGS. 1A-1C, a robotic vehicle200transiting along a path70will eventually detect objects in its path that includes trees31-33, a human40, and a dog50. In some embodiments, a processor of the robotic vehicle200may evaluate the detected objects, such as through visual processing and image recognition methods, to determine a type or classification of each of the objects31-33,40,50. The processor may then determine an appropriate proximity threshold for each of the observed objects31-34,40,50that should be implemented in the collision avoidance system based upon the nature and unpredictability of the objects. Because the various objects have differing appropriate proximity threshold in view of their respective vulnerability to collision and movement unpredictability, the processor may take into account all of the determined proximity thresholds in order to plot an alternative path around all detected objects. In order to do so efficiently, the processor may generate an internal map of each of the detected objects that adds an exclusion perimeter around each object based upon the proximity threshold appropriate for that object in view of that object's vulnerability to collision and movement unpredictability. Such a map enables the processor to then determine a detour or path72a,72baround the detected objects that will ensure the robotic vehicle200does not approach any of the objects closer than their corresponding proximity threshold.

For example, the processor of the robotic vehicle200may generate a map in which trees31-33are assigned an exclusion radius that includes a distance D1that is appropriate for trees considering their low vulnerability to collision and their relatively small movement unpredictability. Similarly, the map generated by the processor of the robotic vehicle200may assign an exclusion radius D3about the person40, and an exclusion radius D6about the dog50that is consistent with their relatively high vulnerability to collision and their respective movement unpredictability. Thus, the exclusion radius D6for the dog50may be larger than the exclusion radius D3for the person40because dogs have a tendency to chase robotic vehicles and are fast, while people are vulnerable to collisions but are less likely to move in an unpredictable manner. Thus, some embodiments enable a robotic vehicle200to determine a detour or path72a,72baround the detected objects that accommodates unpredictability in the movements of the objects.

Additionally, the processor of the robotic vehicle200may increase the exclusion radius assigned to each object within the map to accommodate unpredictability in environmental and other conditions. For example, if an aerial robotic vehicle is maneuvering in gusty wind conditions, the processor may increase the exclusion radius on each object by a factor consistent with a decrease in the precision of vehicle maneuvers and navigation that could be caused by unpredictable wind gusts. Thus, the exclusion radius about the trees31-33may be increased to include a distance D2. Similarly, the exclusion radius about the person40may be increased to distance D4and the exclusion radius about the dog of50may be increased to distance D7. Thus, some embodiments enable a robotic vehicle200to determine a detour or path72a,72baround the detected objects that accommodates unpredictability in the movements of objects and compensates for unpredictability in environmental and other conditions affecting the ability of the robotic vehicle to maneuver and navigate with precision.

Various embodiments may be implemented within a variety of robotic vehicles configured to communicate with one or more communication networks, an example of which in the form of an aerial robotic vehicle200suitable for use with various embodiments is illustrated inFIG. 2. With reference toFIGS. 1A-2, an aerial robotic vehicle200operating in a mission environment20may include a plurality of rotors120(e.g., four rotors), each driven by a corresponding motor125. A body110of the aerial robotic vehicle200may support the plurality of rotors120and motors125. In some instances, the robotic vehicle110may include propeller guards250positioned about the rotors120to reduce the damage that a collision may cause to an object, such as a human.

An aerial robotic vehicle200may include one or more onboard sensors, such as one or more cameras236. The aerial robotic vehicle200may include a processing device210, which may further include one or more attitude sensors, such as an altimeter, a gyroscope, accelerometers, an electronic compass, a satellite positioning system receiver, etc., that may be used by the processor220to determine vehicle attitude and location information for controlling flight and navigating.

Cameras236may be disposed in various locations on the aerial robotic vehicle200and different types of camera may be included. For example, a first set of cameras236may face a side of each of the rotors120in the plane of rotation thereof, such as mounted near a central part of the aerial robotic vehicle200. Additionally, or alternatively, second set of cameras236may be mounted under the rotors120, such as in a position configured to detect whether propeller guards250are present. The aerial robotic vehicle200may also include other types of sensors, including detection and ranging sensors, such as radar, sonar, lidar, and the like.

Image data generated by the cameras236, as well as data from one or more other types of sensors (e.g., radar, sonar, lidar, etc.), may be used by an object avoidance system executing in the processor220. In various embodiments, image data received from cameras236, as well as other sensor data, may be processed by an object avoidance system to detect objects or obstacles in the vicinity of the robotic vehicle200during operation.

The aerial robotic vehicle200may include a processing device210that may be coupled to each of the plurality of motors125that drive the rotors120. The processing device210may be configured to monitor and control the various functionalities, sub-systems, and components of the aerial robotic vehicle200. For example, the processing device210may be configured to monitor and control various modules, software, instructions, circuitry, hardware, etc. related to propulsion, navigation, power management, sensor management, and/or stability management.

The processing device210may house various circuits and devices used to control the operation of the aerial robotic vehicle200. For example, the processing device210may include a processor220that directs the control of the aerial robotic vehicle200. The processor220may include one or more processors configured to execute processor-executable instructions (e.g., applications, routines, scripts, instruction sets, etc.) to control flight, antenna usage, and other operations of the aerial robotic vehicle200, including operations of various embodiments. In some embodiments, the processing device210may include memory222coupled to the processor220and configured to store data (e.g., flight plans, obtained sensor data, received messages/inputs, applications, etc.). The processor220and memory222may be configured as or be included within a system-on-chip (SoC)215along with additional elements such as (but not limited to) a communication interface224, one or more input units226, non-volatile storage memory230, and a hardware interface234configured for interfacing the SoC215with hardware and components of the aerial robotic vehicle200. Components within the processing device210and/or the SoC215may be coupled together by various circuits, such as a bus225,235or another similar circuitry.

The processing device210may include more than one SoC215thereby increasing the number of processors220and processor cores. The processing device210may also include processors220that are not associated with the SoC215. Individual processors220may be multi-core processors. The processors220may each be configured for specific purposes that may be the same as or different from other processors220of the processing device210or SoC215. One or more of the processors220and processor cores of the same or different configurations may be grouped together. A group of processors220or processor cores may be referred to as a multi-processor cluster.

The terms “system-on-chip” or “SoC” are used herein to refer to a set of interconnected electronic circuits typically, but not exclusively, including one or more processors (e.g.,220), a memory (e.g.,222), and a communication interface (e.g.,224). The SoC215may include a variety of different types of processors220and processor cores, such as a general purpose processor, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), an accelerated processing unit (APU), a subsystem processor of specific components of the processing device, such as an image processor for a camera subsystem or a display processor for a display, an auxiliary processor, a single-core processor, and a multicore processor. An SoC215may further embody other hardware and hardware combinations, such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), other programmable logic device, discrete gate logic, transistor logic, performance monitoring hardware, watchdog hardware, and time references. Integrated circuits may be configured such that the components of the integrated circuit reside on a single piece of semiconductor material, such as silicon.

In various embodiments, the processing device210may include or be coupled to one or more communication components232, such as a wireless transceiver, an onboard antenna, and/or the like for transmitting and receiving wireless signals through the wireless communication link25. The one or more communication components232may be coupled to the communication interface224and may be configured to handle wireless wide area network (WWAN) communication signals (e.g., cellular data networks) and/or wireless local area network (WLAN) communication signals (e.g., Wi-Fi signals, Bluetooth signals, etc.) associated with ground-based transmitters/receivers (e.g., base stations, beacons, Wi-Fi access points, Bluetooth beacons, small cells (picocells, femtocells, etc.), etc.). The one or more communication components232may receive data from radio nodes, such as navigation beacons (e.g., very high frequency (VHF) omni-directional range (VOR) beacons), Wi-Fi access points, cellular network base stations, radio stations, etc.

The processing device210, using the processor220, the one or more communication components232, and an antenna may be configured to conduct wireless communications with a variety of remote computing devices, examples of which include the base station or cell tower (e.g., base station20), a beacon, server, a smartphone, a tablet, or another computing device with which the aerial robotic vehicle200may communicate. The processor220may establish the wireless communication link25via a modem and the antenna. In some embodiments, the one or more communication components232may be configured to support multiple connections with different remote computing devices using different radio access technologies. In some embodiments, the one or more communication components232and the processor220may communicate over a secured communication link. The security communication links may use encryption or another secure means of communication in order to secure the communication between the one or more communication components232and the processor220.

The aerial robotic vehicle200may operate in the mission environment20communicating with a base station70, which may provide a communication link to a remote computing device75and/or a remote server80via a communication network90. The base station70may provide the wireless communication link25, such as through wireless signals to the aerial robotic vehicle200. The remote computing device75may be configured to control the base station70, the aerial robotic vehicle200, and/or control wireless communications over a wide area network, such as providing a wireless access points and/or other similar network access point using the base station70. In addition, the remote computing device75and/or the communication network90may provide access to a remote server80. The aerial robotic vehicle200may be configured to communicate with the remote computing device75and/or the remote server80for exchanging various types of communications and data, including location information, navigational commands, data inquiries, and mission data.

Aerial robotic vehicles may navigate or determine positioning using altimeters or navigation systems, such as Global Navigation Satellite System (GNSS), Global Positioning System (GPS), etc. In some embodiments, the aerial robotic vehicle200may use an alternate source of positioning signals (i.e., other than GNSS, GPS, etc.). The aerial robotic vehicle200may use position information associated with the source of the alternate signals together with additional information (e.g., dead reckoning in combination with last trusted GNSS/GPS location, dead reckoning in combination with a position of the aerial robotic vehicle takeoff zone, etc.) for positioning and navigation in some applications. Thus, the aerial robotic vehicle200may navigate using a combination of navigation techniques, including dead-reckoning, camera-based recognition of the land features below and around the aerial robotic vehicle200(e.g., recognizing a road, landmarks, highway signage, etc.), etc. that may be used instead of or in combination with GNSS/GPS location determination and triangulation or trilateration based on known locations of detected wireless access points.

In some embodiments, the processing device210of the aerial robotic vehicle200may use one or more of various input units226for receiving control instructions, data from human operators or automated/pre-programmed controls, and/or for collecting data indicating various conditions relevant to the aerial robotic vehicle200. For example, the input units226may receive input from one or more of various sensors, such as camera(s), microphone(s), position information functionalities (e.g., a global positioning system (GPS) receiver for receiving GPS coordinates), flight instruments (e.g., attitude indicator(s), gyroscope(s), anemometer, accelerometer(s), altimeter(s), compass(es), etc.), keypad(s), etc. In some embodiments, the processing device210may further be configured to use data received from the one or more sensors to monitor environmental and other conditions to detect a level of unpredictability, especially unpredictability or errors in the ability of the robotic vehicle200to maneuver and/or navigate with precision.

Aerial robotic vehicles may be winged or rotor craft varieties. For example, the aerial robotic vehicle200may be a rotary propulsion design that utilizes one or more rotors120driven by corresponding motors to provide lift-off (or take-off) as well as other aerial movements (e.g., forward progression, ascension, descending, lateral movements, tilting, rotating, etc.). The aerial robotic vehicle200is illustrated as an example of a robotic vehicle that may utilize various embodiments, but is not intended to imply or require that various embodiments are limited to a quad-rotor aircraft.

FIG. 3illustrates a method300for adjusting the proximity threshold used by a collision avoidance system of a robotic vehicle to accommodate for unpredictability in environmental and other conditions according to some embodiments. With reference toFIGS. 1A-3, the method300may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In block310, the processor of the robotic vehicle may monitor various aspects of the environment or conditions affecting predictability of vehicle control and/or navigation. Such monitoring may involve a wide range of conditions and states that could impact the vehicle control and/or navigation capabilities. For example, the processor may monitor weather conditions by receiving data from various onboard instruments (e.g., cameras, an anemometer, accelerometers, thermometers, humidity sensors, GPS receivers, etc.) and/or by accessing weather-related information sources (e.g., via the Internet) to determine wind conditions, particularly the likelihood of gusts, detect the presence or likelihood of precipitation, detect fog (which can limit visibility for navigation cameras), measure waves and current for waterborne vehicles, detect ice and snow for land based vehicles, and the like. Such environmental conditions can change rapidly or affect the control of a robotic vehicle faster than a control system can respond, and thus can decrease the precision with which the robotic vehicle can maneuver and maintain navigational/positional control. Additionally, the processor may monitor conditions of the robotic vehicle that may affect the precision of maneuvers and navigation, such as structural integrity, payload position and movements, propeller conditions, and the like.

Also as part of the operations in block310, the processor may monitor images from a camera or cameras used in navigation and obstacle detection to determine whether there are any conditions that may impact computer vision processing algorithms used for navigation and collision avoidance. For example, a number of different lighting and background conditions may affect the ability of such cameras to recognize surfaces, determine distances to surfaces, track objects within images, or otherwise perform certain image processing operations. Such conditions may affect the ability of the robotic vehicle to navigate around objects or even accurately determine the distance to objects, and thus potentially compromising the ability of the collision avoidance system to steer clear of objects. Examples of conditions that could impact computer vision algorithms include: flying over monochrome surfaces (e.g., solid black/white/red/green); flying over highly reflective surfaces; flying over water or transparent surfaces; flying in an area where the lighting conditions change frequently or drastically; flying over extremely dark (e.g., lux<10) or bright (e.g., lux>100,000) surfaces; flying over surfaces without clear patterns or texture; flying over surfaces with identical repeating patterns or textures (e.g., tile); and flying over small and fine objects (e.g., tree branches and power lines).

In determination block320, the processor may determine whether there are any environmental or other conditions resulting in unpredictable impacts on robotic vehicle maneuvering or navigation that could compromise vehicle control or navigation. This determination may involve comparing potential forces on the robotic vehicle to control margins available to the vehicle control system. For example, the processor may determine that control and/or navigation of the robotic vehicle is or could be compromised if the forces on the vehicle from the environment or other conditions exceed or could exceed the ability of the vehicle control system to compensate.

In response to determining that control and/or navigation of the robotic vehicle are unlikely to be compromised by environmental or other conditions (i.e., determination block320=“No”), the processor may continue to monitor the environment in block310.

In response to determining that control and/or navigation of the robotic vehicle are or could be compromised by environmental or other conditions (i.e., determination block320=“Yes”), the processor may adjust the proximity threshold consistent with the impact on or potential compromise of control and/or navigation due to the environmental or other conditions. For example, if the processor determines that environmental conditions (e.g., wind gusts) could result in the robotic vehicle being moved off position or course by a meter before the vehicle can react and recover, the processor may increase the proximity threshold be about one meter. Thus, adjusting the proximity threshold consistent with the environment's actual or potential impact on control and/or navigation enables the collision avoidance system to avoid objects by at least the default proximity threshold despite the potential that the environmental conditions could momentarily and unpredictably divert the robotic vehicle.

In block340, the processor may monitor the environmental or other conditions affecting or potentially affecting the robotic vehicle's control and/or navigation for change.

In determination block350, the processor may determine whether the robotic vehicle's control and/or navigation are still compromised or potentially compromised by the monitored environmental or other conditions. While the processor determines that the robotic vehicle's control and/or navigation are still compromised or potentially compromised by the monitored environmental or other conditions (i.e., determination block350=“Yes”), the processor may continue to monitor the environmental or other conditions affecting or potentially affecting the robotic vehicle's control and/or navigation for change in block340.

In response to determining that the robotic vehicle's control and/or navigation are no longer compromised or potentially compromised by the monitored environmental or other conditions (i.e., determination block350=“No”), the processor may return the proximity threshold to a default value in block360, and continue monitoring various aspects of the environment or conditions affecting predictability of control and/or navigation in block310.

Thus, the method300enables the processor of a robotic vehicle to adjust the proximity threshold used in the collision avoidance system to accommodate unpredictability and reduced precision of the maneuvering and navigational control of the robotic vehicle as a result of environmental or other conditions.

FIG. 4illustrates a method400for operating a robotic vehicle according to some embodiments. With reference toFIGS. 1A-4, the method400may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In block410, a processor of the robotic vehicle may provide control signals to control one or more motors of the robotic vehicle to execute user commands (e.g., a user controlled operation) or a preloaded flight plan while avoiding collisions by at least the proximity threshold (as may be adjusted as described). Initially the collision avoidance system may be operating using a default proximity threshold to determine when evasive maneuvers should be implemented when approaching obstacles. As the operations proceed, the proximity threshold may be adjusted to accommodate objects as described.

In block420, the processor may obtain data from one or more sensors that are configured to detect a presence of one or more objects. Such sensors may include ranging sensors, such as radar, sonar, and lidar, which may be used to detect the presence in range to an object. Such sensors may also include imaging sensors, such as a camera or a set of cameras, that may be used to classify objects. For example, many robotic vehicles are equipped with cameras and a collision avoidance system capability (which may be implemented within a main controller) that uses data from the cameras to identify and avoid colliding with objects while operating under user control, autonomously or semi-autonomously. As another example, some robotic vehicles are equipped with cameras and a navigation system (which may be implemented within a main controller) configured for virtual image odometry (VIO) and/or simultaneous localization and mapping (SLAM) in which data from the cameras are used to identify objects and obstacles for use in autonomous navigation. Similarly, a processor (e.g., a main controller) of robotic vehicles equipped with radar and/or lidar sensors may use data from such sensors, either alone or in combination with data from cameras, to identify objects and obstacles for use in navigation and/or collision avoidance. In some implementations, the processor may detect and locate objects in the vicinity of the robotic vehicle based on ranging sensors, such as radar, sonar and/or lidar returns. In some implementations, the processor may receive image data from multiple cameras and/or radar or lidar sensors, which may enable the processor to determine distances to objects (e.g., through stereoscopy) as well as observe objects over a wide angle. Such sensor data may be stored in local memory, such as a buffer, to support data processing in subsequent operations.

In block430, the processor may analyze data obtained from the one or more sensors to recognize and classify objects in the vicinity of the robotic vehicle. In some embodiments, the processor may use image recognition methods to distinguish objects within images and analyze the shapes of the objects to recognize or classify the objects. For example, the processor may compare image data to object recognition models to determine whether objects can be recognized as particular objects (e.g., the user based on facial recognition) or classified as certain types of objects (e.g., people, animals, trees, cars, buildings, etc.). As another example, using camera image data, the processor may perform image processing analysis to detect edges, masses and other features characteristic of objects within the field of view. As another example, the processor may detect objects in the vicinity of the robotic vehicle based on radar and/or lidar returns. In some embodiments, radar and/or lidar data may be used to detect and locate nearby objects, with that information then leveraged in image processing to characterize the detected objects.

As a further operation in block430, the processor may analyze detected objects to determine an appropriate classification for each object. For example, the processor may perform image recognition algorithms on image data to determine whether an object is recognized as a particular type. Such image recognition processes may compare images of a particular object to databases of classified or classifiable objects to determine whether there is a likelihood of a match. Such image recognition processes may use machine learning to develop models for use in assigning classifications to objects.

In some embodiments, the processor may categorize objects in terms of a few broad classifications in block430, such as whether the objects are animate or inanimate, or whether the type of object is known to be fragile or not, etc. In some embodiments, the processor may determine specific categories of objects in block430, such as recognizing and classifying objects as adults, children, dogs, cats, trees, etc. In some embodiments, the processor may determine specific identities of objects in block430, such as recognizing particular individuals (e.g., using facial recognition) or particular animals (e.g., the family dog) for which the processor may be trained (e.g., through an initial training routine) and for which particular (e.g., user-assigned) unpredictability ratings may be specified. The more specific the classification made in block430, the more different proximity thresholds and/or unpredictability ratings may be assigned to various objects by the processor.

As part of classifying objects in block430, the processor may determine that some objects cannot be classified, in which case the determined classification may be unknown or a default classification. In some embodiments, the operations of classifying objects in block430may be performed for all detected objects in the vicinity of the robotic vehicle. In some embodiments only the closest or closest few objects may be classified in block430.

As part of recognizing or classifying objects in block430, the processor may associate such objects with entries within a database that characterizes objects or classes of objects with a parameter characterizing the unpredictability of the object. For example, objects classified as solid structures (e.g., buildings, trees, etc.) may be characterized with a low unpredictability parameter (or high predictability parameter) since such structures typically do not move in an unpredictable manner. As another example, animals may be characterized with a relatively high unpredictability parameter (or low predictability parameter) since animals move and may do so in an unpredictable manner. In some embodiments, the parameter of unpredictability may be binary, and thus either predictable (e.g., buildings and similar structures) or unpredictable (e.g., any object that is moving or can move). Objects that are not recognized or classified by the processor may be assigned a default unpredictability parameter.

In determination block440, the processor may determine whether any of the objects in the vicinity of the robotic vehicle are classified as being unpredictable. For example, the processor may determine whether any object's unpredictability parameter exceeds a particular threshold.

In response to determining that at least one nearby object is classified as being unpredictable (i.e., determination block440=“Yes”), the processor may adjust the proximity threshold used by the collision avoidance system as appropriate for any objects classified as unpredictable. In some embodiments in which different objects may have different assigned levels of unpredictability, the processor may adjust the proximity threshold consistent with the most unpredictable object that has been recognize and classify. In some embodiments in which different objects may have different assigned levels of unpredictability, the processor may adjust the proximity threshold in a manner that gives greater weight to objects along the path of travel or approaching the robotic vehicle. For example, the processor may identify those objects presenting the greatest risk of collision, and adjust the proximity threshold consistent with those objects.

In response to determining that no nearby objects are classified as being unpredictable (i.e., determination block440=“No”), the processor may maintain the use of a default proximity threshold by the collision avoidance system in block460. In situations in which the proximity threshold has been adjusted to be different than the default proximity threshold in block450, the processor may return the proximity threshold to the default value in block460.

The operations of the method400may be performed continuously as the robotic vehicle operates in the environment. Thus, the method400enables a robotic vehicle to operate with a collision avoidance system using a proximity threshold that adjusts for the degree to which detected objects may behave or move in an unpredictable manner. This capability may improve the operation of robotic vehicles by enabling maneuvering closer to fixed or immovable objects (e.g., walls) while giving wider berth to movable objects, such as animals, that may move quickly and in an unpredictable manner.

FIG. 5illustrates a method500for operating a robotic vehicle in a manner that adjusts the collision avoidance system proximity threshold to account for unpredictability in both environmental and other conditions as well as unpredictability of nearby objects according to some embodiments. With reference toFIGS. 1A-5, the method500may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In the method500, some operations of the method300and the method400may be performed more or less in parallel. Thus, determinations of proximity threshold adjustments made by the robotic device processor to accommodate unpredictability in environmental and other conditions may be determined in parallel and used in conjunction with adjustments to the proximity threshold determined by the processor based upon recognition and/or classification of nearby objects. Thus, the method500may implement operations in blocks310,320,340, and350of the method300and in blocks410-440of the method400as described.

In block310, the processor may monitor the environment and state of the robotic vehicle for conditions that may potentially affect the predictability of vehicle control and/or navigation as described in the method300. Also as part of the operations in block310, the processor may monitor images from a camera or cameras used in navigation and obstacle detection to determine whether there are any conditions that may impact computer vision processing algorithms used for navigation and collision avoidance. In determination block320, the processor may determine whether any such conditions exist that could compromise the control and/or navigation capabilities of the robotic vehicle as described in the method300.

In response to determining that the control and/or navigation of the robotic vehicle may be compromised by environmental or other conditions (i.e., determination block320=“Yes”), the processor may determine a proximity threshold environment adjustment based upon the determined impact or potential impact on control/navigation predictability. Thus, instead of adjusting the proximity threshold as in method300, the processor may determine an adjustment that should be added to other proximity threshold adjustments to account for unpredictability in or due to environmental or other conditions. This proximity threshold environmental adjustment may be stored in memory, such as in a buffer used by the collision avoidance system or vehicle control system. Thus, while environmental or other conditions exhibit unpredictability that could compromise the control or navigation of the robotic vehicle, a suitable adjustment to the proximity threshold may be maintained in memory for use by the processor in determining an appropriate proximity threshold for use by the collision avoidance system in block560.

In block340and determination block350, the processor may monitor the conditions determined to affect or potentially affect the control and/or navigation of the robotic vehicle to detect when such conditions change such that the control/navigation of the vehicle remains compromised as described in the method300.

In response to determining that environmental or other conditions have changed such that the control and/or navigation of the robotic vehicle are no longer compromised (i.e., determination block350=“No”), the processor may reset the proximity threshold environment adjustment to a default value in block520, such as setting the adjustment to 0. Thus, when there are no environmental or other conditions compromising the control or navigation of the robotic vehicle, the processor may determine an appropriate proximity threshold for use by the collision avoidance system based solely upon recognized or classified nearby objects (if any) in block560.

In block410, the robotic vehicle processor may provide control signals to control one or more motors of the robotic vehicle to execute user commands (e.g., a user controlled operation) or a preloaded flight plan while using the collision avoidance system to avoid colliding with objects and obstacles.

In block420, the processor may obtain data from one or more sensors that are configured to detect the presence of one or more objects in the vicinity of the robotic vehicle as described in the method400.

In block430, the processor (or another processor configured to perform such analysis) may analyze the obtained sensor data to recognize and/or classify objects that are in the vicinity of the robotic vehicle as described.

In determination block440, the processor may determine whether there are any objects present in the vicinity of the robotic vehicle that are classified as being unpredictable as described in the method400.

In response to determining that no nearby obstacles are classified as being unpredictable (i.e., determination block440=“No”), the processor may adjust the default proximity threshold by the proximity threshold environment adjustment in block550such as by increasing the default proximity threshold by the proximity threshold environment adjustment determined in block510. Thus, when no objects are nearby, the collision avoidance system may use a proximity threshold based upon the default value adjusted to account for unpredictability in the environment or other conditions.

In response to determining that one or more nearby obstacles are classified as being unpredictable (i.e., determination block440=“Yes”), the processor may adjust the proximity threshold used by the collision avoidance system to a distance corresponding to at least one of the object unpredictability classifications plus an adjustment to account for unpredictability in the environment or other conditions in block560. Thus, the processor may set the proximity threshold by using a proximity threshold corresponding to or appropriate for one or more nearby objects based upon the unpredictability of such objects, and further adjust the proximity threshold to accommodate the degree to which control and/or navigation may be compromised by unpredictable conditions in the operating environment.

The operations of the method500may be performed continuously during operations so that the proximity threshold is adjusted to account for changing environmental and other conditions, and as the robotic vehicle detects and approaches various objects. Thus, the method500improves operations of robotic vehicles by enabling the collision avoidance system to dynamically adapt to changing conditions and encountered objects or obstacles.

FIG. 6illustrates a method600for operating a robotic vehicle that enables path planning while accounting for unpredictability of encountered objects and 4 impacts on maneuvering and/or navigation due to uncertainty in the environment and other conditions according to some embodiments. With reference toFIGS. 1A-6, the method400may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In some embodiments, the processor of the robotic vehicle may use object classifications to adjust proximity thresholds for a variety of detected objects to account for their unpredictability and for unpredictability in environmental and other conditions while performing path planning around such objects. By adjusting the proximity threshold used by the collision avoidance system to account for unpredictability in environmental and other conditions using the method300and evaluating each of the appropriate proximity thresholds for each detected object within the vicinity of the robotic vehicle using some of the operations of the method400, the processor may develop a detour or path plan alternative in the method600that enables the robotic vehicle to maneuver around the objects in a manner that avoids approaching any objects closer than its corresponding proximity threshold. By performing such path planning, the robotic vehicle may follow an efficient path around or through detected objects compared to using the collision avoidance system to avoid colliding with individual obstacles, which could result in the robotic vehicle having to backtrack or follow a random path through a plurality of obstacles. Various methods for accomplishing such path planning are possible. The method600illustrated inFIG. 6provides an example of one method that may be implemented for this purpose.

In blocks310-360, the processor may monitor environmental and other conditions and adjust the proximity threshold accordingly as described in the method300.

In block610, the processor of the robotic vehicle may be controlling one or more motors of the robotic vehicle to execute a preplanned of flight.

In block420, the processor may obtain data from one or more sensors that are configured to detect the presence of objects in the vicinity of the vehicle. As described, such sensors may include one or more cameras, radar, sonar, lidar, etc. Data from such sensors may be stored in memory, such as buffer memory, to enable analysis by the processor.

In block430, the processor may analyze the obtained sensor data to recognize and classify the objects in the vicinity of the vehicle. Such analyses may implement one or more of the operations of the like numbered block of the method400as described.

In determination block620, the processor may determine whether any of the identified and classified objects present obstacles or potential obstacles to the robotic vehicle. In particular, the processor may determine whether any of the objects identified in block430are close to the preplanned flight path.

In response to determining that no obstacles or potential obstacles are present in the vicinity of the robotic vehicle (i.e., determination block620=“No”), the processor may continue to control one or more motors of the robotic vehicle to execute the preplanned flight path in block610.

In response to determining that one or more classified objects do or could present obstacles to the robotic vehicle (i.e., determination block620=“Yes”), the processor may generate a virtual map of the objects in block630that adds in an exclusion perimeter around each object's volume that is based upon the current (i.e., adjusted or default) proximity threshold accounting for environmental or other conditions and that object's classification and uncertainty as determined in block430. Thus, instead of adjusting the proximity threshold used in the collision avoidance system for individual objects, the processor may use the adjusted proximity threshold plus a radial distance appropriate for the object's classification and unpredictability to extend the boundary of an object as represented in the virtual map generated in block630. A distance may be used for adding an exclusion perimeter around objects (i.e., no extension beyond the adjusted priority threshold) for which a classification is not determined.

In block640, the processor may use the map generated in block630to determine a detour route around or through the identified obstacles that ensures the robotic vehicle remains outside of the respective proximity threshold for all obstacles under the current environmental and other conditions. The processor may accomplish such path planning using any of a variety of path planning algorithms. Using the proximity threshold adjusted for current environmental and other conditions plus object classification-specific additional margins to add exclusion perimeters around objects in block630facilitates planning a path that maneuvers around objects that takes into account unpredictability due to object movements and conditions that compromise or may compromise maneuvering and navigational control.

In block650, the processor may control one or more motors of the robotic vehicle to execute the detour. Doing so, the processor may continue to obtain data from various sensors in block420and analyze the sensor data to recognizing classify nearby objects in block430as described. Once the robotic vehicle has cleared the obstacles, such as by completing the detour (i.e., determination block620=“No”), the processor may return to controlling the one or more motors of the robotic vehicle to execute the preplanned flight path in block610, and repeat the operations of the method600as described. Also, the operations in blocks310-360may be performed continuously to enable the proximity threshold to be adjusted dynamically to account for changes in environmental and other conditions.

FIG. 7illustrates a method700for detecting and classifying objects for use in the method600according to some embodiments. With reference toFIGS. 1A-7, the method700may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In block702, the processor may obtain data from image sensors, such as one or more cameras positioned on the robotic vehicle. The processor may also obtain other sensor data, such as radar or lidar data, that is useful for determining the relative location of objects. Such data may be stored in local memory for processing, such as buffer memory.

In block704, the processor may analyze the obtained image and other data to identify the presence and location of the imaged objects. Such image analysis may involve identifying edges zones of different colors and other types of processes that are typically used to identify objects within images. In some embodiments, the robotic vehicle may be equipped with stereoscopic cameras which may enable the processor to determine the distance to various objects using stereoscopy. In embodiments in which the robotic vehicle is equipped with only a monocular camera, distances to various objects may be determined based on the shift in position of objects from one frame to the next as the robotic vehicle moves to the environment. In embodiments in which the robotic vehicle is equipped with radar and/or lidar, the distances to objects may be determined using data from those sensors. In determining the location of objects, the processor may generate a file or database of object coordinates in memory that enables the processor to generate a map of objects in subsequent operations.

Each of the objects that are identified in block704may be individually analyzed using image recognition processes. To do so, the processor may implement a loop to individually investigate each object. Thus, in block706, the processor may select one of the identified objects, and perform object recognition processing on of the image data for the selected object to determine the classification in block708. As described, such image recognition processing may involve comparing image data to the database of classified objects to determine whether there is a close match. Such image recognition processes may involve the use of machine learning techniques.

In determination block710, the processor may determine whether a classification is assigned to the selected object.

In response to determining that no classification is assigned to the selected object (i.e., determination block710=“No”), the processor may assign a default exclusion perimeter distance to the selected object in block712.

In response to determining that a classification is assigned to the selected object (i.e., determination block710=“Yes”), the processor may assign to the selected object in exclusion perimeter distance corresponding to the classification of the selected object in block714.

In determination block716, the processor may determine whether there is another object within the image data to be classified. If so (i.e., determination block716=“Yes”), the processor may select another identified object in block706and repeat the operations of blocks708-714as described. When all objects have been classified (i.e., determination block716=“No”), the processor may proceed to generate a map of objects adding the exclusion perimeter based on each objects classification in block630of the method600as described.

FIG. 8illustrates an example method800for determining control and/or navigational unpredictability and magnitude by monitoring control and attitude of the robotic vehicle according to some embodiments. With reference toFIGS. 1A-8, the method800may be performed by a processor, such as a processor (220) within a processing device (e.g.,210) of a robotic vehicle (e.g.,200) to detect obstacles (e.g.,120) and perform an action in response.

In block802, the processor may control the robotic vehicle to maintain a set position or follow a defined path so that the controllability of the robotic vehicle under the current conditions can be monitored. In some embodiments, this may involve attempting to hold the robotic vehicle (e.g., an aerial or waterborne vehicle) in a set position, such as a hover. For example, an aerial robotic vehicle may use camera data and GPS data in a flight control system in an attempt to hover in a fixed geographic location at a fixed altitude. For example, a waterborne robotic vehicle may use camera data and GPS data in a maneuvering control system in an attempt to remain at a fixed coordinate or to follow a defined path (e.g., a straight line) on the water. For example, a land-based robotic vehicle may use camera data and GPS data in a navigation system in an attempt to follow a defined path (e.g., a straight line) on a roadway. Maneuvering to remain in a fixed position or follow a defined path will require the control system to adjust for any forces acting on the robotic vehicle, such as from wind, waves, precipitation, ice, or snow (such as for land-based vehicles), thereby enabling measurements to be made of the forces acting on the robotic vehicle from environmental conditions.

In block804, the processor may monitor for deviations from the set position or defined path caused by environmental conditions. In some embodiments, the processor may use camera data and GPS data to detect when the robotic vehicle deviates from the set position or defined path. Such deviations may occur when environmental forces acting on the robotic vehicle exceed the control capabilities of the vehicle control system. Momentary deviations may occur when unpredictable changes in environmental forces act on the robotic vehicle before the vehicle control system is able to respond and return to the set position or defined path. For example, a wind gust may blow an aerial robotic vehicle away from a hovering position momentarily until the vehicle control system can apply sufficient thrust to overcome the force of the wind and return to the hovering position. Such monitoring of deviations from the set position or defined path may be performed over a period of time to capture sufficient deviation events to characterize the unpredictable nature of the environment in terms of forces and periodicity.

In block806, the processor may determine the control and/or navigational unpredictability effects of the environment based on the deviations observed in block804. In some embodiments, this may involve determining a maximum deviation that was observed in block804. In some embodiments, this may involve using statistical analysis of the deviations observed in block804to calculate a maximum probable deviation.

The results of the determination made in806may be used by the processor in block330of the methods300or600or in block510of the method500as described.