Patent ID: 12242282

DETAILED DESCRIPTION

In embodiments of the present disclosure, a UAV uses machine learning models to process camera images to detect landing spaces. In some embodiments, the landing spaces may be associated with a charging pad which can inductively charge batteries of the UAV once the UAV lands thereon, such that the machine learning models are configured to detect charging pads in the camera images. To improve the precision and the robustness of the detection, the machine learning models in embodiments of the present disclosure apply pixel-by-pixel labels to the images to associate the pixels with either depicting a landing space or not depicting a landing space, and heuristics may then be applied to groups of labeled pixels to detect actual landing spaces in the imagery. Upon identifying a landing space, the location of the landing space in the imagery may be used to help control the position of the UAV. For example, in some embodiments, the machine learning models may further label pixels as pixels for occupied landing spaces and unoccupied landing spaces in order to help plan a path to an unoccupied landing space.

FIG.1AandFIG.1Billustrate an aerial vehicle or UAV100, in accordance with an embodiment of the present disclosure. The illustrated embodiment of UAV100is a vertical takeoff and landing (VTOL) unmanned aerial vehicle (UAV) that includes separate propulsion units112and propulsion units108for providing horizontal and vertical propulsion, respectively. UAV100is a fixed-wing aerial vehicle, which as the name implies, has a wing assembly124that can generate lift based on the wing shape and the vehicle's forward airspeed when propelled horizontally by propulsion units112.FIG.1Ais a perspective top view illustration of UAV100whileFIG.1Bis a bottom side plan view illustration of UAV100.

The illustrated embodiment of UAV100includes a fuselage120. In one embodiment, fuselage120is modular and includes a battery module, an avionics module, and a mission payload module. These modules are detachable from each other and mechanically securable to each other to contiguously form at least a portion of the fuselage120or UAV main body.

The battery module includes a cavity for housing one or more batteries for powering UAV100. The avionics module houses flight control circuitry of UAV100, which may include a processor and memory, communication electronics and antennas (e.g., cellular transceiver, Wi-Fi transceiver, etc.), and various sensors (e.g., global positioning sensor, an inertial measurement unit (IMU), a magnetic compass, etc.). The mission payload module houses equipment associated with a mission of UAV100. For example, the mission payload module may include a payload actuator for holding and releasing an externally attached payload. In another embodiment, the mission payload module may include a camera/sensor equipment holder for carrying camera/sensor equipment (e.g., camera, lenses, radar, LIDAR, pollution monitoring sensors, weather monitoring sensors, etc.). Other components that may be carried by some embodiments of the UAV100are illustrated inFIG.2.

The illustrated embodiment of UAV100further includes horizontal propulsion units112positioned on wing assembly124, which can each include a motor, shaft, motor mount, and propeller, for propelling UAV100. The illustrated embodiment of UAV100includes two boom assemblies106that secure to wing assembly124.

The illustrated embodiments of boom assemblies106each include a boom housing118in which a boom is disposed, vertical propulsion units108, printed circuit boards116, and stabilizers102. Vertical propulsion units108can each include a motor, shaft, motor mounts, and propeller, for providing vertical propulsion. Vertical propulsion units108may be used during a hover mode where UAV100is descending (e.g., to a delivery location) or ascending (e.g., following a delivery). Stabilizers102(or fins) may be included with UAV100to stabilize the UAV's yaw (left or right turns) during flight. In some embodiments, UAV100may be configured to function as a glider. To do so, UAV100may power off its propulsion units and glide for a period of time.

During flight, UAV100may control the direction and/or speed of its movement by controlling its pitch, roll, yaw, and/or altitude. For example, the stabilizers102may include one or more rudders104for controlling the UAV's yaw, and wing assembly124may include elevators for controlling the UAV's pitch and/or ailerons110for controlling the UAV's roll. As another example, increasing or decreasing the speed of all the propellers simultaneously can result in UAV100increasing or decreasing its altitude, respectively. The UAV100may also include components for sensing the environment around the UAV100, including but not limited to audio sensor122and audio sensor114. Further examples of sensor devices are illustrated inFIG.2and described below.

Many variations on the illustrated fixed-wing aerial vehicle are possible. For instance, aerial vehicles with more wings (e.g., an “x-wing” configuration with four wings), are also possible. AlthoughFIG.1AandFIG.1Billustrate one wing assembly124, two boom assemblies106, two horizontal propulsion units112, and six vertical propulsion units108per boom assembly106, it should be appreciated that other variants of UAV100may be implemented with more or fewer of these components.

It should be understood that references herein to an “unmanned” aerial vehicle or UAV can apply equally to autonomous and semi-autonomous aerial vehicles. In a fully autonomous implementation, all functionality of the aerial vehicle is automated; e.g., pre-programmed or controlled via real-time computer functionality that responds to input from various sensors and/or pre-determined information. In a semi-autonomous implementation, some functions of an aerial vehicle may be controlled by a human operator, while other functions are carried out autonomously. Further, in some embodiments, a UAV may be configured to allow a remote operator to take over functions that can otherwise be controlled autonomously by the UAV. Yet further, a given type of function may be controlled remotely at one level of abstraction and performed autonomously at another level of abstraction. For example, a remote operator may control high level navigation decisions for a UAV, such as specifying that the UAV should travel from one location to another (e.g., from a warehouse in a suburban area to a delivery address in a nearby city), while the UAV's navigation system autonomously controls more fine-grained navigation decisions, such as the specific route to take between the two locations, specific flight controls to achieve the route and avoid obstacles while navigating the route, and so on.

FIG.2is a block diagram that illustrates various components of a non-limiting example embodiment of a UAV according to various aspects of the present disclosure. The UAV200illustrated inFIG.2may be the same as the UAV100illustrated inFIG.1AandFIG.1B, withFIG.2schematically illustrating some of the components of the UAV100. The UAV200is configured to capture imagery using one or more cameras, label pixels in the images as being associated with landing spaces, and using detected landing spaces to control aspects of the operation of the UAV200. As shown, the UAV200includes a communication interface202, one or more vehicle state sensor devices204, a power supply206, one or more processors208, one or more propulsion devices210, one or more cameras220, and a computer-readable medium212.

In some embodiments, the communication interface202includes hardware and software to enable any suitable communication technology for communicating with other components, including but not limited to a model management computing system310as described below. In some embodiments, the communication interface202includes multiple communication interfaces, each for use in appropriate circumstances. For example, the communication interface202may include a long-range wireless interface such as a 4G or LTE interface, or any other type of long-range wireless interface (e.g., 2G, 3G, 5G, or WiMAX), to be used to communicate with the model management computing system310or other fleet management systems while traversing a route. The communication interface202may also include a medium-range wireless interface such as a Wi-Fi interface to be used when the UAV200is at an area near a start location or an endpoint where Wi-Fi coverage is available. The communication interface202may also include a short-range wireless interface such as a Bluetooth interface to be used when the UAV200is in a maintenance location or is otherwise stationary and waiting to be assigned a route. The communication interface202may also include a wired interface, such as an Ethernet interface or a USB interface, which may also be used when the UAV200is in a maintenance location or is otherwise stationary and waiting to be assigned a route.

In some embodiments, the vehicle state sensor devices204are configured to detect states of various components of the UAV200, and to transmit signals representing those states to other components of the UAV200. Some non-limiting examples of a vehicle state sensor device204include a battery state sensor and a propulsion device health sensor.

In some embodiments, the power supply206may be any suitable device or system for storing and/or generating power. Some non-limiting examples of a power supply206include one or more batteries, one or more solar panels, a fuel tank, and combinations thereof. In some embodiments, the power supply206may include an inductive charging unit that allows a battery of the power supply206to be recharged when the UAV200is resting on an charging pad.

In some embodiments, the propulsion devices210may include any suitable devices for causing the UAV200to travel along the path. For example, the propulsion device210may include devices such as, but not limited to, one or more motors, one or more propellers, and one or more flight control surfaces.

In some embodiments, the cameras220include one or more cameras positioned to capture imagery of areas surrounding the UAV200. For example, a camera220may be positioned vertically to capture imagery directly beneath the UAV200. As another example, a camera220may be positioned at the front of the UAV200and angled forward to capture imagery in the direction of travel of the UAV200. The camera220may have a telephoto lens in order to maximize captured detail and minimize geometric distortions, a wide angle lens in order to maximize the captured area, or any other type of lens. Further, the camera220may capture imagery within the visible light spectrum or a non-visible light spectrum (including but not limited to infrared), and may capture imagery in two dimensions or three dimensions.

In some embodiments, the processors208may include any type of computer processor capable of receiving signals from other components of the UAV200and executing instructions stored on the computer-readable medium212. In some embodiments, the processors208may include one or more general purpose processors. In some embodiments, the processors208may include one or more special purpose processors, including but not limited to graphical processing units (GPUs), vision processing units (VPUs), tensor processing units (TPUs), and/or other processors specially adapted to efficiently perform specific types of computations. In some embodiments, the computer-readable medium212may include one or more devices capable of storing information for access by the processor208. In some embodiments, the computer-readable medium212may include one or more of a hard drive, a flash drive, an EEPROM, and combinations thereof. In some embodiments, the functionality of one or more of the processors208and the computer-readable medium212may be combined into a single device, including but not limited to an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

As shown, the computer-readable medium212has stored thereon a model data store214, an image labeling engine216, and a route traversal engine218. In some embodiments, the image labeling engine216is configured to process images captured by the cameras220using a model stored in the model data store214in order to detect landing spaces in the imagery. In some embodiments, the route traversal engine218is configured to cause the propulsion devices210to control positions of the UAV200based on the locations of landing spaces detected by the image labeling engine216.

FIG.3is a block diagram that illustrates aspects of a non-limiting example embodiment of a model management computing system according to various aspects of the present disclosure. The illustrated model management computing system310may be implemented by any computing device or collection of computing devices, including but not limited to a desktop computing device, a laptop computing device, a mobile computing device, a server computing device, a computing device of a cloud computing system, and/or combinations thereof. The model management computing system310is configured to train one or more machine learning models to detect landing spaces in imagery captured by UAVs200.

As shown, the model management computing system310includes one or more processors302, one or more communication interfaces304, a model data store308, an imagery data store314, and a computer-readable medium306.

In some embodiments, the processors302may include any suitable type of general-purpose computer processor. In some embodiments, the processors302may include one or more special-purpose computer processors or AI accelerators optimized for specific computing tasks, including but not limited to graphical processing units (GPUs), vision processing units (VPUs), and tensor processing units (TPUs).

In some embodiments, the communication interfaces304include one or more hardware and or software interfaces suitable for providing communication links between components. The communication interfaces304may support one or more wired communication technologies (including but not limited to Ethernet, FireWire, and USB), one or more wireless communication technologies (including but not limited to Wi-Fi, WiMAX, Bluetooth, 2G, 3G, 4G, 5G, and LTE), and/or combinations thereof.

As shown, the computer-readable medium306has stored thereon logic that, in response to execution by the one or more processors302, cause the model management computing system310to provide an imagery collection engine312and a model management engine316.

As used herein, “computer-readable medium” refers to a removable or nonremovable device that implements any technology capable of storing information in a volatile or non-volatile manner to be read by a processor of a computing device, including but not limited to: a hard drive; a flash memory; a solid state drive; random-access memory (RAM); read-only memory (ROM); a CD-ROM, a DVD, or other disk storage; a magnetic cassette; a magnetic tape; and a magnetic disk storage.

In some embodiments, the imagery collection engine312is configured to receive imagery captured by one or more UAVs200(or from other sources), and to store the captured imagery in the imagery data store314. The imagery collection engine312is also configured to receive ground truth pixel labels, and to store the ground truth pixel labels along with the labeled images in the imagery data store314to create training data. The model management engine316is configured to train one or more machine learning models based on the training data stored in the imagery data store314, to store the trained machine learning models in the model data store308, and to distributed the trained machine learning models to the UAVs200.

Further description of the configuration of each of these components is provided below.

As used herein, “engine” refers to logic embodied in hardware or software instructions, which can be written in one or more programming languages, including but not limited to C, C++, C #, COBOL, JAVA™, PHP, Perl, HTML, CSS, JavaScript, VBScript, ASPX, Go, and Python. An engine may be compiled into executable programs or written in interpreted programming languages. Software engines may be callable from other engines or from themselves. Generally, the engines described herein refer to logical modules that can be merged with other engines, or can be divided into sub-engines. The engines can be implemented by logic stored in any type of computer-readable medium or computer storage device and be stored on and executed by one or more general purpose computers, thus creating a special purpose computer configured to provide the engine or the functionality thereof. The engines can be implemented by logic programmed into an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or another hardware device.

As used herein, “data store” refers to any suitable device configured to store data for access by a computing device. One example of a data store is a highly reliable, high-speed relational database management system (DBMS) executing on one or more computing devices and accessible over a high-speed network. Another example of a data store is a key-value store. However, any other suitable storage technique and/or device capable of quickly and reliably providing the stored data in response to queries may be used, and the computing device may be accessible locally instead of over a network, or may be provided as a cloud-based service. A data store may also include data stored in an organized manner on a computer-readable storage medium, such as a hard disk drive, a flash memory, RAM, ROM, or any other type of computer-readable storage medium. One of ordinary skill in the art will recognize that separate data stores described herein may be combined into a single data store, and/or a single data store described herein may be separated into multiple data stores, without departing from the scope of the present disclosure.

FIG.4is a flowchart that illustrates a non-limiting example embodiment of a method of training a machine learning model to label pixels of an image according to various aspects of the present disclosure.

From a start block, the method400proceeds to block402, where an imagery collection engine312of a model management computing system310receives a plurality of images from one or more UAVs200. In some embodiments, the UAVs200may transmit imagery to the imagery collection engine312as still images or video. In some embodiments, the UAVs200may transmit telemetry information to be associated with the image, including but not limited to a heading, a pitch, a yaw, and an altitude of the UAV200at a time when the image was captured.

In some embodiments, the UAVs200may transmit the imagery via a long-range wireless interface while in flight. In some embodiments, the UAVs200may transmit the imagery via a medium-range wireless interface, a short-range wireless interface, a wired interface, or via exchange of a removable computer-readable medium after landing at a landing space.

At block404, the imagery collection engine312stores the plurality of images in an imagery data store314of the model management computing system310. In some embodiments, the plurality of images may be stored in temporal order, particularly if the plurality of images was received as video, such that multiple images captured in succession may be processed together.

FIG.5Aillustrates an example of an image502that may be received by the imagery collection engine312from a UAV200. The image502depicts a portion of an operating “nest,” or a collection of landing spaces, from which UAVs are arriving and departing. The nest includes a plurality of unoccupied landing spaces510, a plurality of occupied landing spaces514, and a plurality of geofiducials508. In the depicted nest, the location of each landing space is defined by a charging pad on which a UAV may land in order to inductively charge.

The image502illustrates some of the challenges that can be encountered while detecting landing spaces in images of nests captured by UAVs using traditional techniques such as classifier models (e.g., convolutional neural networks) that are trained to detect objects in images. For example, only the unoccupied landing spaces510located in the center of the image502are fully visible. Otherwise, the unoccupied landing spaces510around the perimeter of the image502are not fully visible, which is likely to confuse classifier models even though the unoccupied landing spaces510around the perimeter should still be correctly identified. The image502also shows a shadow512of a UAV200, which is likely to be common over the nest during sunny weather and is also likely to confuse classifier models. Though the features ofFIG.5Aare clearly visible (i.e., the features are depicted as visible with solid lines), one will recognize that in other real-world images captured by UAVs200, environmental factors such as haze, motion artifacts, sun glare, low resolution, and other conditions may make it difficult or impossible to precisely scan the geofiducials508in order to extract useful information from them.

Returning toFIG.4, the method400then proceeds to a for-loop defined between a for-loop start block406and a for-loop end block412, wherein each image of the plurality of images is processed to add it to a set of training data.

From the for-loop start block406, the method400proceeds to block408, where the imagery collection engine312collects ground truth pixel labels for the image. In some embodiments, the imagery collection engine312may generate a user interface that presents the image to a user, and the user may select pixel labels for the pixels of the image using any suitable technique, including but not limited to drawing outlines around groups of pixels and then assigning pixel labels to the pixels within the outlined shapes. In some embodiments, the imagery collection engine312may submit the image for tagging to a crowd computing platform, such as the Crowdsource app by Google Inc., or Mechanical Turk by Amazon Technologies, Inc.

FIG.5Billustrates an example of ground truth pixel labels504applied to the image502. The unoccupied landing spaces510have been labeled with unoccupied landing space pixel ground truth labels518, and the occupied landing spaces514have been labeled with occupied landing space pixel ground truth labels520. The remainder of the pixels have been labeled with non-landing space pixel ground truth labels516. In some embodiments, instead of using non-landing space pixel ground truth labels516, any pixel not labeled as an occupied landing space pixel or an unoccupied landing space pixel may be assumed to be a non-landing space pixel.

Returning again toFIG.4, at block410, the imagery collection engine312stores the ground truth pixel labels in association with the image in a training data set in the imagery data store314. The method400then proceeds to the for-loop end block412. If further images remain to be processed, then the method400returns to for-loop start block406to process the next image. Otherwise, if all of the images have been processed, the method400proceeds to block414.

At block414, a model management engine316of the model management computing system310trains a machine learning model based on the training data set in the imagery data store314. Any suitable type of machine learning model that can be trained to generate a pixel-by-pixel segmentation for an input image may be used. In some embodiments, an encoder-decoder model may be used. The encoder-decoder model may use any suitable architecture, including but not limited to a MobileNet model as an encoder, for which the generated features are then combined and rebuilt into a new image by the decoder. Any suitable training technique, including but not limited to gradient descent and/or an Adam optimizer, may be used.

In some embodiments, the features provided to the machine learning model during training may include the telemetry information provided by the UAV200that captured the images. In some embodiments, more than one image may be provided at once to the machine learning model. For example, a sequence of consecutively captured images (e.g., images captured at time t−1, . . . , t−n) may be provided as the input to the machine learning model so that the machine learning model can be trained to more easily handle uncertainty caused by transient conditions when analyzing a given image (e.g., an image captured at time t).

In some embodiments, the images used for training in the method400may be thoughtfully collected in order to ensure the highest possible performance of the trained machine learning model. For example, in some embodiments, the training data may be organized into buckets by altitude (e.g., a 0-5 meter altitude bucket, a 5-10 meter altitude bucket, a 10-15 meter altitude bucket, etc., or any other suitable ranges), and appropriate amounts (e.g., equal amounts or approximately equal amounts) of training data may be selected from each bucket to ensure that the trained machine learning model will have adequate performance at a variety of altitudes. As another example, in some embodiments, a significant amount (e.g., 50% or more) of the training data may not depict any landing spaces, thus allowing the machine learning model to learn the distinctive features present at landing spaces that are not present elsewhere.

FIG.5Cillustrates a result of processing the image502with the trained machine learning model. As shown, the pixel labels506include unoccupied landing space pixel labels524for the unoccupied landing spaces510, occupied landing space pixel labels526for the occupied landing spaces514, and non-landing space pixel labels522for the remainder of the pixels. One will note that the size and shape of the pixel labels are slightly different than the actual sizes and shapes of the landing spaces to illustrate the uncertainty that may be present along the edges of the labeled areas. One will also note a small area of pixels that was mislabeled as unoccupied landing space pixel labels524due to the presence of the shadow512in the image502. Despite the uncertainty along the edges and the small area of mislabeled pixels, the UAV200can use various heuristics to accurately identify the locations of the landing spaces as described below in the description ofFIG.7.

Returning toFIG.4, at block416, the model management engine316stores the trained machine learning model in a model data store308of the model management computing system310. In some embodiments, once trained, the machine learning model may be sparsified before storage in the model data store308in order to improve performance speed of the machine learning model during processing on the UAVs200. The method400then proceeds to an end block and terminates.

In some embodiments, the landing spaces which the machine learning model is trained to label are charging pads. Charging pads may have a surface texture, an internal structure, a pattern of conductors, or other features that are visible in various lighting conditions and from various angles that are suitable for segmentation by the machine learning model and are a desirable location for landing. While detection of charging pads is a particularly useful embodiment, it should not be seen as limiting, and in other embodiments, other types of landing spaces may be detected.

FIG.6A-FIG.6Billustrate the application of pixel labels to an aerial image taken from an oblique angle with respect to several landing spaces according to various aspects of the present disclosure. These images illustrate more reasons why the image segmentation problem for landing spaces is difficult in the UAV imagery context and is better served using the pixel-by-pixel segmentation described above than with a more common object detector.

InFIG.6A, an oblique image602is shown, which may be obtained by a UAV200at a low altitude and positioned to the side of a first landing space604, a second landing space606, and a third landing space608. Landing spaces may be a standard shape (such as square) and a standard size (such as a meter wide). While it would seem that an object detector would have an easy time detecting such standard shapes,FIG.6Aillustrates why it is nevertheless difficult: the first landing space604, second landing space606, and third landing space608all appear different sizes and shapes due to the distortion caused by the low altitude. Specifically, the edges of the second landing space606are distorted to appear as curved lines and to be out of square, which would therefore be unlikely to be successfully identified by an object detector. Further, the third landing space608is mostly outside of the frame of the oblique image602, and so would also be unlikely to be successfully identified by an object detector. The object detector is similarly unlikely to detect the UAV610present on the third landing space608because it is mostly outside of the frame of the oblique image602. One will recognize that an additional difficulty will be added by the fact that the UAV200may be in a variety of rotational attitudes with respect to the landing spaces, and so a large amount of training data in different rotational arrangements would need to be provided to train an object detector, which may not be able to successfully be trained.

FIG.6Billustrates the result of processing the oblique image602with the machine learning model trained by the method400described above. As shown, pixel labels612are correctly applied to each of the landing spaces, with unoccupied landing space pixel labels614being applied to the first landing space604and the second landing space606, occupied landing space pixel labels616being applied to the third landing space608, and non-landing space pixel labels618applied to the remainder of the pixels. Since the machine learning model is trained to perform a pixel-by-pixel segmentation, the oblique angle and the distorted shapes do not affect the segmentation.

FIG.7is a flowchart that illustrates a method of controlling a UAV by automatically detecting landing spaces according to various aspects of the present disclosure. The method700uses the machine learning model trained by the method400described above to detect landing spaces, and uses the detected landing spaces to determine and apply control strategies for the UAV.

From a start block, the method700proceeds to block702, where a UAV200receives a machine learning model configured to apply labels to pixels including an unoccupied landing space pixel label, an occupied landing space pixel label, and a non-landing space pixel label. The machine learning model may be received from the model management computing system310, and may be trained using the method400described above or by any other suitable method. At block704, the UAV200stores the machine learning model in a model data store214of the UAV200.

At block706, an image labeling engine216of the UAV200receives at least one image from a camera of the UAV200. In some embodiments, the image labeling engine216may receive the image at some point during the flight of the UAV200, such as shortly after takeoff while hovering over the nest, during cruise, or shortly before landing while again located over the nest.

At block708, the image labeling engine216uses the machine learning model to apply labels to each pixel of the at least one image. In some embodiments, the image labeling engine216may provide the image as input to the machine learning model, and the machine learning model may produce a pixel-by-pixel labeling of the image. In some embodiments, the image labeling engine216may provide telemetry information, including but not limited to a heading, a pitch, a yaw, and an altitude of the UAV200as additional input to the machine learning model along with the image. In some embodiments, the image labeling engine216may provide more than one image, such as a set of consecutively captured images, as input to the machine learning model (e.g, images captured at time t−1, . . . , t−n to help label pixels in an image captured at time t).

At block710, the image labeling engine216identifies one or more landing spaces in the at least one image based on the labels of the pixels. In some embodiments, the image labeling engine216may perform post-processing on the image labels in order to increase their fidelity. For example, the image labeling engine216may perform morphological closing, smoothing, or other operations in order to eliminate holes and irregularities in the labeled areas.

Instead of or in addition to the post-processing steps, the image labeling engine216may apply one or more heuristics in order to detect landing spaces. For example, the image labeling engine216may determine a minimum number of pixels that would be present in a landing space based on an altitude of the UAV200, and may ignore groups of pixels that are labeled as a landing space that are smaller than the determined minimum number of pixels. As another example, the image labeling engine216may determine a rough shape that the landing space should appear as based on a pose of the UAV200, and may ignore groups of labeled pixels that are not within a threshold similarity of the expected shape. As another example, the image labeling engine216may determine a proportion of pixels that should be labeled as an unoccupied landing space as opposed to an occupied landing space in order to confirm that a landing space is an unoccupied landing space (e.g., at least 70% of contiguous pixels labeled as a landing space should be labeled as unoccupied instead of occupied, or any other suitable threshold value), otherwise the landing space is identified as an occupied landing space. By using such heuristics, the method700can avoid misidentifying groups of pixels such as the small group of unoccupied landing space pixel labels524ofFIG.5Cassociated with the shadow512ofFIG.5A. In some embodiments, the image labeling engine216may label pixels in multiple consecutively captured images, and may identify landing spaces based on whether the labels have remained consistent for at least a threshold amount of time in order to avoid incorrectly identifying landing spaces due to transient artifacts in the images.

At block712, a route traversal engine218of the UAV200determines a relative position of the UAV200with respect to the one or more landing spaces. In some embodiments, the route traversal engine218may compensate for uncertainty near the edges of the pixels labeled as landing spaces by computing a centroid of the group of labeled pixels, and using the centroid as the location of each landing space. The UAV200can then use the heading, pitch, yaw, and/or altitude of the UAV200to determine the relative position of the UAV200with respect to each landing space. In some embodiments, the route traversal engine218may use an apparent size of the one or more landing spaces, apparent distances between the one or more landing spaces, or other characteristics of the one or more landing spaces and/or their determined positions in order to estimate an altitude of the UAV200.

At block714, the route traversal engine218transmits signals to one or more propulsion devices210of the UAV200based on the determined relative position of the UAV200with respect to the one or more landing spaces. Once the relative position has been determined, the route traversal engine218can use the information for various tasks.

For example, in some embodiments, the route traversal engine218is configured to cause the UAV200to hover in a fixed position while performing various pre-mission calibration and safety checks. Accordingly, the route traversal engine218may use the determined relative position as a reference for a stationary hover, and may generate signals to the one or more propulsion devices210to maintain a fixed relative position with respect to the one or more landing spaces. Advantageously, this allows the UAV200to maintain its stationary hover regardless of whether a GNSS system is functional at the time.

As another example, in some embodiments, the route traversal engine218can use the identified unoccupied landing spaces to choose a landing space at which to land, and the route traversal engine218can plan a navigation path to the determined position of the selected unoccupied landing space, and can then transmit signals to the one or more propulsion devices210to traverse the navigation path to the selected unoccupied landing space. This allows the UAV200to safely choose and land at a landing space in a nest that may be concurrently used by other UAVs200without having to reserve a landing space prior to planning the mission, and without having to communicate with any other UAVs200or fleet management systems.

The method700then proceeds to an end block and terminates. ThoughFIG.7illustrates the method700as ending here for the sake of clarity, in some embodiments, the method700may continue to process images, detect landing spaces, and transmit signals to the propulsion devices210based thereon for a remainder of the duration of a flight cycle of the UAV200.

In the preceding description, numerous specific details are set forth to provide a thorough understanding of various embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

The order in which some or all of the blocks appear in each method flowchart should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that actions associated with some of the blocks may be executed in a variety of orders not illustrated, or even in parallel.

The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machine-executable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC”) or otherwise.

The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.