Techniques for target detection

Systems and methods are provided herein for detecting a marker (e.g., a marker that identifies a delivery location) utilizing an image captured by a camera of an unmanned aerial vehicle. A method may include obtaining marker information associated with a marker, the marker comprising a repetitive visual pattern, the marker being associated with delivery of an item by an unmanned aerial vehicle. Optical pattern information may be obtained that indicates a moiré pattern associated with the marker and the one or more cameras of the unmanned aerial vehicle. Image capture information that is associated with an image comprising the marker may be received. The marker may be detected in the image based at least in part on the image capture information and the moiré pattern associated with the marker.

BACKGROUND

More and more users are turning to network-based resources, such as electronic marketplaces, to purchase items (e.g., goods and/or services). Typically, a user (e.g., a customer) may operate a computing device to access a network-based resource and request information about an item. The network-based resource may provide the information and information about an available delivery method. In turn, the user may purchase the item from the network-based resource and specify a delivery location. One delivery option being considered includes delivering items to the delivery location by an unmanned aerial vehicle (UAV). In the future, once the item is loaded on the UAV, the UAV may be configured to navigate to a particular area (e.g., relatively near the delivery location) in order to identify a marker using one or more cameras. Upon detection of the marker, the UAV may navigate to the marker and place the item on or near the marker to complete the delivery. Utilizing UAVs to deliver items to users of an electronic marketplace presents various challenges. For example, the UAV may generally operate at relatively high altitudes, making marker detection difficult. This is especially true in inclement weather. Poor marker detection can cause delivery failure or delay resulting in a negative impact to the electronic marketplace provider such as loss of revenue or good will.

DETAILED DESCRIPTION

Techniques described herein are directed to marker detection utilizing a marker management engine. In at least one embodiment, a user may navigate to a website of an electronic marketplace. In some examples, the electronic marketplace may be managed by one or more service provider computers (e.g., servers) that host electronic content in the form of an electronic catalog. Users of the electronic marketplace may access the electronic catalog to view, review, discuss, order, and/or purchase items (e.g., physical items that may be stored in a warehouse or other location or electronic items that may be stored on a server and served to electronic devices) from the electronic marketplace. In some examples, the marker management engine may receive order information identifying one or more items to be delivered to the user (e.g., a gallon of milk, a book, an article of clothing, etc.). The order information may further include UAV identification information specifying a particular UAV with which the item will be delivered. The UAV may be associated with a set of attributes (e.g., an identifier, system attributes corresponding to propulsion and landing systems, camera system specifications corresponding to one or more cameras attached to the UAV, etc.). In some examples, a UAV may be assigned to the delivery (e.g., a next available UAV, a UAV that services a particular region, etc.) and attributes of the UAV (e.g., camera system specifications) may be retrieved. The user may be sent electronic marker information for the delivery that specifies a design for the marker. By way of example, the marker information may provide the user with an image (e.g., a marker including a repetitive pattern) that may be printed or otherwise displayed. In other examples, the electronic marketplace provider may provide (e.g., ship) a marker (e.g., a cloth, sheet of plastic, etc.) to the user for item delivery. In still further examples, an electronic display device (e.g., a light-emitting diode (LED) screen, a liquid crystal display (LCD), a projector, or the like) may be utilized to depict the marker information utilizing a surface (e.g., the LED screen, the LCD screen, the ground, a landing pad, etc.).

In at least one embodiment, the user may place the marker (or otherwise display the marker) at a location (e.g., the user's front yard, back yard, rooftop, driveway, shipment area, etc.) in order to mark the location at which the delivery should take place. While in flight, the UAV may detect the marker utilizing a camera attached to the UAV. In at least one example, the marker management engine may be configured to specifically detect objects in the camera image that produce a moiré pattern. A “moiré pattern,” as used herein refers to a visual distortion of an image that occurs when an object in an image exceeds (e.g., by a threshold amount) a sensor resolution of the camera that is capturing the image. In some examples, a moiré pattern produces a visual distortion (e.g., wavy lines), a color distortion, a magnification of a portion of the image, an animation, or the like. Examples of such moiré patterns are discussed further below in connection withFIGS. 3A, 3B, 4A, and 4B. Upon detection, the marker management engine may instruct the UAV to navigate to the location of the detected marker in order to complete the delivery. By utilizing moiré patterns, the UAV may be configured to detect markers from greater distances than otherwise would have been possible, as moiré patterns may by visible from greater distances than other undistorted details of an image. For example, a moiré pattern may cause one or more images of a marker to be magnified and/or the contrast of the image to be altered so that a marker may be more easily detected than if the moiré pattern was not utilized. Additionally, moiré patterns may be more easily detected despite various weather conditions such as rain, sleet, hail, snow, or the like. Still further, the repetitive details of the marker design used to produce the moiré pattern, may enable the marker to be identified utilizing less than the whole marker (e.g., if portions of a marker were blocked by a tree).

In at least one example, the marker management engine may be utilized to generate various markers that are specifically designed to produce a moiré pattern. In some examples, attributes of the UAV (e.g., camera system specifications) may be utilized by the marker management engine to generate a particular marker that is optimized to produce a moiré pattern given the particular camera(s) utilized by the UAV. For example, the marker design may include details that, when viewed by a particular camera sensor produce an image that includes a moiré pattern. In some examples, the marker may be associated with a particular location (e.g., the user's home and/or office), a particular user, a particular order and/or item, or the like.

In at least one embodiment, the marker management engine may generate one or more filter images that, when superimposed over a specially designed marker, produce a moiré pattern. These filter images may be unique to a user, a delivery location, an order and/or item, or the like. The marker management engine may utilize a filter image to detect a marker by superimposing the filter image over a still image or live video feed captured by a camera system of the UAV, for example. In at least one example, the marker management engine may store a moiré pattern. While the UAV is in flight (e.g., travelling horizontally, hovering above the ground, etc.) the marker management engine may analyze the image(s) captured by the UAV's camera(s) and superimpose the filter image over the captured image(s). When the marker management engine identifying a moiré pattern that matches the stored moiré pattern associated with the user, the marker management engine may instruct the UAV to navigate toward the marker (e.g., for delivery).

In accordance with at least one embodiment, the marker management engine may additionally, or alternatively, utilize local tone mapping techniques to detect the location of a marker. “Local tone mapping,” may include image processing techniques that may be used to map one set of colors of a first range to a second set of colors of a second range. Accordingly, local tone mapping techniques maybe used to adjust the exposure corresponding to various portions of an image. In some cases, utilizing tone mapping techniques may maximum the contrast of an image and/or may result in emphasizing as many image details as possible. Local tone mapping may utilize a non-linear function to change at least some of the pixels of an image according to features extracted from surrounding pixels. In other words, the effect of the algorithm changes in each pixel according to local features of the image. By way of example, the marker management engine may analyze an image (e.g., an aerial picture of a yard) to identify portions of the image that are overexposed (e.g., bright) or underexposed (e.g., dark). Utilizing local tone mapping techniques, various pixels relatively near an overexposed portion of the image may be analyzed. The pixels of the overexposed region may be remapped according to a local tone mapping algorithm to map the overexposed pixel values to a different range. Accordingly, the overexposed portion may be darkened by the mapping such that the details within the overexposed portion of the image may be more visible. Similarly, an underexposed portion of an image may be analyzed and the pixels relatively near the underexposed portion may be utilized to map the pixels within the underexposed portion to different pixel values. In this example, the underexposed portion of the image may be brightened to allow for details within the underexposed portion of the image to become more visible. Accordingly, a marker that is placed in a location that falls in the shadow of a tree or building, may be more easily detected as the shadowed portion of an image can be brightened to allow for greater visibility. It should be appreciated that the local tone mapping techniques discussed herein may be used separately from, or in combination with, the moiré pattern detection techniques discussed herein.

In some examples, the marker management engine may utilize images from multiple cameras from which to draw portions of the image that provide the best exposure. In other words, the marker management engine may select a portion of an image because the portion provides a highest number of visible details as compared to other images taken by other cameras of the UAV. Once portions are selected that provide the best exposure, the selected portions may be combined to produce a new image or used to replace one or more sub-optimal (e.g., underexposed, overexposed) portions of a pre-existing image. Utilizing such techniques enables the marker management engine to optimize the exposure of an image, making detection of the marker more likely.

The techniques disclosed herein provide, at least, an efficient method of detecting a marker from one or more images. By utilizing the techniques included herein, the marker may be more easily identified from further distances than detection could have occurred otherwise and/or despite weather conditions (e.g., snow, rain, hail, etc.) that otherwise might make marker detection impossible or difficult. The techniques described above and below may be implemented in a number of ways and in a number of contexts. Several example implementations and contexts are provided with reference to the following figures, as described below in more detail. However, the following implementations and contexts are but a few of many.

FIG. 1illustrates an example operational environment100for delivering an item by UAV utilizing a marker management engine102, in accordance with at least one embodiment. These techniques are described in more detail below in connection with at leastFIGS. 2-10. In the illustrated environment ofFIG. 1, the UAV104may carry a package106for delivery to a location108identified by marker110. The UAV104may be deployed to the location108associated with a user to deliver the package106. In the example operational environment100, the location108may be relatively near a dwelling112of the user. However, the embodiments described herein are not limited as such and may similarly apply to other types of locations and/or other associations to the user. The UAV104may rely on image data from camera system114to detect the marker110and deliver the package to the location108. The camera system114may include one or more cameras affixed to a suitable portion of the UAV104.

In accordance with at least one embodiment, the user may have operated a computing device to access a network-based resource to order one or more items (e.g., a gallon of milk, a book, a video game, etc.). As part of the ordering process, or at another suitable time, the marker management engine102may generate marker information (e.g., a size, one or more colors, one or more moiré patterns, etc.) corresponding to the marker110. By way of example, the marker110may include one or more lines that are repetitive in nature. In some examples, the marker110may include text (e.g., alphanumeric characters, the user's name, the address, coordinates, etc.), one or more images (e.g., a silhouette of an animal, an image associated with the user such as a family portrait, etc.), or any suitable detail. In at least one example, the marker110may contain a series of lines such as those depicted inFIG. 1. The marker110may be specifically generated such that the marker110, when viewed using the camera system114, will produce a moiré pattern that matches stored marker information116associated with the user, the order, the item, or the location. Upon generating the marker information, the marker management engine102may store the marker information as stored marker information116associated with the user, the item, the order, the location108, or the like. In the example depicted inFIG. 1, the marker management engine102may cause the stored marker information116to be associated with the location108. The association, in some examples, may be stored in a user profile of the user. The marker management engine102may cause the stored marker information116to be electronically provided to the user (e.g., provide a marker for printout and/or display) so that the user may display the marker110at the location108. Alternatively, the marker management engine102may execute instructions that cause a physical marker (e.g., a 36-inch by 36-inch sheet of plastic) to be created and shipped to the user. As part of the delivery process, the item(s) may be packaged at a facility and loaded on the UAV104as depicted. The UAV104may be remotely controlled or autonomously operated to fly the package106from the facility to an area that is relatively near the location108(e.g., 2 miles, 5 miles, 400 feet, etc.).

Upon determining that the UAV is within the area near the location108, the marker management engine102(e.g., operating as a software module of the UAV or as a software module of a centralized server) may obtain the stored marker information116. One or more images (e.g., still images and/or video) may be received, or otherwise obtained, by the marker management engine102. By way of example, the image118(e.g., as depicted or as part of a larger image) may be received by the marker management engine102from the camera system114of the UAV. It should be appreciated that, in this example, the marker110, the stored marker information116, and the image118are each intended to depict a same marker.

In accordance with at least one embodiment, the marker management engine102may analyze the image118in order to identify moiré pattern120. The image118is merely an example and it should be appreciated that the image118may include more details than those depicted inFIG. 1. For example, the image118may be an aerial view of the user's property including some portions of the dwelling112, a tree122, a front yard124, a kiddie pool126, a bush128, or the like, as depicted later inFIG. 5. An image including the details depicted inFIG. 1may be analyzed by the marker management engine102in order to identify portions of the image that depict a moiré pattern. By way of example, the marker management engine102may analyze the aerial image502depicted inFIG. 5to determine that the tree122, the front yard124, the kiddie pool126, the bush128, and other portions of the image do not contain moiré patterns. Similarly, the marker management engine102may analyze the image to determine that a portion (e.g., the portion depicting the marker110) does include a moiré pattern. In some examples, the marker management engine102may crop the image to produce the image118. In other examples, the marker management engine102may merely ignore details in the aerial image that do not depict a moiré pattern.

Upon identifying that the image118includes a moiré pattern, the marker management engine102may compare the image118to the stored marker information116. The comparison may result in the marker management engine102determining that the image118does not match the stored marker information116within some confidence threshold (e.g., 90% or more certain, 55% or more certain, 75%-80% certain, etc.). In this case, the marker management engine102may continue to analyze additional images captured by the UAV. In some cases, the marker management engine102may cause the UAV to travel to a different location in order to continue searching for the marker110.

In at least on embodiment, the marker management engine102may determine, through the comparison of the image118to the stored marker information116, that the image118matches (e.g., is similar within a confidence threshold of, for example, 85% or greater, etc.) the stored marker information116. In such cases, the marker management engine102may cause the UAV to navigate to the location108in order to deliver the package106.

FIG. 2illustrates an example unmanned aerial vehicle200configured to detect a marker utilizing a marker management engine (e.g., the marker management engine102ofFIG. 1), in accordance with at least one embodiment. InFIG. 2, an example UAV200configured to deliver an item (and/or pickup an item) is illustrated. The UAV200may be designed in accordance with commercial aviation standards and may include multiple redundancies to ensure reliability. In particular, the UAV200may include a plurality of systems or subsystems operating under the control of, or at least partly under the control of, a management component202.

The management component202may be configured to mechanically and/or electronically manage and/or control various operations of other components of the UAV200. For example, the management component202may include various sensing, activating, and monitoring mechanisms to manage and control the various operations. For instance, the management component202may include or interface with an onboard computing system204hosting a management module for autonomously or semi-autonomously controlling and managing various operations of the UAV200and, in some examples, for enabling remote control by a pilot. The various operations may also include managing other components of the UAV200, such as a camera system203, a propulsion system218to facilitate flights, a payload holding mechanism212to facilitate holding a payload (e.g., a package), and/or a payload releasing mechanism214to facilitate release and delivery of the payload. In at least one embodiment, the management component202provides the functionality described above in connection with the marker management engine102ofFIG. 1.

Portions of the management component202, including mechanical and/or electronic control mechanisms may be housed under the top cover250or distributed within other components such as the payload holding mechanism212and the payload releasing mechanism214. In a further example, components remote from the UAV200may be deployed and may be in communication with the management component202to direct some or all of the operations of the management component202. These remote components may also be referred to as a management component.

In an example, the management component202may include a power supply and assemblies (e.g., rechargeable battery, liquid fuel, and other power supplies) (not shown), one or more communications links and antennas (e.g., modem, radio, network, cellular, satellite, and other links for receiving and/or transmitting information) (not shown), one or more navigation devices and antennas (e.g., global positioning system (GPS), inertial navigation system (INS), range finder, Radio Detection And Ranging (RADAR), and other systems to aid in navigating the UAV200and detecting objects) (not shown), and radio-frequency identification (RFID) capability (not shown).

The UAV200may also include the onboard computing system204. In an example, the onboard computing system204may be integrated with the management component202. In another example, the onboard computing system204may be separate from but may interface with the management component202. The onboard computing system204may be configured to provide an electronic control of various operations of the UAV200, including the ones provided by the marker management engine102ofFIG. 1. In an example, the onboard computing system204may also process sensed data by one or more other components of the UAV, such as the camera system203. In a further example, the onboard computing system204may also electronically control components of the payload holding mechanism212and/or the payload releasing mechanism214. In another example, the onboard computing system204may also electronically control components of the UAV200such as a plurality of propulsion devices, a few of which,230(A)-230(F) are included inFIG. 2.

As illustrated inFIG. 2, the onboard computing system204may be housed within the top cover250and may include a number of components, such as a computer206, a storage device208, the camera system203, and an interface210. The computer206may host the management module configured to provide management operations of the flight and/or other portions of a mission (e.g., a delivery of an item, pick-up of an item, marker detection, etc.) of the UAV200. For example, the data management module may analyze one or more images captured by the camera system203, determine an appropriate delivery surface, determine a distance by which to lower a payload, a speed of lowering the payload, direct the propulsion system to position the UAV200according to this data, activate a release of a package from the payload holding mechanism212, activate a release of a cable, and/or activate other functions of the mission.

In some embodiments, the management component202may perform any suitable operations discussed in connection with the marker management engine102ofFIG. 1. For example, the management component202may receive order information associated with a request for delivery and/or pickup of an item. The management component202may determine and/or assign a particular UAV that will be utilized for delivery/pick-up of the item. The management component202may generate and/or obtain marker information associated with the marker that will be utilized for delivery of the item. The management component202may perform various operations to compare an image captured by the camera system203to stored marker information (e.g., stored in storage device208). The management component202may, upon determining that a portion of an image captured by the camera system203matches (e.g., within a threshold value) stored marker information, may execute instructions that navigate the UAV to a particular location (e.g., to land at/near the identified marker). In at least one example, the management component202may interact with the storage device208. The storage device208may represent one or more storage media, such as a volatile or non-volatile semiconductor, magnetic, or optical storage media.

In an example, the storage device208may be configured to store any operational data of the UAV200, marker information associated with the marker to be used for delivery/pick-up, generated or received data associated with the delivery surface, and/or received camera data (e.g., one or more still images, a video feed, etc.). The operational data may include the distance by which the payload may be lowered and the lowering speed. The marker information may indicate a moiré pattern for the marker when viewed by the camera system203. The marker information may additionally, or alternatively include one or more filters to be utilized during image analysis. In addition, the storage device208may store a set of rules associated with lowering and releasing the payload. This set of rules may specify parameters to determine, where, when, and/or how to deliver the payload such that a likelihood of damaging the payload (or content thereof) and/or interference with the UAV200may be reduced.

The interface210may represent an interface for exchanging data as part of managing and/or controlling some of the operations of the UAV200. In an example, the interface210may be configured to facilitate data exchanges with the management component202, other components of the UAV200, and/or other components remote from the UAV200. As such, the interface210may include high speed interfaces, wired and/or wireless, serial and/or parallel, to enable fast upload and download of data to and from the onboard computing system204.

As shown inFIG. 2, The camera system203may be positioned on top cover250or on the frame226of UAV200. Although not shown, one or more camera systems (e.g., the camera system203) may be mounted in different directions (e.g., downward looking cameras to identify ground objects and/or a landing zone for a payload while UAV200is in flight.

As shown inFIG. 2, the UAV200may also include the payload holding mechanism212. The payload holding mechanism212may be configured to hold or retain a payload. In some examples, the payload holding mechanism212may hold or retain the payload using friction, vacuum suction, opposing arms, magnets, holding, and/or other retaining mechanisms.

As illustrated inFIG. 2, the payload holding mechanism212may include a compartment configured to contain the payload. In another example, the payload holding mechanism212may include two opposing arms configured to apply friction to the payload. The management component202may be configured to control at least a portion of the payload holding mechanism212. For example, the management component202may electronically and/or mechanically activate the payload holding mechanism212to hold and/or release the payload. In an example, the payload may be released from the payload holding mechanism212by opening the compartment, pushing the payload, moving one or both of the opposing arms, and/or stopping an application of friction, vacuum suction, and/or magnetic force.

The UAV200may also include the payload releasing mechanism214. In an example, the payload releasing mechanism214may be integrated with the payload holding mechanism212. In another example, the payload releasing mechanism may be separate from the payload holding mechanism212. In both examples, the payload releasing mechanism214may be configured to lower, using a cable, a payload released from the payload holding mechanism212and to release the cable once the payload is lowered by a distance.

As such, the payload releasing mechanism214may include a lowering mechanism and a release mechanism. For example, the lowering mechanism may include a cable and/or an electronic or mechanical control configured to lower the cable at a controlled speed. For example, this control may include a winch, a spool, a ratchet, and/or a clamp. The cable may couple the payload with the UAV200. For example, one end of the cable may be connected, attached, or integral to the payload. Another end of the cable may be coupled to one or more components of the payload releasing mechanism214, the payload holding mechanism212, the frame of the UAV200, and/or other component(s) of the UAV200. For example, the cable may be coiled around the winch or spool or may be stowed or coiled inside the compartment (if one is used as part of the payload holding mechanism212). The cable may have a configuration selected based on the mission of the UAV200, the mass of the payload, and/or an expected environment associated with the delivery location (e.g., the potential interference).

In an example, the release mechanism may be integrated with the lowering mechanism. In another example, the release mechanism may be separate from the lowering mechanism. In both examples, the release mechanism may be configured to release the cable when the payload may have been lowered by a certain distance. Releasing the cable may include severing the cable, weakening the cable, and/or decoupling the cable from the UAV200(e.g. from the payload releasing mechanism214) without severing or weakening the cable.

To sever the cable, the release mechanism may include a sharp surface, such as a blade to, for example, cut the cable when applied thereto. To weaken the cable, the release mechanism may include a sharp head, edge, and/or point, such as a hole puncher, or a friction surface to cause a damage to the integrity of the structure of the cable. Other release mechanisms may also be used to sever or weaken the cable. An example may include a mechanism configured to apply a thermoelectric effect to the cable. For instance, a contact surface, such as one using an electrical conductor, may be configured to release heat upon application of a voltage. The contact surface may come in contact with the cable or may be integrated within different sections of the cable. Upon application of the voltage, the contact surface may sever or weaken the cable by applying heat to the cable. To decouple the cable from the UAV200, the cable may be in the first place insecurely coupled to the UAV200such that, upon an unwinding of the cable, the cable may become detached from the UAV200. For example, the cable may be coiled around the winch or spool without having any of the cable ends attached to the winch or spool or to another component of the UAV200. In another example, the cable may be coupled to a component of the UAV200through a weak link such that upon a tension generated based on the mass of the payload, the link may be broken to free the cable from the UAV200.

The release mechanism may be electronically or mechanically controlled. This control may be effected based on, for example, the distance by which the payload may have been lowered and/or based on an amount of a tension of the cable, an increase in the amount, a decrease in the amount, or a sudden or fast change in the amount. Various configurations may be used to measure the distance, the amount of tension, and the change in the amount. For example, the distance may be determined from the number of rotations of a winch or spool if one is used or based on a distance or cable length sensor. The amount of the tension and the change in the amount may be determined based on spring-based or electronic-based sensors.

Further, the release mechanism may be electronically activated based on a signal generated in response to detecting that the distance may have been traveled and/or the amount or change in the amount of tension. In another example, the release mechanism may be activated based on a mechanical configuration. For example, as the cable may be lowered, a ratchet may load a spring that may be coupled to release mechanism. Upon the load exceeding a threshold, the spring may be released, thereby activating the release mechanism. In another example, a tension of the cable may be used to hold the release mechanism away from the cable. As soon as the tension changes (e.g., the cable becomes loose indicating that the payload may be resting on the ground), the release mechanism may be activated to sever or weaken the cable.

Further, the UAV200may include a propulsion system218. In some examples, the propulsion system218may include rotary blades or otherwise be a propeller-based system. As illustrated inFIG. 2, the propulsion system218may include a plurality of propulsion devices, a few of which,230(A)-230(F), are shown in this view. Each propeller device may include one propeller, a motor, wiring, a balance system, a control mechanism, and other features to enable flight. In some examples, the propulsion system218may operate at least partially under the control of the management component202. In some examples, the propulsion system218may be configured to adjust itself without receiving instructions from the management component202. Thus, the propulsion system218may operate semi-autonomously or autonomously.

The UAV200may also include landing structure222. The landing structure222may be adequately rigid to support the UAV200and the payload. The landing structure222may include a plurality of elongated legs which may enable the UAV200to land on and take off from a variety of different surfaces. The plurality of systems, subsystems, and structures of the UAV200may be connected via frame226. The frame226may be constructed of a rigid material and be capable of receiving via different connections the variety of systems, sub-systems, and structures. For example, the landing structure222may be disposed below the frame226and, in some examples, may be formed from the same material and/or same piece of material as the frame226. The propulsion system218may be disposed radially around a perimeter of the frame226or otherwise distributed around the frame226. In some examples, the frame226may attach or be associated with one or more fixed wings.

Hence, a UAV, similar to the UAV200, may be deployed on a mission to, for example, deliver and/or retrieve a payload (e.g., an item). The UAV may autonomously or semi-autonomously complete or perform a portion of the mission. For example, coordinates of a delivery location may be provided to the UAV (e.g., as part of a process of ordering an item from an electronic marketplace provider). The UAV may hold the payload in a payload holding mechanism and fly to the delivery location. Utilizing the camera system203, the UAV may capture one or more images (e.g., still and/or video) to detect a marker. Upon detection of the marker, the UAV may navigate to the delivery location identified by the marker. Upon arrival (e.g., landing) at the location, the UAV may release the payload from the payload holding mechanism.

FIGS. 3A and 3Bare schematic diagrams illustrating an example moiré patterns. For example,FIG. 3Adepicted a moiré pattern created by image302, image304, and image306. Each image contains slightly different line widths with respect to the other images. A moiré pattern may be created by superimposing the image304over the image302and the image306as depicted inFIG. 3. The size of the moiré pattern as well as its direction of movement may depend on the relative size of the intersecting patterns. For example, at time1, the image304may begin to move downward to the position indicated at time2. As the image304is moved downward, a moiré pattern308may be created. The moiré pattern308may appear as if the bold lines are moving in the opposite direction (upward) as the image304. At the same time, superimposing the image304may produce a moiré pattern310. As the image304is moved downward, the moiré pattern310may appear as if the bold lines are moving in the same direction (downward) as the image304. Similarly, if the image304was moved in the reverse direction (e.g., from the position indicated at time2to the position indicated at time1), the moiré pattern308may appear as if the bold lines are moving downward, while at the same time the moiré pattern310may appear as if the bold lines are moving upward.

In at least one embodiment, similar line patterns may be utilized by the marker management engine102ofFIG. 1to produce a moiré pattern with respect to a marker. For example, marker information may be generated that specifies a number, width, and/or orientation (e.g., vertical, horizontal, or a combination of both) of lines to be included in a marker. While attempting to detect a moiré pattern such as the ones depicted inFIGS. 3A and 3B, the marker management engine102may superimpose a filter image similar to the image304to images captured by a UAV (e.g., the UAV ofFIG. 2). Upon superimposing the filter image, the marker management engine102may analyze the resulting effect to determine if any portion of the captured images produce a moiré pattern such as the moiré pattern308or the moiré pattern310. For example, the moiré pattern308may be stored (e.g., in a record associated with the user, the item, the order, the location, or the like). Upon detecting a moiré pattern produced from the superimposed image, the marker management engine102may determine if the produced moiré pattern matches the moiré pattern308associated with the location. If the moiré patterns match (e.g., within a threshold value such as a confidence level, a score, etc.), the marker management engine102may determine that the correct marker has been detected and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to approach the location of the detected marker. If the moiré patterns do not match, the marker management engine102may determine that the correct marker has not been detected and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to proceed with procedures for searching for a marker (e.g., continue executing a search pattern, conduct a new search pattern, continue capturing images, etc.).

FIG. 3Bdepicts another example moiré pattern produced by a base image312and a revealing image314. The moiré pattern316depicted inFIG. 3Bis an example of a “shape moiré pattern” that demonstrates moiré magnification. In the example ofFIG. 3B, the base image312includes a number of shapes (e.g., “A,” “B”, “C”) that is compressed along a vertical access. The revealing image314includes a number of horizontal lines. When the revealing image314is superimposed over the base image312, the shapes of the base image312may appear magnified as depicted by the moiré pattern316. In at least one example, as the revealing image314is moved, the magnified shapes of the moiré pattern316may appear to move in the reverse direction. The moiré pattern may be created due to the period of the shapes in the base image312being within a threshold range of the period of lines in the revealing image314.

In at least one embodiment, a similar moiré pattern may be utilized by the marker management engine102ofFIG. 1to detect a marker. For example, the marker management engine102may generate marker information that specifies the orientation of the shapes as included in the base image312. While attempting to detect the moiré pattern316, the marker management engine102may superimpose the revealing image314over images captured by a UAV. The marker management engine102may then analyze the resulting images to determine if any portion of those images produce a moiré pattern such as the moiré pattern316. As a non-limiting example, the moiré pattern316may be stored (e.g., in a record associated with the user, the item, the order, the location, or the like). Upon detecting the moiré pattern316, the marker management engine102may determine if the moiré pattern316matches the stored moiré pattern associated with the location. As described above, if the moiré patterns match, the marker management engine102may determine that the correct marker has been detected and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to approach the location of the detected marker. If the moiré patterns do not match, the marker management engine102may determine that the correct marker has not been detected and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to proceed with procedures for searching for a marker (e.g., continue executing a search pattern, conduct a new search pattern, continue capturing images, etc.).

FIGS. 4A and 4Bare schematic diagrams illustrating additional example moiré patterns (e.g. a moiré pattern406and a moiré pattern412, respectively).FIG. 4Adepicts a marker image402and a marker filter image404. The marker management engine102ofFIG. 1may utilize the marker image402(e.g., collected by the UAV200ofFIG. 2) and the marker filter image404(e.g., a stored filter image) to detect the existence of a particular marker. For example, an image including the marker image402may be received, or otherwise obtained by the marker management engine102. The marker management engine102may superimpose the marker filter image404over the marker image402. The resulting effect may include a moiré pattern406. The marker management engine102may compare the moiré pattern406to a stored moiré pattern. In at least one example, the mere existence of the moiré pattern406may be indicative of correct marker detection which may obviate the need to compare the moiré pattern to a stored moiré pattern. Upon detecting the moiré pattern406, and/or comparing the moiré pattern406to the stored moiré pattern, the marker management engine102may determine that the correct marker (e.g., the marker being searched for) has been identified in the received/obtained image and may perform one or more remedial actions. Such remedial actions include instructing the UAV (or causing the UAV to be instructed) to approach the location of the detected marker. If the moiré pattern406does not appear, the marker management engine102may determine that the correct marker has not been detected and may perform one or more remedial actions. Such remedial actions include instructing the UAV (or causing the UAV to be instructed) to proceed with procedures for searching for a marker (e.g., continue executing a search pattern, conduct a new search pattern, continue capturing images, etc.).

FIG. 4Bdepicts a marker image408and a marker filter image410. In a similar manner as described above, the marker management engine102ofFIG. 1may utilize the marker image408(e.g., collected by the UAV200ofFIG. 2) and the marker filter image410to detect the existence of a particular marker. For example, an image including the marker image408may be received, or otherwise obtained by the marker management engine102. The marker management engine102may superimpose the marker filter image410over the marker image408(or otherwise combine the images). The resulting effect may include a moiré pattern412. The marker management engine102may detect the moiré pattern412and compare it to a stored moiré pattern. In some cases, the mere existence of the moiré pattern412may be indicative that the correct marker has been detected. Upon detecting the moiré pattern412, the marker management engine102may determine that the correct marker (e.g., the marker being searched for) has been identified in the received/obtained image and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to approach the location of the detected marker. If the moiré pattern412does not appear, the marker management engine102may determine that the correct marker has not been detected and may perform one or more remedial actions such as instructing the UAV (or causing the UAV to be instructed) to proceed with procedures for searching for a marker (e.g., continue executing a search pattern, conduct a new search pattern, continue capturing images, etc.).

FIG. 5illustrates an example operational environment500for detecting a marker utilizing local tone mapping techniques, in accordance with at least one embodiment. Contextually, there may be some situations that a dynamic range of an image may be outside of the capability of a camera system (e.g., the camera system203ofFIG. 2). For example, a marker may be made of highly reflective materials, a portion of the image may contain a reflective surface (e.g., a water puddle), or the like. In some cases, an image captured by a UAV camera system may inadvertently overexpose a portion of the image (e.g., a portion containing the marker110). In at least one example, the camera system may underexpose other portions of the image, such as areas that are darker due to shadows of other objects, for example. In either case, local tone mapping techniques may be employed by the marker management engine102in order to provide an optimal contrast of the image.

For example, consider the aerial image502depicted inFIG. 5. For purposes of explanation, the aerial image502depicts the operational environment100ofFIG. 1as viewed from above. Within the aerial image502the dwelling112, the tree122, the kiddie pool126, the bush128, and a fence504can be seen. Shadow area506may result from a shadow cast by the tree122. Shadow area508may be cast by the bush128. Shadow area510may be cast by the kiddie pool126. Shadow area512may be cast by the dwelling112. Shadow area514may be cast by the fence504. Additionally, reflective surface516may be depicted inFIG. 5resulting from, for example, sun light reflecting off of water contained in the kiddie pool126. Similarly, a reflective surface518is depicted resulting from, for example, a patch of water on a roadway. As a non-limiting example, the marker110may be located as depicted inFIG. 5within the shadow area506.

In accordance with at least one embodiment, multiple cameras of a camera system (e.g., the camera system203ofFIG. 2) operating as part of a UAV (e.g., the UAV200ofFIG. 2) may be utilized for marker detection. In at least one example, the marker management engine102ofFIG. 1may utilize such cameras to detect a marker within the aerial image502. By way of example, each camera may be configured to capture images utilizing different camera settings. For example, camera A may capture an image using an exposure E1 while camera B may capture an image using an exposure E1−n, where n is a multiplier. “Exposure,” in this context, is determined by three camera settings: aperture, ISO, and shutter speed. “Aperture” refers to an opening of a lens's diaphragm through which light passes. “ISO” refers to the level of sensitivity of the camera sensor to available light. “Shutter speed,” refers to a length of time that a camera shutter is open to expose light to the camera sensor.

In at least one embodiment, camera A may be configured to utilize exposure E1 in order to capture the aerial image502. Based on the exposure E1, the image captured by camera A may generally depict more details than the image captured by camera B with respect to the areas illustrated by the reflective surface516and the reflective surface518ofFIG. 5. Based on the exposure E1×n, the image captured by camera B may generally depict more details than the image captured by camera A with respect to the shadow areas depicted inFIG. 5. In at least one embodiment, the images captured by camera A and camera B may be combined for marker searching purposes.

By way of example, the marker management engine102may generate a histogram of each image to identify areas of underexposure and overexposure (e.g., the areas illustrated by the reflective surface and the shadow areas ofFIG. 5). As a non-limiting example, the marker management engine102may analyze the image captured by camera A to determine that the shadow area506is underexposed, that is, that the details within the shadow area506are distorted or difficult to make out due to that portion of the image being too dark. The marker management engine102may analyze the image captured by camera B with respect to the shadow area506. In at least one example, the marker management engine102may determine that the shadow area506includes more details (e.g., is not underexposed, is brighter, etc.) in the image captured by camera B than in the image captured by camera A. In some examples, the marker management engine102may utilize a portion of the image captured by camera B (e.g., the shadow area506) to replace the same area of the image captured by camera A. Similarly, the marker management engine102may determine that image captured by camera A of the area illustrated by the reflective surface516includes more details (e.g., is not overexposed) with respect to the same area as captured by camera B. Accordingly, the marker management engine102may utilize the portion of the image capture by camera A to depict the area illustrated by the reflective surface516. Each portion of the images that are overexposed/underexposed, as specified by the histogram, may be identified and the marker management engine102may determine which camera provides the best exposure (e.g., which image depicts the most detail) for a given portion of the image. In some examples, the marker management engine may generate a new image that includes some portions of the images captured by camera A and camera B. In the example depicted inFIG. 5, the marker110is located within the shadow area506. By utilizing the images produced by the different settings corresponding to camera A and camera B, the marker management engine102may adjust the exposure of the shadow area506(as well as the remaining shadow areas and reflective surfaces depicted) in the aerial image502so that the image, as a whole, is provided at an optimal exposure, making the marker110easier to identify.

In accordance with at least one embodiment, the marker110may include reflective material in either/or the pattern (e.g., the lines) of the marker110or the background (e.g., the spaces) of the marker110. The marker management engine102may analyze the marker information for the marker110to determine that the marker110includes at least one reflective portion. In at least one example, such marker information may be utilized by the marker management engine102as a tool for detecting the marker110in an image captured by camera A and/or camera B. For example, the marker management engine102may execute instructions that identify one or more overexposed portions of the image. In at least one example, the area defined by the marker110may be identified as an overexposed portion of the image. Once identified, the marker management engine102may utilize one or more local tone mapping algorithms to adjust the exposure of the image (e.g., to be similar to pixel values of relatively nearby pixels). A local tone mapping algorithm may include a non-linear function that changes each pixel according to features extracted from surrounding pixels. Accordingly, the marker management engine102may utilize a local tone mapping algorithm to modify the overexposed region corresponding to the area defined by the marker110such that the area is no longer overexposed. In some examples, utilizing the local tone mapping algorithm may reduce the exposure of the area by mapping previous values for the pixels within the area illustrated by the reflective surface518to new values (e.g., values that do not result in overexposure of the reflective surface518). In this manner, the marker110becomes easier to identify since the marker110may be depicted at a reasonable exposure that allows for the details of the marker110to be visible.

In another example, the marker110may be placed at a location within the shadow area506. In this example, the marker management engine102, utilizing one or more local tone mapping algorithms, may brighten the corresponding portion of the image. In at least one example, the marker management engine102may incrementally brighten the shadow area506until it can detect at least one more object within the shadow area506. In other examples, the marker management engine102may produce a set of images utilizing one or more local tone mapping algorithms, each image of the set of images individually depicting different exposures of the shadow area506. The marker management engine102may then select an optimal image, or in other words, an image for which a greatest number of objects may be observed. In still further examples, the marker management engine102may incrementally bright the shadow area506until the marker110is detected. Upon detecting the marker110, the marker management engine may cease executing instructions related to local tone mapping.

FIG. 6illustrates a flow of an example process600for detecting a marker utilizing the marker management engine102ofFIG. 1, in accordance with at least one embodiment. Although the example below illustrates a use case in which the computing system is a component of a UAV (e.g., the UAV200ofFIG. 2), it should be appreciated that the process600may equally apply in situations where the functionality is being provided by a remote system (e.g., a server computer remote to the UAV200). The process600may begin at block602, where the marker management engine102(e.g., a computing system of the UAV200ofFIG. 2) may receive instructions to deliver an item to a delivery location utilizing the unmanned aerial vehicle as a mode of transportation, the unmanned aerial vehicle comprising a camera. The instructions may include item identification, delivery options (e.g., UAV delivery, two-day delivery, same-day delivery, or any suitable combination of the above), delivery preferences (e.g., time of day for preferred delivery, packaging options, etc.), or the like.

At block604, the computing system (e.g., the marker management engine102) may obtain physical marker information associated with a physical marker (e.g., the marker110ofFIG. 1). In some examples, the physical marker may comprise a repetitive visual pattern (e.g., vertical/horizontal lines, wavy lines, checkered pattern, etc.). In some cases, the physical marker may be a physical object that bears a marker design such as a piece of paper, a sheet of plastic, or the like. In other examples, the physical marker may include a display screen or projected image that depicts the marker design.

At block606, the computing system (e.g., the marker management engine102) may obtain optical pattern information associated with the physical marker and the camera. In at least one example, the optical pattern information may indicate a moiré pattern associated with the physical marker and the camera. The optical pattern information may be a pre-generated image and/or pre-generated information specifying one or attributes of a moiré pattern produced by a combination of a particular marker design and a particular camera. In at least one example, the moiré pattern is produced based at least in part on a spatial frequency associated with an image of the physical marker being within a threshold frequency range to a resolution associated with the camera.

At block608, the computing system (e.g., the marker management engine102) may receive, from the unmanned aerial vehicle, image capture information associated with an image captured by the camera comprising the physical marker. Image capture information, in some cases, may specify one or more details of the image (e.g., the exposure of the image, the data needed to electronically reproduce the image, etc.).

At block610, the computing system (e.g., the marker management engine102) may compare the image capture information from the camera to the optical pattern information.

At block612, the computing system (e.g., the marker management engine102) may determine that the physical marker exists in the image based at least in part on the comparison. Any suitable number and type of image recognition techniques may be employed to determine whether the physical marker exists in the image.

At block614, in response to determining that the physical marker exists in the image, the computing system (e.g., the marker management engine102) may instruct the unmanned aerial vehicle to approach the physical marker for delivery. For example, the marker management engine102may communicate information to a propulsion system of the UAV200, or to a module responsible for controlling propulsion to cause the UAV to calculate a heading for the UAV in order to bring the UAV to the delivery location corresponding to the physical marker.

FIG. 7illustrates a flow of an additional example process700for detecting a marker utilizing the marker management engine102ofFIG. 1, in accordance with at least one embodiment. Although the example below illustrates a use case in which the computing system is a component of a UAV (e.g., the UAV200ofFIG. 2), it should be appreciated that the process600may equally apply in situations where the functionality is being provided by a remote system (e.g., a server computer remote to the UAV200). The process700may begin at block702, where the marker management engine102(e.g., a computing system of the UAV200ofFIG. 2) may obtain marker information associated with a marker, the marker comprising a repetitive visual pattern, the marker being associated with delivery of an item by an unmanned aerial vehicle.

At block704, the computing system (e.g., the marker management engine102) may obtain optical pattern information indicating a moiré pattern associated with the marker and the one or more cameras of the unmanned aerial vehicle.

At block706, the computing system (e.g., the marker management engine102) may receive, from the one or more camera systems of the unmanned aerial vehicle, image capture information associated with an image comprising the marker.

At block708, the computing system (e.g., the marker management engine102) may detect that the marker exists in the image based at least in part on the image capture information and the moiré pattern associated with the marker.

FIG. 8illustrates a flow of still one further example process800for detecting a marker utilizing the marker management engine102ofFIG. 1, in accordance with at least one embodiment. Although the example below illustrates a use case in which the computing system is a component of a UAV (e.g., the UAV200ofFIG. 2), it should be appreciated that the process600may equally apply in situations where the functionality is being provided by a remote system (e.g., a server computer remote to the UAV200). The process800may begin at block802, where the marker management engine102(e.g., a computing system of the UAV200ofFIG. 2) may receive instructions associated with delivery of an item, the delivery to be carried out utilizing an unmanned aerial vehicle (UAV) as a mode of transportation, the UAV comprising one or more cameras.

At block804, the computing system (e.g., the marker management engine102) may generate an image of a delivery location marker for the delivery based at least in part with an attribute of the one or more cameras.

At block806, the computing system (e.g., the marker management engine102) may obtain optical pattern information indicating a moiré pattern associated with the delivery location marker and the one or more cameras.

At block808, the computing system (e.g., the marker management engine102) may detect the delivery location marker in the image based at least in part on the image capture information and the optical pattern information.

FIG. 9illustrated an example architecture900for using a UAV904(e.g., the UAV104ofFIG. 1) for a particular mission (e.g., delivering one or more items to a location, picking up one or more items from a delivery location, etc.).FIG. 9illustrates a package delivery mission, although other missions are likewise possible. The architecture900may include one or more service provider computers902. The service provider computers902may support an electronic marketplace (not shown) and interface with purchase and delivery services of the electronic marketplace. In this manner, the service provider computers902may coordinate delivery of items via UAVs, such as UAV904, to customers of the electronic marketplace. In some examples, the service provider computers902may be a stand-alone service operated on its own or in connection with an electronic marketplace. In either example, the service provider computers902may be in communication with the UAV904via one or more network(s)906(hereinafter, “the network906”). The network906may include any one or a combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, radio networks, and other private and/or public networks. Thus, the service provider computers902may be configured to provide back-end control of the UAV904prior to, during, and/or after completion of its delivery plan. In some examples, the UAV904may be configured to accomplish its delivery plan (e.g., detect a marker, deliver an item to a location of the marker, etc.) with little to no communication with the service provider computers902.

User devices908(1)-908(N) (hereinafter, “the user device908”) may also be in communication with the service provider computers902and the UAV904via the network906. The user device908may be operable by one or more users910(hereinafter, “the users910”) to access the service provider computers902(or an electronic marketplace) and the UAV904via the network906. The user device908may be any suitable device capable of communicating with the network906. For example, the user device908may be any suitable computing device such as, but not limited to, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a thin-client device, a tablet PC, a desktop computer, a set-top box, or other computing device. In some examples, the user device908may be in communication with the service provider computers902via one or more web servers constituting an electronic marketplace (not shown) connected to the network906and associated with the service provider computers902.

Turning now to the details of the UAV904, the UAV904may include an onboard computer912including at least one memory914and one or more processing units (or processor(s))916. The processor(s)916may be implemented as appropriate in hardware, computer-executable instructions, software, firmware, or combinations thereof. Computer-executable instruction, software or firmware implementations of the processor(s)916may include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described. The memory914may include more than one memory and may be distributed throughout the onboard computer912. The memory914may store program instructions (e.g. UAV management module918that may implement functionalities of the marker management engine102ofFIG. 1and/or the management component202ofFIG. 2) that are loadable and executable on the processor(s)916, as well as data generated during the execution of these programs. Depending on the configuration and type of memory including the UAV management module918, the memory914may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, or other memory). The memory914may also include additional removable storage and/or non-removable storage including, but not limited to, magnetic storage, optical discs, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory914may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM. Turning to the contents of the memory914in more detail, the memory914may include an operating system920and one or more application programs, modules or services for implementing the features disclosed herein including at least the UAV management module918.

In some examples, the onboard computer912may also include additional storage922, which may include removable storage and/or non-removable storage. The additional storage922may include, but is not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices.

The memory914and the additional storage922, both removable and non-removable, are examples of computer-readable storage media. For example, computer-readable storage media may include volatile or non-volatile, removable, or non-removable media implemented in any suitable method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. As used herein, modules may refer to programming modules executed by computing systems (e.g., processors) that are part of the onboard computer912. The modules of the onboard computer912may include one or more components. The onboard computer912may also include input/output (I/O) device(s)924and/or ports, such as for enabling connection with a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, speakers, a printer, or other I/O device. The I/O device(s)924may enable communication with the other systems of the UAV904(e.g., other parts of the control system, power system, communication system, navigation system, propulsion system, and the retaining system).

The onboard computer912may also include data store926. The data store926may include one or more databases, data structures, or the like for storing and/or retaining information associated with the UAV904. In some examples, the data store926may include databases, such as customer information database928, landing zone database930, and or marker information database931. Customer information database928may be configured to store any suitable customer information that may be used by the UAV904in implementing and/or affecting its delivery plan. For example, the customer information database928may include profile characteristics and/or order information for the user(s)910. The profile characteristics may include a shipping address, delivery preferences, or the like. The order information may indicate one or more items that were ordered by the user(s)910utilizing an electronic marketplace provided by the service provider computers902. The order information may include delivery attributes that indicate that the delivery is to be achieved utilizing a particular UAV (e.g., the UAV904) or generally utilizing a UAV (e.g., a next available UAV). The landing zone database930may store suitable landing zones or drop-off zones associated with a particular user. The landing zone database930may include GPS coordinates and/or images of landing zones associated with a particular user. The marker information database931may include marker information (e.g., the stored marker information116ofFIG. 1) that provide details regarding a particular marker that is to be utilized for delivery. In at least one embodiment, the marker information database931may store a set of pre-defined markers. A particular marker of the set may be selected for the order/item/delivery based on, for example, a camera associated with the delivering UAV.

Turning now to the details of the user device908. The user device908may be used by the user(s)910for interacting with the service provider computers902. The user device908may therefore include a memory, a processor, a user-interface, a web-service application, and any other suitable feature to enable communication with the features of architecture900. The web service application may be in the form of a web browser, an application programming interface (API), virtual computing instance, or other suitable application. In some examples, when the service provider computers902are part of, or share an association with, an electronic marketplace, the user device908may be used by the user(s)910for procuring one or more items from the electronic marketplace. The user(s)910may request delivery of the purchased item(s) using the UAV904, or the service provider computers902may coordinate such delivery on its own.

The service provider computers902, perhaps arranged in a cluster of servers or as a server farm, may host web service applications. These servers may be configured to host a website (or combination of websites) viewable via the user device908. The service provider computers902may include at least one memory932and one or more processing units (or processor(s))934. The processor(s)934may be implemented as appropriate in hardware, computer-executable instructions, software, firmware, or combinations thereof. Computer-executable instruction, software or firmware implementations of the processor(s)934may include computer-executable or machine-executable instructions written in any suitable programming language to perform the various functions described.

The memory932may include more than one memory and may be distributed throughout the service provider computers902. The memory932may store program instructions (e.g., server management module936) that are loadable and executable on the processor(s)934, as well as data generated during the execution of these programs. Depending on the configuration and type of memory including the server management module936, the memory932may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, or other memory). The service provider computers902may also include additional removable storage and/or non-removable storage including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory932may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM.

Turning to the contents of the memory932in more detail, the memory932may include an operating system938and one or more application programs, modules or services for implementing the features disclosed herein including at least the server management module936. The server management module936, in some examples, may function similarly to the UAV management module918. For example, when the UAV904is in network communication with the service provider computers902, the UAV904may receive at least some instructions from the service provider computers902as the server management module936is executed by the processors934. In some examples, the UAV904executes the UAV management module918(e.g., to implement the features described with respect to the marker management engine102and/or the management component202ofFIG. 2) to operate independent of the service provider computers902.

In some examples, the service provider computers902may also include additional storage940, which may include removable storage and/or non-removable storage. The additional storage940may include, but is not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices.

The memory932and the additional storage940, both removable and non-removable, are examples of computer-readable storage media. For example, computer-readable storage media may include volatile or non-volatile, removable, or non-removable media implemented in any suitable method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. As used herein, modules may refer to programming modules executed by computing systems (e.g., processors) that are part of the service provider computers902. The modules of the service provider computers902may include one or more components. The service provider computers902may also include input/output (I/O) device(s)942and/or ports, such as for enabling connection with a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, speakers, a printer, or other I/O device.

In some examples, the service provider computers902may include a user interface944. The user interface944may be utilized by an operator, or other authorized user to access portions of the service provider computers902. In some examples, the user interface944may include a graphical user interface, web-based applications, programmatic interfaces such as application programming interfaces (APIs), or other user interface configurations. The service provider computers902may also include data store946. The data store946may include one or more databases, data structures, or the like for storing and/or retaining information associated with the service provider computers902. The data store946may include databases, such as customer information database948, landing zone database950, and marker information database951. The customer information database948, the landing zone database950, and the marker information database951may include similar information as the customer information database928, the landing zone database930, and the marker information database931of the onboard computer912. In some examples, the service provider computers902may store a larger amount of information in the data store946than the onboard computer912is capable of storing in the data store926. Thus, in some examples, at least a portion of the information from the databases in the data store946is copied to the databases of the data store926, e.g., periodically, occasionally, in connection with an event, or otherwise. In this manner, the data store926may have up-to-date information, without having to maintain the databases. In some examples, this information may be transferred as part of a delivery plan prior to the UAV904beginning a delivery mission.

In at least one embodiment, the UAV management module918and/or the server management module936(hereinafter, the “management modules”) may provide the functionality of the marker management engine102ofFIG. 1from a UAV centric or server centric viewpoint, respectively. For example, the management modules may be configured to receive delivery information associated with an item to be delivered by UAV. The management modules may receive order information identifying the item(s) to be delivered. In some cases, the delivery information may specify a particular UAV to be utilized for delivery, while in other cases the management modules may select a particular UAV for delivery. The management modules may select a marker design from, for example, a number of pre-defined marker designs (e.g., stored in the marker information database931and/or the marker information database951). In some examples, a marker may be designed to incorporate personal data associated with the user (e.g., a family portrait, the user's name, the user's address) and the marker may be modified or otherwise generated to include repetitive patterns that, when captured by a camera of the UAV904, will result in a moiré pattern. Optical pattern information indicating a design pattern and/or moiré pattern that the selected marker will produce may be stored, for example, as an association or as part of the order information in a user profile included in the customer information database928or the customer information database948. Additionally, or alternatively, the optical pattern information may be stored as an association with the marker within the marker information database931or the marker information database951.

In at least one embodiment, the management modules may execute instructions to cause marker information (e.g., a marker design) to be sent electronically to the user who requested delivery of the item. In some cases, the user may print the marker on a piece of paper or utilize an electronic device to (e.g., LED screen, LCD screen, projector, etc.) to otherwise display the marker. In some cases, the management modules may execute instructions that cause a physical marker to be created on, for example, a sheet of plastic. The physical marker may then be shipped to the user.

At a suitable point in time, the UAV904may receive instructions specifying a delivery time and a delivery location for a requested item. The item may be loaded onto the UAV904and the UAV may execute instructions to navigate to an area relatively near the requested delivery location. At a suitable distance (e.g., 1 mile, 2 miles, 1000 feet, etc.) within the delivery location, the management modules may execute instructions that cause a camera system (e.g., the camera system203ofFIG. 3, a component of the onboard computer912) of the UAV to begin capturing one or more images of the area (e.g., an aerial view of the ground from the location of the UAV).

Upon receiving the one or more images of the area, the management modules may analyze the images utilizing any combination of the techniques described herein in order to ascertain whether or not the marker associated with, for example, the delivery location is visible. For example, the management modules may analyze the image(s) to determine whether or not a moiré pattern is visible in the image(s). If a moiré pattern exists in the image(s), the management modules may compare the image(s) and/or moiré pattern to stored marker information (e.g., stored in the marker information database931and/or the marker information database951). If the moiré pattern in the captured image(s) matched (e.g., above some confidence threshold value), to the stored marker information, then the management modules may execute instructions that instruct the UAV904to navigate to the delivery location indicated by the detected marker. If a moiré pattern is not detected, or the moiré pattern detected does not match the stored marker information, the management modules may execute instructions that cause the UAV to continue a search pattern or to execute a new search pattern in order to enable the UAV to continue searching for the marker in the area or in another area.

In at least one embodiment, the management modules may analyze images utilizing the marker filter images discussed inFIGS. 3A, 3B, 4A and/or 4B. For example, the management modules may execute instructions to cause one or more marker filter images to be superimposed over the captured image. The management modules may analyze the resulting image to determine whether a moiré pattern is produced that matches stored marker information. For example, a marker may be generated that by itself does not create any sort of recognizable features. However, a marker filter image (e.g., vertical lines) may be utilized that, when superimposed over the marker and moved (e.g., according to the motion of the UAV) may produce an animation (e.g., a rider on horseback, a moving sequence of arrows, a “blinking” target, or the like). The management modules, upon determining that the animation matches the stored marker information, may execute instructions to cause the UAV to navigate toward the detected marker.

In at least one embodiment, the management modules may analyze an image utilizing any combination of the local tone mapping techniques discussed herein. For example, the management modules may identify one or more overexposed and/or underexposed portions of an image/video. The management modules may, upon detection of overexposure and/or underexposure may enable multiple cameras of the UAV904to capture an image, each camera utilizing different exposure settings. Alternatively, the management modules may initially instruct multiple cameras to capture images at different exposures prior to any determination that an underexposed and/or overexposed portion of an image exists. The management modules may determine, in some cases, a particular portion of a particular image that comprises the best exposure for the portion of the image from the set of images captured by the various UAV cameras. For example, the management modules may be configured to produce a histogram identifying portions of overexposure and/or underexposure within the image. In some cases, the management modules may replace overexposed/underexposed portions of an image with a portion of another image that includes a best exposure for that portion. In at least one example, the management modules may combine various best exposed portions of a variety of images in order to produce an optimal image. Once an optimal combination of portions of images are assembled, the management modules may analyze the image to detect a marker. If a marker is detected, and if the marker matches the stored marker information for the delivery location, for example, then the management modules may execute instructions to cause the UAV to navigate toward the marker and to complete its mission by delivering the item. The same methodology may be utilized in missions in which the UAV is picking up the item from the delivery location.

In at least one embodiment, the management modules may selectively brighten/darken portions of the captured images that are deemed to be too dark (e.g., within a threshold measure of darkness) or too light (e.g., within a threshold measure of lightness). For example, the management modules may be configured to produce a histogram identifying portions of overexposure and/or underexposure. Accordingly, the management modules may utilize a local tone mapping algorithm to adjust the pixels within the identified regions to a pixel value that is commensurate with the pixel values within a range (e.g., within 10 pixels, 5 pixels, etc.) of the pixel being adjusted.