Patent Description:
Drone delivery is positioned to change the way lightweight goods are moved between locations (e.g. from retailer to consumer). However high-density urban spaces, particularly multi-level buildings, present a challenge to aircraft that are attempting to make a delivery. The most straightforward and desirable delivery approaches have an unobstructed path for the drone to follow during its route to drop off a package, which is challenging to identify and allocate in high-density urban spaces. The challenge of identifying the correct delivery destination is particularly acute for urban apartment residences that are tightly situated with low incidence of externally distinguishing features. Compounding the problem is the navigational challenge of maneuvering inside the "urban canyon" where traditional navigation methods (e.g. GPS) are unreliable.

<CIT> Al discloses a method and system for determining a delivery location. The method includes enabling a UAV delivery application specifying a delivery location for delivery of a package. A street address defining the delivery location is received and an eyewear based video device is enabled and directed towards a geographical area associated with the delivery of the package. GPS data associated with the geographical area is retrieved from the eyewear based video device and first GPS coordinates identifying a location of an embedded computing device are retrieved from a GPS system. A distance between the first GPS coordinates and the geographical area is calculated. Additionally, second GPS coordinates identifying the geographical area are calculated based on the GPS data, the distance, and the first GPS coordinates and it is determined if the second GPS coordinates are located within a specified perimeter surrounding the street address defining the delivery location.

<CIT> Al discloses an unmanned aerial vehicle that includes a storage storing information regarding a plurality of regions and a control circuitry. The control circuitry is configured to select a first one of the plurality of regions that has a highest priority among the plurality of regions as a destination, and change the destination to a second one of the plurality regions that has a next highest priority among the plurality of regions in response to determining that a selected path to the destination is not suitable for flight.

Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Not all instances of an element are necessarily labeled so as not to clutter the drawings where appropriate. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles being described.

Embodiments of a system, apparatus, and method of operation for validation and enrollment of balconies into a parcel delivery service using unmanned aerial vehicles (UAVs) are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

One solution for delivering lightweight parcels to multi-level buildings, such as high-density urban apartment units, is to use unmanned aerial vehicles (UAVs) to deliver parcels to the balconies of these units. This delivery route often provides a clear, unobstructed approach to a location that is conveniently accessed by the customer. Not only are balconies convenient locations for the customer, but they are typically secure locations over which only the customer has ready access. The convenience and security of private balconies makes these locations a particularly attractive delivery drop-off destination from the customer's perspective. As such, balconies are well suited for UAV deliveries of lightweight parcels, such as letters and documents, prescription drugs, small grocery items, prepared or hot meals, small electronics, etc..

<FIG> illustrates the delivery of a parcel <NUM> by a UAV <NUM> to a balcony <NUM> of a multi-level building <NUM>, in accordance with an embodiment of the disclosure. However, before UAV <NUM> can commence deliveries of parcels <NUM>, balcony <NUM> must be validated and then enrolled in the aerial delivery service. Validation of balcony <NUM> to receive aerial deliveries includes identifying the correct balcony <NUM> from the other closely located and similar looking neighboring balconies. This task alone can be challenging in large, high-density, multi-level urban buildings. Identification of the correct balcony for enrollment with a corresponding customer account ensures future parcels <NUM> are delivered to the correct end-user/customer. In addition to identifying the correct balcony, validation includes ensuring that parcels <NUM> can be reliably and safely delivered to the requested balcony. Reliable, safe deliveries require an unobstructed path to balcony <NUM> to permit safe ingress and egress by UAV <NUM>. Sufficient space and clearance on or around balcony <NUM> is also needed.

Embodiments herein describe a procedure for validating balcony <NUM> to receive deliveries of parcels <NUM>. The validation techniques ensure correct identification and association of balcony <NUM> with a given customer along with determination of whether ingress and egress to balcony <NUM> is unobstructed and whether balcony <NUM> is adequately sized and/or free of obstructions to receive aerial deliveries. Once validation is successfully achieved, enrollment to receive future parcels <NUM> via UAV aerial deliveries can proceed.

The illustrated embodiment of UAV <NUM> is a vertical takeoff and landing (VTOL) UAV that includes separate propulsion units for providing horizontal and vertical propulsion. UAV <NUM> is a fixed-wing aerial vehicle, which as the name implies, has an airfoil that can generate lift based on the wing shape and the vehicle's forward airspeed when propelled horizontally. The illustrated embodiment of UAV <NUM> is merely demonstrative, and other types of UAVs, fixed wing or not, may be used to provide aerial deliveries to balconies of multi-level buildings.

It should be understood that references herein to an "unmanned" aerial vehicle or UAV can apply equally to autonomous and semi-autonomous aerial vehicles. In a fully autonomous implementation, all functionality of the aerial vehicle is automated; e.g., pre-programmed or controlled via real-time computer functionality that responds to input from various sensors and/or pre-determined information. In a semi-autonomous implementation, some functions of an aerial vehicle may be controlled by a human operator, while other functions are carried out autonomously. Further, in some embodiments, a UAV may be configured to allow a remote operator to take over functions that can otherwise be controlled autonomously by the UAV. Yet further, a given type of function may be controlled remotely at one level of abstraction and performed autonomously at another level of abstraction. For example, a remote operator may control high level navigation decisions for a UAV, such as specifying that the UAV should travel from one location to another (e.g., from a warehouse in a suburban area to a delivery address in a nearby city), while the UAV's navigation system autonomously controls more fine-grained navigation decisions, such as the specific route to take between the two locations, specific flight control inputs to achieve the route and avoid obstacles while navigating the route, and so on.

<FIG> illustrates a UAV parcel delivery system <NUM>, in accordance with an embodiment of the disclosure. The illustrated embodiment of UAV parcel delivery system <NUM> includes one or more UAVs <NUM>, a UAV enrollment & validation backend <NUM> running on one or more cloud-based servers <NUM>, one or more databases <NUM>, and a client-side application <NUM> executable on a computing device <NUM> (e.g., mobile computing device). The illustrated embodiment of database <NUM> includes secure user data <NUM>, aerial maps <NUM>, and detailed maps <NUM>.

Backend <NUM> communicates with application <NUM> and UAVs <NUM> to manage user accounts, solicit user data and balcony information from both application <NUM> and UAVs <NUM> to validate balconies and enroll new customers for the aerial delivery service. Backend <NUM> maintains secure user data <NUM> in database <NUM> related to the customer accounts and data related to the physical location, size, geometry, delivery routes, identifiable attributes, etc. of the customers' balconies. User data <NUM> is anonymized where possible, and such anonymization may occur within any component of system <NUM> starting with the immediate information gateways or sources of user data <NUM>, such as application <NUM> or UAV <NUM>. Backend <NUM> also accesses aerial maps <NUM> from database <NUM>, which detail buildings and the vicinities surrounding buildings associated with customer accounts. Aerial maps <NUM> not only aid in identification of a customer's building during the enrollment and validation process, but may also be analyzed for ingress and egress obstructions to balconies, such as trees, telephone poles, adjacent buildings, towers, etc. Detailed maps <NUM> may include various visual models or pictures of buildings for the purposes of navigation to/from and identification of balconies. Where possible, access to user data <NUM> is limited to the automated machines and algorithms needed to enroll and validate a balcony, and anonymized or obscured from potential human operators. Application <NUM> provides a client-side interface to enable customers to communicate with backend <NUM>, enroll in the aerial delivery system, and potentially place orders for delivery of one or more parcels <NUM> after successful enrollment.

Although system <NUM> illustrates backend <NUM> as residing on a single server <NUM> and secure user data <NUM>, aerial maps <NUM>, and detailed maps <NUM> stored in a single database <NUM>, it should be appreciated that system <NUM> may be implemented as a distributed system with components stored and executing in many locations spread across many remote computing platforms.

<FIG> is a flow chart illustrating a process <NUM> for operation of the UAV parcel delivery system <NUM> to enroll and validate balcony <NUM> for aerial parcel deliveries, in accordance with an embodiment of the disclosure. The order in which some or all of the process blocks appear in process <NUM> should not be deemed limiting. Rather, one of ordinary skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel. Furthermore, several of the processing blocks depict steps that are optional and may be omitted.

In a process block <NUM>, backend <NUM> obtains an identification of a general location of a new customer's balcony via application <NUM> executing on computing device <NUM>. In one embodiment, computing device <NUM> is the customer's mobile computing device, such as a smartphone, tablet, laptop, or otherwise. In one embodiment, the identification of the general location is a physical address of the customer's building. If the building is a multi-dwelling unit, then the physical address may also include a unit number (e.g., suite or apartment number). Additionally (or alternatively), the customer may be solicited to enter one or more of: the floor level of their unit, an approximate height from ground of their balcony, the number of balconies above or below their balcony, or other general location information.

The general location information (e.g., physical address with unit number) is used by backend <NUM> to select an image from database <NUM> (e.g., selected from aerial maps <NUM> or detailed maps <NUM>) to aid the user in precisely identifying the location of their balcony <NUM> on their building <NUM>. If the customer's building has not been mapped, or mapped sufficiently well to know the precise location of their balcony based upon their unit number (decision block <NUM>), then process <NUM> continues to a process block <NUM>. In process block <NUM>, backend <NUM> sends an aerial map <NUM> to application <NUM> for display to the customer. Aerial map <NUM> may be selected based upon the user's physical address. <FIG> illustrates an example aerial map image <NUM> that includes the customer's building <NUM> and surrounding vicinity. The customer is then solicited to precisely identify the location of their balcony <NUM> on aerial map image <NUM> by end-user interaction with aerial map <NUM> (process block <NUM>). For example, the end-user of application <NUM> may physically touch (or otherwise indicate) a position on aerial map image <NUM> to convey the precise location of their balcony <NUM> on their building <NUM>.

In some instances, the customer's building may be sufficiently mapped (decision block <NUM>) for backend <NUM> to display a visual model or a picture of a side of building <NUM> that the system believes corresponds to the customer's unit number (process block <NUM>). For example, enough existing customers within building <NUM> may have already enrolled in the aerial delivery service such that backend <NUM> has constructed a model or a picture of building <NUM> and is able to make an educated determination or localization of the precise location of balcony <NUM> on building <NUM>. <FIG> illustrates an example visual model or picture <NUM> of the side of building <NUM> that may be presented to the prospective new customer. The end-user may then be asked to identify their balcony <NUM> via physical interaction with the image (e.g., touching the precise location). Alternatively, the system may highlight a position believed to correspond to the prospective new customer's balcony <NUM>, and solicit a confirmation of the precise balcony location (process block <NUM>).

The precise location determined in process block <NUM> may be translated by backend <NUM> into global positioning system (GPS) coordinates of the customer's balcony. However, in most scenarios, balconies <NUM> are still too densely populated in multi-level buildings for unique identification or remote determination of deliverability to balcony <NUM>. Accordingly, in a process block <NUM>, the end-user is solicited to use application <NUM> and computing device <NUM> to capture one or more pictures of balcony <NUM> or one or more pictures from balcony <NUM>.

For example, application <NUM> may request the end-user to capture one or more pictures while standing on balcony <NUM>. These pictures may be pictures outward from balcony <NUM> to capture images of any potential ingress or egress obstructions for UAV <NUM>. The outward balcony pictures may also be used by UAV <NUM> to aid visual navigation to balcony <NUM> and identify balcony <NUM> from the other balconies in building <NUM>. These pictures may be pictures of the specific location on balcony <NUM> where the customer wants their packages dropped off. Application <NUM> may guide the end-user to acquire pictures from several different vantage points and/or directions. In one embodiment, application <NUM> may request that the end-user stand in one or more locations on balcony <NUM> and acquire one or more panoramic pictures or pictures in specified directions. The pictures may be of a quality and nature to not only identify any ingress/egress obstructions for an aerial vehicle, but also extract visual measurements so that backend <NUM> can estimate a size of balcony <NUM> (e.g., width and depth) and/or a size of the access opening to balcony <NUM> (e.g., area above railing and below the next story balcony). Thus, backend <NUM> may analyze the pictures to determine if there is sufficient space on balcony <NUM> for parcel deliveries.

In some embodiments, application <NUM> may request the end-user to obtain one or more pictures of their balcony from the ground. The ground based pictures may also be used to identify any obstructions or hazards for UAV <NUM>, and/or further aid visual navigation by UAV <NUM> to the correct balcony.

In a process block <NUM>, application <NUM> may further request authorization from the end-user to gather additional location identification information from computing device <NUM>. Computing device <NUM> may be queried for GPS coordinates, barometric pressure, or other available data that can aid locating balcony <NUM>. In one embodiment, computing device <NUM> may be queried, with the user's permission, for networking identifiers that identify wireless network signals used by computing device <NUM> to wirelessly communicate with other computing devices. For example, application <NUM> may obtain the name or identifier of the wireless access point (e.g., service set identifier) or Bluetooth signal used by computing device <NUM>. This information may be used by UAV <NUM> to hasten the search and identification of the correct balcony <NUM> once it arrives on scene for a flyby validation. This identifying information is for use by the automated systems, and thus anonymized where possible, or otherwise obscured from human operators when human operators perform supporting roles.

Once backend <NUM> has acquired the balcony's general location (e.g., building address, unit number, floor number), precise location (as identified on the aerial map image, visual model, or picture), along with any additional information (e.g., pictures, locational data, networking identifiers), the collected data is analyzed to calculate a deliverability score (decision block <NUM>). The deliverability score may be a measure or estimation of aerial obstructions to ingress/egress and whether balcony <NUM> has sufficient size or clearances to accept parcel deliveries via air. The deliverability score may also take into consideration previous validations/enrollments (or rejections thereof) for neighboring balconies in the same building. In some scenarios, the data may be sufficiently clear (e.g., above a threshold high score) to enroll the customer and their balcony without the need for a flyby validation by UAV <NUM> (process block <NUM>). In other scenarios, the data may be sufficiently clear (e.g., below a threshold low score) so as to summarily reject the enrollment without need for a flyby validation (process block <NUM>). However, in many cases, the deliverability score may fall in a middle range that calls for dispatching UAV <NUM> to perform an initial or one-time flyby validation. In this mid-range scenario where a flyby validation is ordered, process <NUM> continues to a process block <NUM>.

In process block <NUM>, the end-user may be provided with a fiducial marker that is associated with balcony <NUM> by backend <NUM>. The fiducial marker operates as a visual code that UAV <NUM> can seek out once it arrives on scene and use to identifying the correct balcony. The fiducial marker may be provided to the end-user via application <NUM> in the form of a visual code, such as a quick response (QR) code. In one embodiment, the end-user may print the fiducial marker and attach it to their balcony. In yet other embodiments, the fiducial marker may simply be displayed by application <NUM> on computing device <NUM> for the end-user to display to UAV <NUM> from balcony <NUM> (e.g., see fiducial marker <NUM> in <FIG>).

Once UAV <NUM> arrives in the vicinity of building <NUM>, backend <NUM> may notify the end-user that UAV <NUM> is on scene. The notification may be in the form of a text message, phone call, or in-app notification. The message may be viewed as an alert that UAV <NUM> is currently seeking out balcony <NUM>. If the end-user opted not to print out the fiducial marker and adhere it to balcony <NUM>, then the end-user may simply hold computing device <NUM> outward in a prominent manner for UAV <NUM> to discover (e.g., see <FIG>).

When UAV <NUM> arrives on scene for the flyby validation, UAV <NUM> may initially go to building <NUM> identified by the physical address provided in process block <NUM>, and further the side of building <NUM> identified by the end-user in process block <NUM>. Additionally, UAV <NUM> may hover at an initial altitude or height determined from the end-user's unit number or floor level (process block <NUM>). Alternatively (or additionally), UAV <NUM> may hover at an initial altitude matching the barometric pressure obtained from computer device <NUM>, if provided. From this initial altitude, UAV <NUM> may use visual based navigation techniques to search for balcony <NUM>. These visual based techniques may reference the images provided by the end-user in process block <NUM> while also optically scanning for the fiducial marker. Additionally, UAV <NUM> may electronically scan for the user's network identifier (e.g., WiFi network name or service set identifier (SSID), Bluetooth signal, etc.) as further clues of whether UAV <NUM> is in the vicinity of the customer's balcony <NUM>. The search may continue until UAV <NUM> is able to identify fiducial marker <NUM> (process block <NUM>), or for a finite period of time. While searching for balcony <NUM>, UAV <NUM> would take precautions to preserve privacy by anonymizing private or sensitive visual or optical information, for example by blurring windows, faces, or other such information.

In one embodiment, application <NUM> may solicit the end-user to provide directional feedback to aid UAV <NUM> in the identification of balcony <NUM> and/or refinement of a precise target delivery location on balcony <NUM> (process block <NUM>). For example, in one embodiment, application <NUM> may display a user interface, such as user interface <NUM> illustrated in <FIG>, that permits the end-user to provide directional feedback to aid UAV <NUM> to get on target. The directional feedback may not provide the end-user with actual or direct navigation control over UAV <NUM>, but rather is considered information feedback to UAV <NUM>, which it may take under advisement. In one embodiment, UAV <NUM> may permit the end-user to provide directional feedback in the form of gestures, such as the gesture <NUM> illustrated in <FIG> to identify target delivery location <NUM> on balcony <NUM>. Once the target delivery location <NUM> is identified, both balcony <NUM> and target delivery location <NUM> may be recorded/registered by backend <NUM>.

In a process block <NUM>, UAV <NUM> captures one or more images of balcony <NUM>, which may include fiducial marker <NUM> (see <FIG>). The images are analyzed by backend <NUM> not only for identification, but also to determine whether balcony <NUM> is viable for receiving aerial parcel deliveries. Deliverability is based upon clear ingress and egress along with sufficient clearances and space. One of the images may also serve as a confirmatory image sent back to the end user via backend <NUM> and application <NUM>. In a process block <NUM>, the confirmatory image captured by UAV <NUM> is provided to the end user and a final confirmation solicited. If the end user confirms the confirmatory image and backend analysis of the images deems balcony <NUM> to be deliverable (decision block <NUM>), then balcony <NUM> is validated and enrolled (process block <NUM>). If the end user fails to confirm the confirmatory image or backend <NUM> analysis determines that balcony <NUM> is not accessible or safe for deliveries (decision block <NUM>), then validation fails and enrollment is rejected (process block <NUM>).

Claim 1:
A computer-implemented method for validating a balcony (<NUM>) of a multi-level building (<NUM>) comprising multiple balconies to receive aerial delivery of a parcel via an unmanned aerial vehicle, UAV (<NUM>), the method comprising:
obtaining (<NUM>), via a computing device (<NUM>), a first identification of a general location of the balcony, wherein the computing device is a computing device of an end-user;
generating (<NUM>; <NUM>), for display on the computing device, a first image representing the multi-level building, wherein the multi-level building includes the balcony and the first image is selected based upon the general location identified by the end-user, wherein generating the first image representing the multi-level building comprises providing (<NUM>) a visual model of a side of the multi-level building that includes the balcony;
obtaining (<NUM>), via the computing device, a second identification or a confirmation of a precise location of the balcony in the multi-level building, wherein the second identification or the confirmation are received in response to an end-user interaction with the first image;
determining (<NUM>) a deliverability score based at least in part on the precise location of the balcony; and
indicating an enrollment status to the end-user, via the computing device, the enrollment status generated based upon the deliverability score.