Patent Description:
Determination of an emergency landing zone in any terrain/region requires information about depth and sematic understanding of the scene in real time, especially to find dynamic objects that would need to be avoided before proceeding to land. Static information about terrain and semantics, such as buildings, highways, etc., can be obtained through pre-recorded databases. While dynamic information can be obtained through various sensors, such light detection and ranging (LiDAR), radar, and camera devices on the vehicle, such information is very raw and requires various kinds of perception algorithms to make sense of the input data and find a safe region to proceed for landing. <NPL> relates to full autonomy for fixed-wing unmanned aerial vehicles requiring the capability to autonomously detect potential landing sites in unknown and unstructured terrain. <CIT> relates to a technique for autonomous landing by an aerial vehicle. <NPL> relates to a system that locates candidate landing sites for a forced landing using mid-level discriminative patches.

A system and method for localization of safe landing zones is provided. The system comprises at least one image-capture device onboard an aerial vehicle, and at least one processor onboard the aerial vehicle and operatively coupled to the image-capture device. The processor is operative to execute instructions to perform a method that comprises: receive, from the image-capture device, two or more overlapping images of a landscape underneath the aerial vehicle; generate, based on the overlapping images, a landing quality heatmap of the landscape, wherein the landing quality heatmap is processed to generate a binary mask based on a threshold value, wherein the binary mask contains pixels with good landing quality and pixels with bad landing quality, and the pixels with bad landing quality are clustered using a connected component labelling strategy to produce a binary mask with connected component labelling; identify, based on the binary mask with connected component labelling, one or more regions of the landscape having one or more potential landing zones and obstacles; and determine a location of a safe landing zone using a distance transform of the one or more regions of the landscape. The location of the safe landing zone is in an area within one of the potential landing zones that is farthest from the obstacles. The location of the safe landing zone is then stored in a database.

In the following detailed description, embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It is to be understood that other embodiments may be utilized without departing from the scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense.

A system and method for localization of safe landing zones using dense depth and landing quality heatmaps is described herein. The present approach provides for using several algorithms to find a safe landing zone for an autonomous aerial vehicle with more reliability.

The system allows for real-time determination of potential emergency landing zones as a vehicle flies over terrain data not in the training set of algorithms. These landing zones can then be queried and used for landing in case of an emergency when it is not possible to reach predetermined landing zones.

While there are various methods to determine depth and semantic information of a camera scene, methods to evaluate landing quality of a scene only produce a pixel-level heatmap. Determination of a landing zone or a target for landing guidance requires further analysis with computational geometry algorithms and morphological operations. The present system utilizes one or more sensors and processing units that can host the necessary processing algorithms, and interact with a guidance and control system to proceed with a safe landing.

The present approach is intended to solve the problem of identifying an emergency landing region in real time, by making use of several perception pipelines such as semantic segmentation and depth estimation, which is then further processed to generate a landing zone quality map. This quality map or heat map is then passed through computational geometry-based algorithms to determine "safe" zones/regions. These regions are then further localized in three-dimensional (3D) coordinate frames for guidance. The present system achieves this by making use of a camera and an inertial navigation source.

The present approach is applicable for providing an emergency safety feature in autonomous aerial operations of unmanned aerial systems and urban aerial mobility (UAM) services. This allows single pilot operations and fully autonomous operations to be more widely accepted, with the added safety feature of emergency landing. Various service providers can then save cost on training and paying the pilots, as the UAM industry scales up with more and more vehicles in the urban air space. The features in the present system allow for handling off-nominal situations, especially emergency situations, safely and reliably. Due to logistics and regulation constraints, it would be difficult to scale UAM operations to Beyond Visual Line of Sight without having an emergency landing feature that allows the vehicle to determine a safe landing zone in real time, and provide autonomous landing guidance to avoid an imminent crash or accident.

In addition, the features of the present system and method can be implemented for use in various manned aerial systems, which can make use of these features in case of an emergency.

In one aspect, the present system utilizes a post processing algorithm that allows for the determination of safe landing zones with more reliability. The post processing algorithm uses morphological operations and computational geometry over pixels and 3D voxels to determine locations that are farthest from obstacles detected by an upstream perception pipeline. These operations attempt to isolate solids objects or blobs of obstacles, and calculate distances of open space pixels/voxels from these obstacles. The open spaces are characterized by geometric and semantic information encoded in a pixel heatmap depicting landing quality. The detected safe zones are then localized using navigation data and stored in an onboard database. The database can be queried in real-time in case of an emergency situation based on its nature, such as high winds/gusts, state of charge or system/subsystem failure.

The onboard system can determine parameters such as expected quality of the landing zone, for example, degree of roughness/flatness, slope, size, and distance from the current location, to find the best possible landing location. An autopilot onboard the vehicle then simply guides the vehicle to the landing location determined as per the prescribed regulations.

Further details of the present system and method are described as follows and with reference to the drawings.

<FIG> is a block diagram of a system <NUM> for providing localization of a safe landing zone for an aerial vehicle <NUM>, according to one embodiment. The aerial vehicle <NUM> can be an unmanned aerial vehicle (UAV), UAM vehicle, unmanned aircraft system (UAS), or the like, which employs a visual navigation system. The aerial vehicle <NUM> has at least one image-capture device <NUM> mounted thereon, such as a machine vision camera, which is operative to capture images of an area over which aerial vehicle <NUM> traverses. The aerial vehicle <NUM> includes at least one processor <NUM> operatively coupled to image-capture device <NUM>.

The processor <NUM> is operative to perform a method <NUM> for localization of a safe landing zone for aerial vehicle <NUM>. The method <NUM> generally includes the following steps, which are described in further detail hereafter. The method <NUM> begins with receiving, from image-capture device <NUM>, two or more overlapping images depicting a landscape underneath aerial vehicle <NUM> (block <NUM>). The method <NUM> then generates, based on the two or more overlapping images, a heatmap for the landscape (block <NUM>). The method <NUM> identifies, based on the heatmap, one or more regions of the landscape having one or more potential landing zones and obstacles (block <NUM>). The method <NUM> then determines a location of a safe landing zone using a distance map transform of the one or more regions of the landscape (block <NUM>). The location of the safe landing zone is in an area within one of the potential landing zones that is farthest from the obstacles. Finally, method <NUM> stores the location of the safe landing zone in a database (block <NUM>). The location of the safe landing zone is then available as needed for an emergency landing of aerial vehicle <NUM>.

<FIG> is a block diagram of an onboard system <NUM>, according to an exemplary embodiment, which can be implemented to provide localization of a safe landing zone for an aerial vehicle <NUM>, which can be a UAV, UAM, UAS, or the like. The onboard system <NUM> includes a ground truth navigation source, such as an onboard Global Navigation Satellite System (GNSS)/ Inertial Navigation System (INS) navigator <NUM>. In some embodiments, GNSS/INS navigator <NUM> can be fused with other alternative systems, such as camera, LiDAR, or radar based position, velocity, and attitude estimation systems.

An onboard machine vision camera <NUM> is operatively coupled to GNSS/INS navigator <NUM>, and is operative to capture real-time images from aerial vehicle <NUM>. An onboard processor <NUM> is operatively coupled to GNSS/INS navigator <NUM> and to machine vision camera <NUM>. The processor <NUM> is configured to host a neural network and machine vision algorithms, and to interface with other onboard peripheral systems.

An autopilot <NUM> is operatively coupled to processor <NUM>, and is configured to listen to guidance commands and control aerial vehicle <NUM>. The autopilot <NUM> can be coupled to processor <NUM> through a Universal Asynchronous Receiver/Transmitter (UART), which is operative receive and transmit serial data. A Precision Time Protocol (PTP) switch and clock <NUM> is operatively coupled to GNSS/INS navigator <NUM>, machine vision camera <NUM>, and processor <NUM>, through an Ethernet connection. The PTP switch provides for time synchronization between all onboard systems, which is required to ensure the coherence between the detected landing zones and the navigation source.

<FIG> is a flow diagram of an exemplary method <NUM> for providing localization of a safe landing zone for an aerial vehicle, such as a UAV, UAM, UAS, or the like. The aerial vehicle has at least one machine vision camera mounted thereon, which is operative to capture, in real-time, a set of images <NUM> that are added to an image buffer <NUM>. The set of images <NUM> include overlapping images of a landscape underneath the aerial vehicle. The set of images <NUM> are fed to a deep neural network <NUM>, which is hosted in an onboard processor. The deep neural network <NUM> is configured to compute and generate a landing score/zone heatmap <NUM>. The heatmap <NUM> is processed at <NUM> to generate a binary mask <NUM>, which is labelled with landing zone blobs <NUM> and other objects <NUM> for post processing at <NUM>. The safest landing spot is then calculated using a distance map transform of binary mask <NUM>. This produces a landing zone distance map <NUM> showing a safest point <NUM> for landing. A landing spot <NUM> is then detected and selected based on safest point <NUM> of landing zone distance map <NUM>. The landing spot <NUM> is shown in zone selection image <NUM>. The location of landing spot <NUM> is then stored in a database <NUM>, in case of an emergency.

<FIG> is a block diagram of an onboard system architecture <NUM> for an aerial vehicle, according to one embodiment, which is operative to provide localization of a safe landing zone. The system architecture <NUM> includes a camera <NUM> that operates with a camera driver <NUM>, which is configured to send image information <NUM> from camera <NUM> to a tracking model <NUM>. The tracking model <NUM> is operatively coupled to a mapping model <NUM> and a multi-sensor pose fusion module <NUM>. The tracking model <NUM> provides image and pose information to mapping model <NUM>, which in turn provides depth information back to tracking model <NUM>. The mapping model <NUM> generates a pixelwise depth map and a landing quality pixel map. The landing quality pixel map incorporates the depth map information with other semantic data.

A post processing module <NUM> is operatively coupled to mapping model <NUM>. The post processing module <NUM> is configured to calculate potential landing regions based on information from mapping model <NUM>, including one or more the landing quality pixel maps. The multi-sensor pose fusion module <NUM> is configured to receive pose information (PoseSFM) from tracking model <NUM>, and estimate vehicle state data comprising data regarding the six degrees of freedom (Pose: 6DOF), including 3D rotation and 3D translation. The multi-sensor pose fusion module <NUM> is configured to output the vehicle state data (Pose NED/LLA) to post processing module <NUM>.

A database of landing regions <NUM> is operatively coupled to post processing module <NUM> and multi-sensor pose fusion module <NUM>. A landing zone tracker <NUM> is operatively coupled to post processing module <NUM>. During operation, one or more potential landing regions is projected in 3D using a camera projection matrix and the depth map in post processing module <NUM>. The potential landing regions can then be stored for a future emergency in database of landing regions <NUM>, or can be used in real time with landing zone tracker <NUM>.

A guidance module <NUM> is operatively coupled to database of landing regions <NUM> and landing zone tracker <NUM>. The guidance module <NUM> is operative to send a request to database of landing regions <NUM> for a landing region candidate. The best landing region candidate is then sent from database of landing regions <NUM> to guidance module <NUM>.

The system architecture <NUM> also includes an autopilot <NUM> that operates with an autopilot driver <NUM>, which is operatively coupled to multi-sensor pose fusion module <NUM>. The autopilot <NUM> is configured to receive guidance commands, including landing target information, from guidance module <NUM> through autopilot driver <NUM>. The autopilot <NUM> is configured to send various commands (AP ARM, AP Mode, emergency signal) to a supervisor module <NUM>, which is operative to send an initiate land command to guidance module <NUM>. The autopilot <NUM> is also configured to send pose commands (PoseEKF) to multi-sensor pose fusion module <NUM>.

As shown in <FIG>, camera <NUM> and autopilot <NUM> are oriented on the body of the aerial vehicle in an X-Y-Z coordinate system, such that the optical axis of camera <NUM> is pointed along the Z-direction.

<FIG> is a flow diagram of an exemplary post processing method <NUM>, illustrating various aspects of how a safe landing zone identification is performed. <FIG> is a flow diagram of an exemplary guidance processing method <NUM>, which can be used in conjunction with post processing method <NUM> of <FIG>.

In post processing method <NUM>, a set of images <NUM>, captured by a camera mounted on the vehicle, is sent to an images sequence buffer <NUM>. Information about the images is sent to a mapping model <NUM> that is configured to generate a landing quality map <NUM>, which is a pixelwise heatmap of landing zone quality. This heatmap is generated by a neural network that combines depth and semantic information to generate a landing zone quality for each pixel in the image of the heatmap.

The post processing method <NUM> then performs thresholding (block <NUM>) of the heatmap from mapping model <NUM>, to generate a binary mask <NUM> based on a threshold value. The thresholding can provide blurring as a preconditioning step before generating binary mask <NUM>. The threshold value is selected based on "safe" landing quality margins.

Next, post processing method <NUM> performs connected component labelling (CCL) (block <NUM>) of binary mask <NUM> to produce a binary mask with CCL <NUM>. The binary mask contains pixels with good landing quality and with bad landing quality. Pixels with bad landing quality are clustered using a connected component labelling strategy. These clustered pixels are called blobs.

The post processing method <NUM> then performs a distance transform (block <NUM>), to produce a landing zone distance map <NUM> based on the binary mask with CCL. The image with blobs (bad landing quality pixels) and the background (good landing quality pixels) are transformed such as by using a Euclidean distance transform function, or other standard methods for distance transform. The distance transform assigns a distance value to each background pixel from the nearest blob pixel.

Thereafter, post processing method <NUM> selects a landing target pixel and radius of confidence (block <NUM>), based on the landing zone distance map produced by the distance transform. The pixel on the landing zone distance map with the maximum distance is selected as the landing target pixel, and the distance value is the radius of confidence of the landing region in pixels. An example of the landing target pixel and radius of confidence is shown as a landing spot <NUM> in a zone selection image <NUM>. The landing target pixel is further processed as described with respect to <FIG>, to determine a landing spot location.

As shown in <FIG>, guidance processing method <NUM> is carried out in conjunction with post processing method <NUM>. As described above, the images captured by the camera on the vehicle are sent to images sequence buffer <NUM>. Information about the images is sent from image sequence buffer <NUM> to a depth module <NUM>, which includes mapping model <NUM> (<FIG>) and a tracking model <NUM>.

As described above, mapping model <NUM> is configured to generate a landing quality map <NUM>, which is a pixelwise heatmap (depth map) of landing zone quality. Landing quality map post processing (block <NUM>) is performed based on the information from mapping model <NUM> as described above for <FIG> (thresholding, connected component labelling, and distance transform). This produces the landing target pixel and radius of confidence (block <NUM>), such as landing spot <NUM> in zone selection image <NUM>.

The tracking model <NUM> is configured to provides image and pose information to mapping model <NUM>, which in turn provides depth information back to tracking model <NUM>. A multi-sensor pose fusion unit <NUM> is configured to receive pose information from tracking model <NUM>. The multi-sensor pose fusion unit <NUM> is configured to estimate vehicle state data.

The guidance processing method <NUM> generates a landing zone three-dimensional (3D) projection (block <NUM>) from landing quality map <NUM> and the landing target pixel such as in zone selection image <NUM>. For example, the landing target pixel is projected onto a 3D coordinate frame centered at the camera using a camera projection model (such as Pinhole) and the depth estimated for that pixel by the neural network, to generate a 3D landing target. The 3D landing target is then transformed to a global coordinate frame (block <NUM>) using vehicle state data estimated by multi-sensor pose fusion unit <NUM>, to produce a target location for guidance (block <NUM>).

The processing units and/or other computational devices used in the method and system described herein may be implemented using software, firmware, hardware, or appropriate combinations thereof. The processing unit and/or other computational devices may be supplemented by, or incorporated in, specially-designed application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or graphics processing units (GPUs). In some implementations, the processing unit and/or other computational devices may communicate through an additional transceiver with other computing devices outside of the navigation system, such as those associated with a management system or computing devices associated with other subsystems controlled by the management system. The processing unit and/or other computational devices can also include or function with software programs, firmware, or other computer readable instructions for carrying out various process tasks, calculations, and control functions used in the methods and systems described herein.

The methods described herein may be implemented by computer executable instructions, such as program modules or components, which are executed by at least one processor or processing unit. Generally, program modules include routines, programs, objects, data components, data structures, algorithms, and the like, which perform particular tasks or implement particular abstract data types.

Instructions for carrying out the various process tasks, calculations, and generation of other data used in the operation of the methods described herein can be implemented in software, firmware, or other computer readable instructions. These instructions are typically stored on appropriate computer program products that include computer readable media used for storage of computer readable instructions or data structures. Such a computer readable medium may be available media that can be accessed by a general purpose or special purpose computer or processor, or any programmable logic device.

Suitable computer readable storage media may include, for example, nonvolatile memory devices including semi-conductor memory devices such as Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory devices; magnetic disks such as internal hard disks or removable disks; optical storage devices such as compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs; or any other media that can be used to carry or store desired program code in the form of computer executable instructions or data structures.

Claim 1:
A system comprising:
at least one image-capture device (<NUM>) onboard an aerial vehicle (<NUM>, <NUM>); and
at least one processor (<NUM>, <NUM>) onboard the aerial vehicle (<NUM>, <NUM>) and operatively coupled to the at least one image-capture device (<NUM>), wherein the at least one processor (<NUM>, <NUM>) is operative to execute instructions to perform a method that comprises:
receive, from the image-capture device (<NUM>), two or more overlapping images of a landscape underneath the aerial vehicle (<NUM>, <NUM>);
generate, based on the overlapping images, a landing quality heatmap of the landscape;
processing the landing quality heatmap to generate a binary mask (<NUM>, <NUM>) based on a threshold value, wherein the binary mask (<NUM>, <NUM>) contains pixels with good landing quality and pixels with bad landing quality, and the pixels with bad landing quality are clustered using a connected component labelling (CCL) strategy to produce a binary mask with connected component labelling;
identify, based on the binary mask with connected component labelling, one or more regions of the landscape having one or more potential landing zones and obstacles;
determine a location of a safe landing zone using a distance transform of the one or more regions of the landscape, wherein the location of the safe landing zone is in an area within one of the potential landing zones that is farthest from the obstacles; and
storing the location of the safe landing zone in a database (<NUM>).