Patent Description:
Vision augmented aircraft landing is a technology that can significantly reduce onboard equipment, both in terms of weight and cost. A camera and processing computer can replace multiple other sensors for landing such as a radio altimeter and an onboard instrument landing system (ILS) receiver. However, determining features of the landing location using techniques like edge detection to find an expected orientation and gradient of the landing markers can be a challenge when landing a vehicle in a target landing zone. These determinations can be hindered by a lot of false positives present alongside the required features. For example, detecting the edge of heliport markings is hindered significantly by the presence of similar angled lines around the landing pad. Likewise, detecting a runway edge is hindered by the presence of similar angled lines around the actual runway edges. These hinderances limit the usage of simple techniques for vision-based landing systems.

<NPL>, discloses a vision-aided automatic landing system with an optical navigation computer, a laser scanner, a CMOS camera, an IR camera, a radar altimeter, air data sensors and a GNSS receiver operatively coupled to the computer. The main goal of the computer vision unit is to find the outer edges of the runway expressed in image coordinates. A Gaussian smoothing algorithm is used to remove noise effects, followed by the use of a Canny Edge Detector and a two-dimensional orientation mask is created by convolving a linear edge detection function aligned normal to the edge direction with a projection function parallel to the edge direction. <NPL> discloses automated lane marking detection in road vehicles.

According to the present invention, there is provided a vision-based landing system as set out in claim <NUM> below. Optional features of the invention are set out in the dependent claims below.

In the following detailed description, embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It is to be understood that other embodiments may be utilized without departing from the scope of the invention. The following detailed description is, therefore, not to be taken in a limiting sense.

A system and method for adaptive feature extraction to detect letters and/or edges on vehicle landing surfaces is described herein. The present approach provides a way of extracting well defined structures and features from visual data, such as images of runway edges, helipad markings, landing pad markings, or the like. The visual data is captured using one or more onboard vehicle cameras.

The present method provides for estimating a vehicle pose relative to a plurality of known well-defined features on the ground. The present method reduces the number of detected candidates for a desired feature, and therefore increases integrity of the system by reducing the probability of a mismatch.

A significant challenge in doing vision-based pose estimation for an aerial vehicle is the accurate detection of corners and edges of a region of interest. Since these scenarios are used in well-defined conditions, such as well-marked runways and helipads, the present approach utilizes this information from a navigation database to improve detection of such edges. For example, a heliport landing area typically has clearly demarcated edges and threshold markings, with well-defined edge thicknesses and lengths.

In traditional edge detection methods, a conventional gradient filter is used to extract sharp gradients in an image. Gradient filters are usually unidirectional, which means one filter can only extract edges in a particular direction. Hence, multiple filters have to be applied to the same image over and over again, where each filter comprises a particular direction. This traditional approach does a good job of extracting all possible edges in the image, but fails at highlighting a desired edge from undesired edges, and needs multiple iterations of filters on the same image to yield the desired result. Therefore, additional filtering is required to extract required edges from a large set of candidates. The present approach provides such filtering and ensures that the well-defined knowledge of a region of interest is utilized effectively. In addition, using localization information from certified vehicle sensors and a navigation database makes it easier to prove the integrity of the system.

The present approach provides a visual navigational landing system for aerial vehicles, which includes a technique to adapt feature detection to any predefined shape or feature, such as aruco markers, runway edges, heliport markings, vertiport markings, or any other predefined landing markings. The present approach can be applied in the navigation of various aerial vehicles, including traditional fixed wing airplanes, helicopters, vertical take-off and landing (VTOL) vehicles such as hybrid VTOL vehicles, unmanned aerial vehicles (UAV), unmanned aircraft systems (UAS) vehicles for package delivery, air taxis such as urban air mobility (UAM) vehicles, or the like. The present approach does not require any ground augmentation apart from a distinct runway, strip, helipad, vertiport, or any other designated landing area, on which the aerial vehicle can land. The present method can be used in applications such as providing visual position estimates, sense and avoid, precision landing of aerial vehicles, aircraft visual docking at gates, and the like.

The present method uses other source of information such as a vehicle's coordinates and a navigation database to get an estimate of a desired edge's slope and length being detected. Moreover, the present method can detect letter marking edges, such as the "H" or "V" or any other known markings on any heliport or vertiport, for various vehicle landing applications.

The present approach provides richer information in the form of guidance and navigation cues that can be provided to a pilot. The present algorithm can also work in longer ranges (using zoom techniques), where visual acquisition by humans is not feasible.

In one example, the present method can enable a vision-based UAM landing from a height of about <NUM>-<NUM>, without any extra equipment on the ground. In another example, the present method can be used for an aircraft glideslope and localizer deviation estimation at distances of around <NUM> nautical miles from a runway aiming point without the ILS equipment physically being present on the runway.

Further details related to the present system and method are described as follows and with reference to the drawings.

<FIG> illustrates a vision-based landing system <NUM>, according to one embodiment, which employs adaptive feature extraction to detect edges on vehicle landing surfaces. The vision-based landing system <NUM> comprises at least one vision sensor <NUM> such as a camera, located onboard a vehicle <NUM> such as an aircraft, and at least one processor <NUM> onboard vehicle <NUM>. The processor <NUM> is operative to receive image data from vision sensor <NUM>. The processor <NUM> is in operative communication with a data storage unit <NUM>, which includes a navigation database <NUM> and a landing area database <NUM>. In one embodiment, data storage unit <NUM> is located onboard vehicle <NUM>. In other embodiments, information from navigation database <NUM> and landing area database <NUM> can be relayed to vehicle <NUM> via a communication link, such as when data storage unit <NUM> is located on the ground.

The processor <NUM> is also operatively coupled to one or more vehicle aiding sensors <NUM>, which can include one or more inertial sensors such as an inertial measurement unit (IMU), a global navigation satellite system (GNSS) receiver such as a global positioning system (GPS) receiver, a light detection and ranging (LiDAR) device, a radio detection and ranging (RADAR) device, or the like. The processor <NUM> hosts an adaptive feature extraction module <NUM>, which has processor readable instructions to perform a method to detect letters and edges on a landing surface for vehicle <NUM>.

<FIG> is flow diagram of the method performed by adaptive feature extraction module <NUM>, according to an exemplary implementation. The method comprises capturing at least one image of a landing area for vehicle <NUM> using vision sensor <NUM>, with the landing area including a plurality of edge features (block <NUM>). The method also includes calculating an estimated slope of an expected edge feature of the landing area (block <NUM>), and calculating an expected gradient direction of the expected edge feature (block <NUM>). The method also includes selecting a horizontal basis kernel for an expected edge feature of the landing area, along an expected horizontal gradient (block <NUM>), and selecting a vertical basis kernel for an expected edge feature of the landing area, along an expected vertical gradient (block <NUM>). The method calculates a combined convolution kernel for the expected edge feature, based on the horizontal and vertical basis kernels, the estimated slope of the expected edge feature, and the expected gradient direction of the expected edge feature (block <NUM>). The method then performs a convolution operation on the captured image using the combined convolution kernel to obtain a resulting edge feature image of the landing area (block <NUM>).

Additional details related to implementing the adaptive feature extraction method are described hereafter.

The adaptive feature extraction method can be implemented as a computer vision algorithm in various applications to detect edges and/or letters on a vehicle landing surface. For example, <FIG> is a schematic diagram of an aircraft runway <NUM>, for which the adaptive feature extraction method such as used by vision-based landing system <NUM> (<FIG>), can be employed to detect edges on the aircraft runway <NUM>. <FIG> is a schematic diagram of a helipad <NUM>, for which the adaptive feature extraction method such as used by vision-based landing system <NUM>, can be used to detect markings on helipad <NUM> (e.g., the letter "H"). <FIG> is a schematic diagram of a vertiport <NUM>, for which the adaptive feature extraction method such as used by vision-based landing system <NUM>, can be employed to detect markings on vertiport <NUM> (e.g., the letter "C").

<FIG> is flow diagram of a method <NUM> for adaptive feature extraction, according to an exemplary implementation. The method <NUM> captures an original image of a landing area with a vehicle camera (block <NUM>). Various features in the landing area can be detected with the camera, such as certain objects or markers, using edges, color profiles, heat maps (for infrared images), shapes and patterns.

The method <NUM> uses information from a navigation database <NUM>, which provides information about a vehicle landing area of interest such as a heliport or runway, including a position, orientation, dimensions, and marking features of the landing area. For example, information about the landing area can include latitude, longitude, altitude (LLA). The method <NUM> also uses information from one or more vehicle sensors <NUM>, which provide information about vehicle states, including the vehicle geodetic coordinates and the vehicle attitude such as from onboard inertial sensors. The information from navigation database <NUM> and vehicle sensors <NUM> is used to compute an estimated slope of an expected edge feature in the landing area (block <NUM>). In addition, method <NUM> uses information from a landing area feature database <NUM>, which provides information about various features in the landing area, such as a feature's coarse shape, color, gradient direction, and the like. The information from feature database <NUM> is used to compute an expected gradient direction of the expected edge feature (block <NUM>).

The method <NUM> selects a horizontal kernel for the expected edge feature of the landing area (block <NUM>), along an expected horizontal gradient. The method <NUM> also selects a vertical kernel for an expected edge feature of the landing area (block <NUM>), along an expected vertical gradient. The method <NUM> obtains a kernel along the gradient of the expected edge feature (block <NUM>) based on the horizontal and vertical kernels (from blocks <NUM> and <NUM>), the estimated slope of the expected edge feature (from block <NUM>), and the expected gradient direction of the expected edge feature (from block <NUM>). The method <NUM> then calculates a combined convolution kernel adapted to the expected gradient of the edge feature (block <NUM>).

The method <NUM> then performs a convolution operation (block <NUM>) on the captured image (from block <NUM>) using the combined convolution kernel (from block <NUM>) to obtain a resulting edge feature image. After similar convolution operations of the entire image, a filtered image with edge features extracted is produced (block <NUM>). The filtered image is then further processed to produce a final output for use by the vision navigation system of the vehicle.

<FIG> is flow diagram of a method <NUM> for adaptive feature extraction for use in landing a vehicle at a heliport, according to one example implementation. The method <NUM> initially captures an original image <NUM> of the heliport landing area, such by a vehicle camera. As shown in image <NUM>, a lower pixel intensity pavement is located near the heliport letter "H" feature at <NUM>. A higher pixel intensity pavement is located on the heliport letter "H" feature at <NUM>. A sharp gradient defines the edge of the heliport letter "H" feature at <NUM>. Based on the heliport's information from a database, and a location of the vehicle, original image <NUM> is narrowed down to a region of interest (ROI) <NUM>. An enlarged view of ROI <NUM>, depicting pixels in a zoomed in section is shown in a digital image <NUM>, which shows how an edge looks in a digital format. The "gradient" referred to here is the rate of change of pixel intensity while moving horizontally or vertically in an image. An edge is qualitatively defined by the sharp change in gradient.

The method <NUM> uses a few prerequisites from known sources, including information from a navigation database and vehicle sensors. The navigation database provides prior information about the heliport, including LLA of the landing area. The vehicle sensors provide information about the states of the vehicle, including the vehicle geodetic coordinates, and the vehicle attitude from onboard inertial sensors. This data is used to create a coarse prediction of the heliport, using projection geometry. The information from the navigation database and the vehicle sensors is used to compute an estimated slope of an expected edge feature. The information from a feature database is used to compute an expected gradient direction of the expected edge feature.

For example, edge features in a determined heliport trapezoid are computed from the known sources as indicated at <NUM>, which represents the edges of the letter H" feature in the projected coarse ROI created. The errors in the projection are bounded by the errors in the navigation database and the states of the vehicle. Therefore, in the image frame, the coarse projected heliport is defined by a set of lines whose parameters like slope and y intercept can be computed.

As shown in <FIG>, the cosine of the edge's slope from <NUM> is calculated at <NUM>, and the sine of the edge's slope from <NUM> is calculated at <NUM>. These parameters are calculated using the following steps. Initially, the heliport geodetic coordinates and current vehicle geodetic coordinates are converted to Earth-Centered Earth-Fixed (ECEF) coordinates. The ECEF coordinates of the heliport are then converted to vehicle relative North-East-Down (NED) coordinates (navigation frame). Using the attitude of the vehicle from the onboard inertial sensors, the heliport coordinates are converted to a camera frame. With the known intrinsic matrix for the respective camera, the heliport coordinates are converted from the camera frame to an image frame in terms of pixels. Using the start and end pixels representing a letter edge in the image frame, the cosine and sine of the edge's slope is calculated, using the expression: <MAT> In the cosine and sine of the edge's slope, an extra π/<NUM> is added to correct for the direction of gradient convention.

The gradient convention of elements as used herein is shown in <FIG>, which is a schematic depiction of the heliport landing area. The angle of a selected edge is shown at <NUM>, and the various arrows represent the direction of the gradient. The origin of the image coordinates (<NUM>, <NUM>) for the X and Y directions is also shown.

Further details on a method of using image frame transforms to produce the estimated slope of an expected edge feature and an expected gradient direction are described below with respect to <FIG>.

Returning to <FIG>, a simple horizontal kernel is computed based on the cosine of the edge's slope from <NUM>, and a simple vertical kernel is computed based on the sine of the edge's slope from <NUM>. The simple horizontal and vertical kernels are used as basis kernels. An example matrix <NUM> for the simple horizontal kernel is shown in <FIG>. A positive horizontal gradient is defined from left to right in matrix <NUM>, as represented by the increasing magnitude of values in matrix <NUM>. An example matrix <NUM> for the simple vertical kernel is also depicted. A positive vertical gradient is defined from top to bottom in matrix <NUM>, as represented by the increasing magnitude of values in matrix <NUM>. The magnitude of values in matrix <NUM> and matrix <NUM> can be changed according to an expected gradient strength and the edge's dilution in pixels.

A matrix <NUM> is then obtained that represents a combined convolution kernel for the determined edge. The matrix <NUM> is obtained after respectively multiplying the cosine and sine values from <NUM> and <NUM> with the horizontal and vertical basis kernels represented by matrices <NUM> and <NUM>. This can be expressed as: <MAT> where kij represents elements of the convolution kernel; Hij represents elements of the horizontal basis kernel; Vij represents elements of the vertical basis kernel; and the dot represents elementwise multiplication of the matrix.

A convolution operation <NUM> is then performed on digital image <NUM> using the combined convolution kernel represented by matrix <NUM>. The convolution operation <NUM> can be expressed as: <MAT> where g(x, y) represent the convolved image; k is the combined convolution kernel; and f(x,y) represents the original image. The elements of the convolution kernel are considered by the indices i and j, that is: -a ≤ i ≤ α and -b ≤ dy ≤ b , where α and b represent the size of the kernel as: <MAT>.

The convolution operation <NUM> produces a set of resulting image pixels <NUM>, as shown in <FIG>. After binarizing resulting image pixels <NUM>, the desired edge gets demarcated by white pixels (value = <NUM>) of thickness of <NUM>-<NUM> pixels. Both sides of the edge are set to black pixels (value = <NUM>).

A similar convolution operation is then performed on the entire original image, as indicated at <NUM>. A resulting image <NUM> is produced, with lines extracted, after the convolution operation. This can be followed by image morphology operations such as erosion and dilation. As shown in <FIG>, the desired edges can be distinguished in resulting image <NUM>. Thereafter, an operation to extract and combine the lines from resulting image <NUM> is performed, as indicted at <NUM>. This can be done using any common line detection/extraction method like Edge Drawing (ED) lines, Hough transform, or the like.

A final image output <NUM>, with all the desired edges and lines, is then produced by method <NUM>. The lines can be used in post processing to get outputs like ego position, relative position, and other geospatial estimates.

As mentioned above, <FIG> is a flow diagram of an exemplary method <NUM> of using image frame transforms to produce the estimated slope of an expected edge feature, and an expected gradient direction. Although method <NUM> is described with respect to a heliport, it should be understood that this method is applicable to other vehicle landing areas.

Initially, a heliport LLA <NUM> is obtained from a navigation database <NUM>, to provide heliport geodetic coordinates, and a vehicle LLA <NUM> is obtained from vehicle sensors <NUM>, to provide vehicle geodetic coordinates. A conversion operation is performed at <NUM> to convert the geodetic coordinates to ECEF coordinates, which results in heliport ECEF coordinates <NUM> and vehicle ECEF coordinates <NUM>. A conversion operation is then performed at <NUM> to convert the ECEF coordinates to vehicle relative NED coordinates, resulting in heliport coordinates in vehicle relative NED frame at <NUM>.

Using a vehicle attitude <NUM> from vehicle sensors <NUM>, a conversion operation is then performed at <NUM> to convert the relative NED frame to a vehicle body frame, resulting in heliport coordinates in vehicle body frame at <NUM>. A conversion operation is then performed at <NUM> to convert the body frame relative to the camera frame, resulting in heliport coordinates in camera frame at <NUM>.

Using a known camera intrinsic matrix from a database <NUM>, a conversion operation is then performed at <NUM> to convert the camera frame to an image frame, resulting in heliport coordinates in pixels in an image frame at <NUM>. Geometrical slope computations are then performed on the pixels in the image frame at <NUM>, and an estimated slope of the expected edge is output at <NUM>.

As shown in <FIG>, method <NUM> also uses information from a feature database <NUM>, which provides information about various features in the heliport landing area. Such features include a feature's coarse shape, color information, gradient direction, and the like, as indicated at <NUM>. These features are used to compute an expected gradient direction of the expected edge at <NUM>.

<FIG> show the results of the present approach as applied in a simulation, for visual assisted landing of a vehicle on a helipad. <FIG> depicts an original image <NUM> of the helipad captured by a vehicle camera. <FIG> shows a processed image <NUM> based on original image <NUM>, with the left edges of the helipad features extracted after application of a kernel. <FIG> shows a processed image <NUM> based on original image <NUM>, with the right edges of the helipad features extracted after application of the kernel.

<FIG> show the results of the present approach as applied in a simulation. In particular, <FIG> depicts an original image <NUM> of an aircraft runway. <FIG> shows an ROI and synthetic projection <NUM> overlaid on original image <NUM>, which is detected using the navigation database, ownship position, and camera parameters. <FIG> shows a processed image <NUM> of a right edge kernel based on the estimated slope (as per <NUM> in <FIG>) of the runway edges from synthetic projection <NUM>. <FIG> depicts a processed image <NUM> with right edge extraction after application of the right edge kernel. <FIG> shows an output image <NUM> with the combined edge detections, and <FIG> depicts a trimmed final output image <NUM>, which can be employed in various post processing navigation operations.

The processing units and/or other computational devices used in the method and system described herein may be implemented using software, firmware, hardware, or appropriate combinations thereof. The processing unit and/or other computational devices may be supplemented by, or incorporated in, specially-designed application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, the processing unit and/or other computational devices may communicate through an additional transceiver with other computing devices outside of the system, such as those associated with a management system or computing devices associated with other subsystems controlled by the management system. The processing unit and/or other computational devices can also include or function with software programs, firmware, or other computer readable instructions for carrying out various process tasks, calculations, and control functions used in the methods and systems described herein.

The methods described herein may be implemented by computer executable instructions, such as program modules or components, which are executed by at least one processor or processing unit. Generally, program modules include routines, programs, objects, data components, data structures, algorithms, and the like, which perform particular tasks or implement particular abstract data types.

Instructions for carrying out the various process tasks, calculations, and generation of other data used in the operation of the methods described herein can be implemented in software, firmware, or other computer readable instructions. These instructions are typically stored on appropriate computer program products that include computer readable media used for storage of computer readable instructions or data structures. Such a computer readable medium may be available media that can be accessed by a general purpose or special purpose computer or processor, or any programmable logic device.

Suitable computer readable storage media may include, for example, non-volatile memory devices including semi-conductor memory devices such as Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory devices; magnetic disks such as internal hard disks or removable disks; optical storage devices such as compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs; or any other media that can be used to carry or store desired program code in the form of computer executable instructions or data structures.

Claim 1:
A vision-based landing system, comprising:
at least one processor configured for being arranged onboard an aerial vehicle,
at least one vision sensor configured for being arranged onboard the aerial vehicle and operatively coupled to the at least one processor,
one or more aiding sensors configured to provide information about the geodetic coordinates and attitude of the aerial vehicle, the one or more aiding sensors being configured for being arranged onboard the aerial vehicle and operatively coupled to the at least one processor; and
a data storage unit in operative communication with the at least one processor;
wherein the at least one processor hosts an adaptive feature extraction module, which has processor readable instructions to perform a method to detect edges on a landing area for the aerial vehicle, the method comprising:
capturing at least one image of the landing area with the at least one vision sensor, wherein the landing area includes a plurality of edge features;
calculating an estimated slope of an expected edge feature of the landing area;
calculating an expected gradient direction of the expected edge feature;
selecting a horizontal basis kernel for the expected edge feature along an expected horizontal gradient;
selecting a vertical basis kernel for the expected edge feature along an expected vertical gradient;
calculating a combined convolution kernel for the expected edge feature based on the horizontal and vertical basis kernels, the estimated slope of the expected edge feature, and the expected gradient direction of the expected edge feature; and
performing a convolution operation on the at least one image using the combined convolution kernel to obtain a resulting edge feature image of the landing area,
wherein the data storage unit includes a navigation database and a landing area feature database and the navigation database is configured to provide information about the landing area, including a position, orientation and dimensions of the landing area,
characterised in that the navigation database is additionally configured to provide information about marking features of the landing area and in that the information from the navigation database and the one or more aiding sensors is used to calculate the estimated slope of the expected edge feature.