Vision-Based Aircraft Landing Aid

The present invention discloses a vision-based aircraft landing aid. During landing, it acquires a sequence of raw runway images. The raw runway image is first corrected for the roll angle (γ). The altitude (A) can be calculated based on the runway width (W) and the properties related to both extended runway edges on the rotated (γ-rotated) runway images. Smart-phone is most suitable for vision-based landing aid.

It should be noted that all the drawings are schematic and not drawn to scale. Relative dimensions and proportions of parts of the device structures in the figures have been shown exaggerated or reduced in size for the sake of clarity and convenience in the drawings. The same reference symbols are generally used to refer to corresponding or similar features in the different embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Those of ordinary skills in the art will realize that the following description of the present invention is illustrative only and is not intended to be in any way limiting. Other embodiments of the invention will readily suggest themselves to such skilled persons from an examination of the within disclosure.

Referring now toFIG. 1, an aircraft10with a preferred vision-based landing aid20is disclosed. The vision-based landing aid20is mounted behind the wind-shield of the aircraft10and faces forward. It could be a camera, a computer-like device with camera function, or a cellular phone such as a smart-phone. The principal point of its optics is denoted O′. This landing aid20measures its altitude A to the ground0using computer vision. A runway100is located in the front on the ground0. Its length is L and width is W. A ground frame is defined as follows: its origin o is the projection of O′ on the ground0, its x axis is parallel to the longitudinal axis of the runway100, its y axis is parallel to the lateral axis of the runway100, and its z axis is perpendicular to its x-y plane. The z axis, uniquely defined by the runway surface, is used as a common reference in many frames (coordinate systems) of the present invention.

Referring now toFIGS. 2A-2C, three preferred vision-based landing aid20are disclosed. The preferred embodiment ofFIG. 2Acomprises a camera30and a processor70. It calculates altitude A using the runway width W and the image acquired by the camera30. The runway width W can be manually input with information obtained from the Airport Directory. It may also be retrieved electronically from an airport database. The vision-based landing aid can measure altitude, predict future altitude based on measured data and provide visual/audible instructions to a pilot before decision point. For example, two seconds before a landing maneuver (e.g. flare or a pre-touchdown maneuver), two short beeps and a long beep are generated. The pilot is instructed to ready themselves for the maneuver at the first two short beeps and initiate the maneuver at the last long beep.

Compared withFIG. 2A, the preferred embodiment ofFIG. 2Bfurther comprises a sensor40, e.g. an inertia sensor (e.g. a gyroscope) and/or a magnetic sensor (i.e. a magnetometer), which measures the orientation angles (ρ, α, γ). The altitude calculation is simplified by using these orientation angles. For example, the measured y can be directly used to rotate the raw runway image; the measured ρ and α can be directly used to calculate altitude (referring toFIG. 8). Using the sensor data reduces the workload of the processor and can expedite image processing.

The preferred embodiment ofFIG. 2Cis a smart-phone80. It further comprises a memory50, which stores a landing application software (app)60. By running the landing app60, the smart-phone80can measure altitude, predict future altitude and provide instructions to a pilot before decision point. With the ubiquity of the smart-phones, vision-based landing aid can be realized without adding new hardware, but simply by installing a “Landing Aid” app in the smart-phone. This software solution has the lowest cost.

Referring now toFIGS. 3-5, a method to extract the roll angle (γ) on the captured image is described. InFIG. 3, the roll angle (γ) of the camera30is defined. Because the image detector32(e.g. CCD sensor or CMOS sensor) of the camera30is rectangular in an imaging plane36, a raw image frame can be easily defined: its origin O is the principal point of the detector32, and its X, Y axis are the center lines of the rectangle with its Z axis perpendicular to the X-Y plane. Here, a line of cord N is defined as the line perpendicular to both z and Z axis and it is always parallel to the runway surface. The roll angle (γ) is defined as the angle between the Y axis and the line N. A rotated (γ-corrected) image frame X*Y*Z* is defined as the image frame XYZ rotated around the Z axis by −γ. Here, the line N is also the Y* axis of the rotated image frame.

FIG. 4is a raw runway image100iacquired by the camera30. Because the roll angle of the camera30is γ, the image120iof the horizon is tilted. It has an angle γ with the Y axis. The raw runway image100iis γ-corrected by rotating it around its principal point O by −γ.FIG. 5is the rotated (γ-corrected) runway image100*. The image120* of its horizon is now horizontal, i.e. parallel to the Y* axis. On the rotated runway image, the horizontal line (i.e. Y* axis) passing its principal point O is referred to as the principal horizontal line H and the vertical line (i.e. X* axis) passing its principal point O is referred to as the principal vertical line V. The rotated runway image100* will be further analyzed inFIGS. 6-8.

Referring now toFIG. 6, the pitch angle (ρ) of the camera30is defined. An optics frame X′Y′Z′ is defined by translating the rotated image frame X*Y*Z* by a distance of f along the Z* axis. Here, f is the focal distance of the optics38. Then a rotated (α-corrected, referring toFIG. 7) ground frame x*y*z* is defined. Its origin o* and z* axis is same as the ground frame xyz, while its x* axis is in the same plane as the X′ axis. The distance of the principal point of the optics O′ to the ground (i.e. origin o*) is the altitude A. The pitch angle (ρ) is the angle between the Z′ axis and the x* axis. For a point R on the ground0with coordinate (x*, y*,0) (in the rotated ground frame x*y*z*), the coordinates (X*, Y*,0) of its image on the image sensor32(in the rotated image frame X*Y*Z*) can be expressed as: δ=ρ−atan(A/x*); X*=−f*tan(b); Y*=f*y*/sqrt(x*̂2+Â2)/cos(δ).

Referring now toFIG. 7, the yaw angle (α) of the camera30is defined. This figure shows both the ground frame xyz and the rotated (α-corrected) ground frame x*y*z*. They differ by a rotation of α around the z-axis. Note that α is in reference to the longitudinal axis of the runway100. Although the x axis is parallel to the longitudinal axis of the runway100, the rotated ground frame x*y*z* is more computationally efficient and therefore, is used in the present invention to analyze the runway image.

Referring now toFIG. 8, the steps to perform the altitude measurement is disclosed. First of all, the roll angle γ is extracted from the horizon120iof the raw runway image100i(FIG. 4, step210). After obtaining γ, the raw runway image100iis γ-corrected by rotating about its principal point by −γ (FIG. 5, step220). On the rotated runway image100*, the intersection of the extended left and right runway edges160*,180* is denoted by P. Its coordinates (XP, YP) (XPis the distance between the intersection P and the principal horizontal line H; YPis the distance between the intersection P and the principal vertical line V) can be expressed by: XP=f*tan(ρ); YP=f*tan(α)/cos(ρ). Consequently, the pitch angle ρ can be extracted (FIG. 5, step230), i.e. ρ=atan(XP/f); and the yaw angle α can be extracted (FIG. 5, step240), i.e. α=atan[(YP/f)*cos(ρ)].

Finally, the distance Δ between the intersections A, B of both extended runway edges160*,180* and the principal horizontal line H is used to extract altitude A (FIG. 5, step250), i.e. A=W*sin(ρ)/cos(α)/(Δ/f). Alternatively, the angles θA, θBbetween both extended runway edges160*,180* and the principal horizontal line H can also be used to extract altitude A, i.e. A=W*cos(ρ)/cos(α)/[cot(θA)−cot(θB)].

It should be apparent to those skilled in the art, the steps inFIG. 8can change order or be skipped. For example, when the sensor40is used to measure orientation angles (ρ, α, γ), the measured γ can be directly used to rotate the raw runway image (skip the step210); the measured ρ and α can be directly used to calculate altitude (skip the steps230,240). Using the sensor data reduces the workload of the processor and can expedite image processing.

Referring now toFIGS. 9A-9B, a preferred gravity-oriented landing aid20is disclosed. It keeps the horizon in the raw runway image horizontal. As a result, the raw runway image does not need to be γ-corrected, which simplifies the altitude calculation. To be more specific, the landing-aid20(e.g. a smart-phone) is placed in a gravity-oriented unit19, which comprises a cradle18, a weight14and a landing-aid holder12. The cradle18is supported by ball bearings16on support17, which is fixed mounted in the aircraft10. This makes the cradle18move freely on the support17. The weight14ensures that the landing aid20(e.g. one axis of the image sensor32) is always oriented along the direction of gravity z, no matter the aircraft10is in a horizontal position (FIG. 9A) or has a pitch angle ρ (FIG. 9B). The weight14preferably contains metallic materials, and forms a pair of dampers with the magnets15. These dampers help to stabilize the cradle18.

While illustrative embodiments have been shown and described, it would be apparent to those skilled in the art that may more modifications than that have been mentioned above are possible without departing from the inventive concepts set forth therein. For example, although the illustrative embodiments are fixed-wing aircrafts, the invention can be easily extended to rotary-wing aircrafts such as helicopters. Besides manned aircrafts, the present invention can be used in unmanned aerial vehicles (UAV). The invention, therefore, is not to be limited except in the spirit of the appended claims.