Patent Description:
Landing an aircraft is one of the most demanding maneuvers performed during flight. When landing an aircraft, a pilot needs to make a decision of "continue approach" or "go-around" at a minimum decision height. This can be a challenging task and the pilot is required to make the decision within a short time period. In making the decision as to whether to continue the approach or to go-around, the pilot should be able to clearly see the landing environment for the approaching runway. The pilot should also check other conditions in order to continue landing, such as lateral and glideslope deviation, and must maintain with a range to complete the landing task. If the aircraft flies too high or too low, or if the aircraft is not aligned with a center of a target runway, a go-around should be considered even if the pilot is able to make visual contact with the runway. Whilst pilots are trained to consider the various factors when deciding upon a go-around, it remains a matter of pilot judgement as to whether the aircraft is too high or too low, too left or too right for landing tasks with visual confirmation requirements. The effectiveness of a pilot in safely landing the aircraft depends on the experience and judgment of the pilot. Pilots with varying levels of experience and training may respond differently to the same situation, and some pilot responses may provide a less than optimal landing. Further, the decision to go-around or continue a landing approach places a cognitive burden on the pilot during an already strenuous flight sector.

Accordingly, an object of the present disclosure is to provide a landing aid for an aircraft and associated methods using processing efficient software and existing hardware for many aircraft. The landing aid assists a pilot in deciding whether to go-around or continue an approach whilst ensuring compliance with a requirement for visual contact with the target runway. The landing aid may make the go-around or continue decision faster and better than many human pilots. Furthermore, other desirable features and characteristics of the disclosure will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure. Relevant prior art is known from documents <CIT>, and <CIT>.

The present disclosure provides computer vision systems and methods for an aircraft. The computer vision systems and methods identify a runway based on video frame data from a camera mounted to the aircraft, extract runway pixel positions associated with the runway, determine the aircraft position in a real world coordinate frame based on the pixel positions, receive predefined aircraft position data and go-around rules for a compliant approach to the runway, calculate an aircraft position deviation based on the aircraft position and the predefined aircraft position data, determine whether the aircraft position deviation is in conformance with the go-around rules, and output an indication of conformity with the go-around rules, receive reference aircraft position data from an aircraft sensor system; cross-check the aircraft position with the reference aircraft position data; and when the cross-check indicates a discrepancy beyond a threshold, outputting a warning and discontinuing determining whether the aircraft position deviation is in conformance with the go-around rules based on the video frame data. , wherein the reference aircraft position data is received from a global positioning system.

This summary is provided to describe selected concepts in a simplified form that are further described in the Detailed Description.

The present disclosure will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and wherein:.

All of the embodiments described herein are exemplary embodiments provided to enable persons skilled in the art to make or use the invention and not to limit the scope of the invention, which is defined by the claims.

Methods described herein include a camera mounted to an aircraft that outputs video frame data to a computer vision processor. The computer vision processor identifies a position of the runway in image space and determines a transformation function to the runway position in real world space based on data obtained from a navigation database. The transformation function can then be applied to the runway position in real world space to determine the camera position in real space, which can be used to determine the aircraft position. Using the determined aircraft position, a continue approach or go-around decision is made. The decision is additionally based on predefined go-around rules and predefined aircraft position data that is determined based on the planned aircraft path to the target runway.

The present disclosure provides systems and methods using a camera (e.g. a monocular camera) and a computer vision processor to compute aircraft a position (and optionally also speed) vector relative to a target runway in real-time. The systems and methods further compute aircraft position deviation from a current planned flight path, which provides predefined aircraft position data. In an Instrument Landing System (ILS) approach, the planned flight path is a predefined GS (Glide Scope) and LOC (Localizer) path. In a Localizer Performance with Vertical guidance (LPV) approach, there is a predefined virtual glide path to a target runway control point. The systems and methods compare the calculated aircraft position from predefined aircraft position. The systems and methods announce an indication of conformity with go-around rules to the pilot, which can be a final decision as to whether to continue approach or to go-around. The systems and methods include a process for cross checking the determined aircraft position (e. g latitude, longitude and altitude) and optional speed vector (both horizontal and vertical) calculated by the computer vision processor against data coming out of aircraft sensors such as a global positioning system (GPS).

The systems and methods described herein can provide a more accurate go-around decision in that computer/machine decision making based on computer vision processing and a predefined algorithm can be more accurate and consistent than a human decision based on pilot experiences. The integrity of the decision is assured as it relies on visual contact of the camera with the runway and may further include a cross-check feature. If the cross-check output is negative, the systems and methods will declare failure and not generate further guidance and annunciations. The computer vision algorithm can complete object detection (runway detection in this case) and other required calculations for positioning and decision making much faster than a human decision could usually be made.

<FIG> depicts an exemplary embodiment of a computer vision system <NUM> associated with, or included within, an aircraft <NUM>. This embodiment of the system <NUM> includes, without limitation, a camera <NUM>, a navigation/airport database <NUM>, a processing system <NUM>, an output system <NUM>, aircraft sensors <NUM>, a computer vision processing module <NUM>, a global positioning system (GPS) module <NUM>, an aircraft position deviation module <NUM>, go-around rules <NUM>, a go-around decision module <NUM> and a cross-check module <NUM>. It should be understood that <FIG> is a simplified representation of the computer vision system <NUM>, and <FIG> is not intended to limit the application or scope of the subject matter in any way. In practice, the system <NUM> and/or the aircraft <NUM> will include numerous other devices and components for providing additional functions and features, as will be appreciated in the art.

In some embodiments, the aircraft <NUM> is a fixed wing aircraft or a rotorcraft or a UAV with remote piloting data link. The aircraft may be a passenger or cargo aircraft and can be any of various dimensions and weight capacities. The aircraft <NUM> has a camera <NUM> mounted thereon. In the depicted embodiment, the camera <NUM> is mounted to the wing of the aircraft <NUM>, but embodiments are contemplated in which the camera <NUM> is mounted to the fuselage of the aircraft <NUM>. Rather than being laterally displaced as in the exemplary embodiment, the camera <NUM> may be located along a central longitudinal axis of the aircraft <NUM>.

In embodiments, the camera <NUM> is a monocular digital camera, although other types of camera <NUM> are contemplated. Other vision cameras are possible including stereo cameras, infrared cameras, etc.. The camera <NUM> can include a plurality of cameras including combinations of different types of cameras such as those described above. The images are generated via single camera or real-time fusion of multiple cameras. The resolution of the camera may be no less than WUXGA resolution (1920x1080 pixels) or application dependent for pixel feature extractions The camera <NUM> is mounted on the front of aircraft <NUM> so as to provide a front view of a scene ahead of the aircraft <NUM> that will encompass a target runway at a decision height for the aircraft <NUM>. The camera <NUM> may be mounted on a center of the front of a nose of the aircraft or may be laterally offset. As will be described further herein, any lateral offset can be factored in by the computer vision processing module <NUM>. In one exemplary embodiment, the camera <NUM> has a <NUM> lens, which translates to a field of view of <NUM> (deg) and <NUM> (deg) on X and Y axis, respectively, for an FX formatted image sensor (<NUM> x <NUM>). In the exemplary embodiment, the camera <NUM> is mounted with a certain range of pitch angle measured with respect to the aircraft body longitudinal axis. Given the aircraft gliding path is often <NUM> degrees, a normal pitch angle of -<NUM> degrees of the aircraft during approach, a vertical FOV of the <NUM> lens is <NUM> degrees and the camera mounting pitch angle should be <NUM> or <NUM> degrees.

In embodiments, the camera <NUM> produces video frame data <NUM> that can include a target runway. The format of the video frame data <NUM> could be <NUM> bits depth grayscale data with a resolution of 1920x1080. However, greater sensitivity (bits) or color data could also be provided by the camera <NUM>. The video frame data <NUM> is provided to the computer vision processing module <NUM> for processing to identify the target runway and from that to determine a real world position of the aircraft <NUM>.

In accordance with various embodiments, the computer vision system <NUM> includes a navigation database <NUM> that stores navigation data used by various avionics systems. The navigation database <NUM> provides data <NUM> for the real world position of a target runway. The target runway may be established by a flight plan from a flight management system (FMS) or selected by a pilot or derived based on a location of the aircraft <NUM> from the GPS module <NUM>. The navigation database <NUM> can be interrogated to determine data points representing the target runway in a real world coordinate frame. The navigation database <NUM> may, alternatively, be a dedicated airport runway database. The navigation database <NUM> may be stored digitally in an airborne avionics computer. For example, many modern aircraft <NUM> have an FMS and a navigation database that contains runway location data. The navigation database may be updated regularly through wired or wireless data connections. The data <NUM> provided by the navigation database <NUM> may include a runway centerline, a length of the runway, a width of the runway, coordinates for corner points of the runway, coordinates for a center point of the runway and other data describing dimensions and location of the runway. The coordinates may be relative to the runway or in a global coordinate frame, both of which are in a real world coordinate frame rather than an image space coordinate frame. The data <NUM> can be combined with corresponding data in image space extracted by the computer vision processing module <NUM> in order to determine a translation matrix that is used in subsequent processing to derive the aircraft position in the real world coordinate frame.

In accordance with various embodiments, the computer vision system <NUM> includes the aircraft sensors <NUM> and the GPS module <NUM>. The aircraft sensors <NUM> include at least an inertial measurement unit (IMU). The GPS module <NUM> may include a Global Positioning System (GPS) or global navigation satellite system (GNSS) receiver. The IMU includes one or more gyroscopes and accelerometers. The aircraft sensors <NUM> and the GPS module <NUM> individually or in combination determines location (and optionally also orientation information) for the aircraft <NUM>. The GPS module <NUM> determines location data based on global position data obtained from satellites, e.g. by trilateration with three or more satellites and based on inertial measurements. In some embodiments, the GPS module <NUM> determines location of the aircraft <NUM> based on Wide Area Augmentation System (WAAS) or other augmented satellite-based global position data. A network of ground-based reference stations provides measurements of small variations in the GPS satellites' signals so that onboard GPS or GNSS receivers use the corrections while computing their positions to improve accuracy of location measurements. The IMU of the aircraft sensors <NUM> may additionally or alternatively allow the global position and orientation of the aircraft to be derived. The aircraft sensors <NUM> and/or the GPS module <NUM> outputs reference aircraft position data <NUM>, which can be used in a cross-check process to determine whether the real world aircraft position <NUM> determined by the computer vision processing module <NUM> is reasonable.

In an exemplary embodiment, the computer vision system <NUM> includes an output system <NUM> including at least one of an aural annunciator <NUM> and a display device <NUM>. The output system <NUM> receives a go-around decision <NUM> and indicates the go-around decision <NUM> to the pilot. The indication may be an aural annunciation such as a tone or a text/command to speech output. Additionally or alternatively, a graphical or textual indication of the go-around decision <NUM> may be displayed on the display device <NUM>. The indication may be output for only a decision to go-around, for only a decision to continue approach or different indications may be output to indicate both decisions. In another possibility, an indicator light may be included on an instrument panel of the aircraft <NUM>. The aural annunciator <NUM> and the display device <NUM> are located in a cockpit of the aircraft <NUM>. In embodiments, the display device <NUM> includes a head down display (HDD), a head up display (HUD), a wearable HUD, a portable display or any combination thereof. The display device <NUM> may be dashboard/cockpit integrated display or an Electronic Flight Bag (EFB) device. The aural annunciator may include a cockpit integrated speaker or may be included in a headset of the pilot.

System <NUM> further includes the processing system <NUM> including one or more processors that are configured to execute computer programming instructions stored on non-transitory memory (not shown). Functions of the computer vision system <NUM> and steps of the method <NUM> (<FIG>) are executed by one or more processors of the processing system <NUM> and the associated computer programming instructions. Modules, sub-modules and the processing system <NUM> as described herein refer to any hardware, software, firmware, electronic control component, processing logic, and/or processor device, individually or in any combination, including without limitation: application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The computer vision processing module <NUM> includes a runway identification sub-module <NUM>, a translation matrix calculation sub-module <NUM>, an aircraft position calculation sub-module and a computer vision processor <NUM>. The computer vision processor <NUM> may be a dedicated processor adapted for image processing operations or the computer vision processor <NUM> may be a general purpose processor included in the processing system <NUM>. The computer vision processor <NUM> executes programming instructions that cause the at least one processor to execute the functions and steps of the computer vision processing module <NUM> as described herein.

The runway identification sub-module <NUM> processes the video frame data <NUM> and executes an object finding function or algorithm tuned to identify a runway in the video frame data <NUM>. Numerous runway detection algorithms could be constructed. In one example, a machine learning classifier is used that is trained with images including a runway. One example classifier is the Cascade Classifier. However, it should be appreciated that a classical image processing approach could also be used, which does not rely on machine learning algorithms. For example, feature or pattern matching algorithms could be used to identify the runway. The object finding function returns a positive or negative indication concerning whether a runway has been found and returns a region of interest (ROI) outline (e.g. a rectangle) encompassing the identified runway. The ROI may be a little larger and contain the convex hull of the runway. The runway identification sub-module <NUM> further processes, e.g. by contrast filtering and/or edge detection, the ROI to determine runway pixel positions (or other location data in image space) <NUM> describing the location of the runway in image space. In one embodiment, three or four corner points of the runway in image or pixel coordinates are output by the runway identification sub-module <NUM>.

The translation matrix calculation sub-module <NUM> receives the pixel positions <NUM> in two dimensional image coordinates (e.g. x and y). The translation matrix calculation sub-module <NUM> determines a translation matrix <NUM> for transforming the position the position of the runway in real world coordinates to a position of the camera <NUM> in real world coordinates. The translation matrix <NUM> is determined based on intrinsic parameters of the camera <NUM>, the pixel positions <NUM> and the data <NUM> for real world position of the runway. The translation matrix <NUM> represents a transformation from camera coordinates to real world coordinates. If the translation matrix <NUM> is known, it is possible to determine the position of the camera (and thus the real world aircraft position <NUM>) from the data <NUM>.

In one embodiment, the translation matrix calculation sub-module <NUM> solves the following equation: <MAT>.

In equation <NUM>, M is the camera intrinsic matrix including focus length fx, fy and center of image offset cx, cy: <MAT>.

In equation <NUM>, W = (R, t), wherein R is a rotation matrix (e.g. a <NUM> by <NUM> rotation matrix in Rodrigues transform format) and t is the translation matrix <NUM>. Q is the data <NUM> for real world position of the runway as a three dimensional space vector (X, Y, Z) in the form of a <NUM> by <NUM> matrix. The data <NUM> can be in the global coordinates, in local airport coordinates or in coordinates relative to the runway. The data <NUM> may be three or four corners of the runway in some embodiments but other trackable features of the runway may be used provided the location of corresponding features can be extracted by the computer vision processing module <NUM>. In equation <NUM>, q are the pixel positions <NUM> as a <NUM> by <NUM> matrix representing the computer vision determined points of the runway in camera body coordinates.

The translation matrix calculation sub-module <NUM> solves equation <NUM> with respect to the translation matrix <NUM> based on the following operation: <MAT>.

A detailed example of the operations by translation matrix calculation sub-module <NUM> will now be provided. Given M (3x3 Camera intrinsic Matrix), q (2x1 pixel positions <NUM> on the camera <NUM>), Q (3x1 runway position of the pixel in runway/real world coordinates) and R (rotation matrix of the camera <NUM>), the t matrix is solved based on intrinsic camera parameters, the pixel positions <NUM> for the runway determined by computer vision processing of the video frame data <NUM> and the known position of the runway based on the data <NUM> for the real world position of the runway.

For a <NUM> lens of an FX format image sensor (<NUM> * <NUM>), the camera intrinsic matrix M is, in the present example: <MAT>.

For Q, the runway threshold position is taken for the position of the runway and the length of the runway is retrieved from the navigation database <NUM>. In this example: <MAT>.

For R, it is assumed that the aircraft <NUM> is aligned with the runway and the camera <NUM> has its light axis (longitudinal x axis) parallel with a longitudinal axis of the body of the aircraft <NUM>, has a downward pitch (including the pitch of the aircraft <NUM>) of <NUM>° and there is no rotation on any other axis. In this example: <MAT>.

Equation <NUM> can thus be solved based on the pixel positions <NUM> coming from the runway identification sub-module <NUM> to derive t. In this simplified example, the pixel positions correspond to the runway threshold point in the camera image in x, y form. However, in more complicated examples, the R matrix is also unknown such that additional runway points are required to solve both R and t. Optimally, at least <NUM> non-relative points (not on a straight line) are included in the pixel positions. In embodiments described herein, three or four corner points for the runway are used to solve for R and t based on corresponding corner points for the runway included in the data <NUM> taken from the navigation database <NUM>.

The translation matrix <NUM> is received by the aircraft position calculation sub-module <NUM> and processed to derive a position of the camera <NUM> and, based thereon, the real world aircraft position <NUM>. The camera position is calculated based on the translation matrix <NUM> as follows: <MAT>.

The aircraft position can be determined based on the camera position by taking into account any lateral, and optionally also, height differential between the camera <NUM> and a center point of the aircraft <NUM>. Specifically: <MAT>.

tinstallation is a matrix representing spatial offset of the aircraft <NUM> from lateral, and optionally also, longitudinal and height axes of the aircraft <NUM>. The aircraft position calculation sub-module <NUM> may also determine a speed vector <NUM> for the aircraft based on a time derivative of the real world aircraft position <NUM>. The speed vector <NUM> can be used as an additional variable for the cross-checking process described in the following.

The computer vision system <NUM> includes an aircraft position deviation module <NUM> that received a planned flight path <NUM> and the real world aircraft position <NUM>. The planned flight path <NUM> can be obtained from ILS data or from an FMS. The planned flight path <NUM> includes a desired glide slope and a desired position laterally with respect to the target runway. The planned flight path <NUM> thus provides predefined aircraft position data. The aircraft position deviation module <NUM> compares the real world aircraft position <NUM> and the planned flight path <NUM> and determines a deviation therebetween in the form of an aircraft position deviation <NUM>. More specifically, the aircraft position deviation module <NUM> determines an aircraft position deviation with respect to the Glide Slope (GS) and the Localizer (LOC) in the planned flight path <NUM>. The glide slope deviation DGS is: <MAT>.

In equation <NUM>, ZPOS is the Z value in the matrix of the real world aircraft position <NUM>. ZGS is determined based on the glide slope angle (θGS) from the planned flight path <NUM>, namely: <MAT>.

The localizer or lateral deviation (DLOC) is determined based on: <MAT>.

In equation <NUM>, XPOS is the X value in the matrix of real world aircraft position <NUM> and XLOC is taken from the planned flight path <NUM>.

The computer vision system <NUM> includes a go-around decision module <NUM> that references go-around rules <NUM> to issue a decision <NUM> as to whether the aircraft <NUM> should continue the landing approach or to go-around. The go-around decision module <NUM> may become active when approaching a minimum decision altitude (MDA), e.g. within a predetermined distance or time from the MDA or after descending past a fixed or variably calculated altitude during a landing approach. Alternatively, the go-around decision module <NUM> may be active throughout a landing approach section of a flight plan or during a defined portion thereof. The go-around decision module <NUM> may cease continues operation of the processes described herein at/after descending through the MDA, e.g. within a predetermined distance or time after passing the MDA and before touchdown. Alternatively, the go-around decision module <NUM> may cease operation at a fixed or variably calculated height. The MDA varies depending on the category of the aircraft <NUM> and some other factors and is available in reference information accessible by the FMS. For CAT <NUM> aircraft, the MDA is greater than or equal to <NUM> feet (<NUM>). For CAT II aircraft, the MDA is greater than or equal to <NUM> feet. For CAT III aircraft, the MDA may be less than <NUM> feet or there may be no MDA. The go-around rules <NUM> may be defined by an aviation authority, by the aircraft carrier or by a system provider. The go-around rules <NUM> define maximum deviations from the planned flight path <NUM>. The go-around decision module <NUM> compares the go-around rules <NUM> with the aircraft position deviation <NUM> to determine conformance with the rules. When there is a lack of conformance, a decision <NUM> is output to go-around. When the rules are complied with, a decision <NUM> is output to continue the approach (or there is an absence of a decision <NUM> to go-around). The go-around rules <NUM> are stored on memory of the computer vision system <NUM>. In one example, the go-around rules are taken from the International Civil Aviation Organization (ICAO) Standards and Recommended Practices (SARPs) and are as follows:.

The computer vision system <NUM> includes a cross-check module <NUM> for verifying whether the real world aircraft position <NUM> is sufficiently reasonable by comparing it with corresponding reference aircraft position data <NUM> from the aircraft sensors <NUM> and/or from the GPS module <NUM>. In some examples, the speed vector <NUM> from the computer vision processing module <NUM> is compared to reference speed data from the aircraft sensors <NUM> and/or the GPS module <NUM> as an additional cross-check parameter. The cross-check module <NUM> determines whether the real world aircraft position <NUM> and optionally also the speed vector <NUM> are sufficiently close to the reference data based on thresholds. When the thresholds are breached, the real world aircraft position <NUM> is considered unreliable. When the real world aircraft position <NUM> is determined to be unreliable, the go-around decision module <NUM> ceases to operate and the pilot is warned that the automatic go-around decision process is no longer operational. The warning is provided through aural and/or visual indication by the output system <NUM>. Example thresholds for the difference between the real word aircraft position and speed data <NUM>, <NUM> and the reference data <NUM> are provided in table <NUM> below. When the computer vision values differ from the reference values by an amount within the thresholds, the cross-check module <NUM> does not interrupt the go-around decision module <NUM> and the automated go-around decision processes described herein continue.

Whilst the computer vision system <NUM> is described herein as being located in the aircraft <NUM>, it should be understood that remote or cloud processing capabilities and data sources could be used by the aircraft <NUM> to implement at least some parts of the computer vision system <NUM>. As such, one or more components described with respect to the aircraft <NUM> may not necessarily be located onboard the aircraft <NUM>.

<FIG> is a flowchart illustrating an exemplary method <NUM> for providing a go-around or continue approach decision <NUM>. The decision <NUM> is based on computer vision processing of video frame data <NUM> to identify the runway and the location of the runway in camera coordinates. Based on a relation between the location of the identified runway in camera coordinates and the real world position of the runway <NUM>, the real word aircraft position <NUM> can be derived from the computer vision processing. The aircraft position deviation <NUM> can then be determined to allow a go-around decision to be output when the deviation is too great. Steps of method <NUM> are performed by the computer vision system <NUM> of <FIG>). Method <NUM> commences, in some embodiments, when the aircraft <NUM> is determined to be within a predetermined proximity of the target runway based on a location of the aircraft <NUM> from the global positioning system module <NUM> and/or from the aircraft sensors <NUM>. In one embodiment, method <NUM> is invoked when the altitude of the aircraft <NUM> is less than a predetermined altitude during a landing approach. For example, when the aircraft altitude is less than <NUM> feet. The aircraft altitude may be determined based on altitude data from a radio altimeter included in the aircraft sensors <NUM>. The method <NUM> may repeat so as to continuously process each frame (or every n frames) of incoming video frame data <NUM>. The method <NUM> may cease when the altitude of the aircraft <NUM> is less than a set threshold such as less than <NUM> feet. The method may also cease when the cross-check process indicates insufficiently reliable data is being determined by the computer vision processing module <NUM>.

In step <NUM>, the video frame data <NUM> is received from the camera <NUM>. In step <NUM>, the video frame data <NUM> is processed by the computer vision processing module <NUM> to localize the runway. Step <NUM> may include execution of an object finding algorithm that is tuned to identify and localize runways. Step <NUM> produces pixel positions for the identified runway, which may include one, two, three or more discrete locations on the runway. In step <NUM>, data <NUM> for the real world position of the runway is retrieved from the navigation database <NUM>. In some embodiments, the computer vision system <NUM> derives a target runway based on a flight plan or based on aircraft position and heading information from the GPS module <NUM> and/or the aircraft sensors <NUM> and information on the most likely runway at a nearby airport. The most likely runway can be derived from factors such as proximity and alignment, which are available from the navigation database <NUM>.

In step <NUM>, the translation matrix <NUM> is calculated based on intrinsic parameters of the camera <NUM>, the pixel positions <NUM> representing the runway location in camera coordinates and the data <NUM> representing the position of the runway in real world coordinates from the navigation database <NUM>. In step <NUM>, the real world aircraft position <NUM> in world coordinates is determined based on the translation matrix <NUM> and the data <NUM> representing the position of the runway in real world coordinates.

In step <NUM>, an automated decision is made as to whether go-around conditions are met based on the real world aircraft position <NUM> and whether a deviation between the real world aircraft position <NUM> and the planned flight path <NUM> exceed thresholds specified by go-around rules. In step <NUM>, the go-around decision <NUM> (which may be a continue approach or go-around state) is output to the pilot via the output system <NUM> in aural and/or visual form. The pilot can then take go-around or continue approach action based on the output from step <NUM>.

In some embodiments, the method includes step <NUM> in which a cross-check is carried out on the real world aircraft position <NUM> as compared to reference aircraft position data <NUM> from the aircraft sensors <NUM> and/or from the GPS module <NUM>. A cross-check on the speed vector <NUM> may also be implemented. If the cross-check indicates that the data determined by the computer vision processing module <NUM> is insufficiently reliable, then method <NUM> may cease and the pilot makes the go-around decision without additional support from the computer vision system <NUM>.

In a simpler embodiment, the computer vision system <NUM> could be used only to indicate to a pilot, through the output system <NUM>, that visual contact has been made with the runway by processing the video frame data <NUM>. In another embodiment, this indication could be combined with later indications concerning the go-around/continue approach decision.

Embodiments of the computer vision system <NUM> have been described in terms of functional and/or logical block components and various processing steps.

The use cases and the depictions provided here are only exemplary in nature. It should be possible to use different symbology and semantics to accomplish the same concepts described herein.

Claim 1:
A computer vision system (<NUM>) for an aircraft (<NUM>), comprising:
a camera (<NUM>) mounted to the aircraft and configured to output video frame data (<NUM>) of a scene ahead of the aircraft;
a processor (<NUM>) in operable communication with the camera, the processor configured to execute program instructions, wherein the program instructions are configured to cause the processor to process the video frame data to:
identify a runway;
extract runway pixel positions (<NUM>) associated with the runway;
determine the aircraft position in a real world coordinate frame based on the pixel positions;
receive predefined aircraft position data and go-around rules (<NUM>) for a compliant approach to the runway;
calculate an aircraft position deviation (<NUM>) based on the aircraft position and the predefined aircraft position data;
determine whether the aircraft position deviation is in conformance with the go-around rules; and
output an indication of conformity with the go-around rules, characterized in that the program instructions are further configured to cause the at least one processor to:
receive reference aircraft position data (<NUM>) from a global positioning system of the aircraft;
cross-check the aircraft position with the reference aircraft position data; and
when the cross-check indicates a discrepancy beyond a threshold, outputting a warning and discontinuing determining whether the aircraft position deviation is in conformance with the go-around rules based on the video frame data.