METHOD OF OPTICALLY LOCATING AND GUIDING A VEHICLE RELATIVE TO AN AIRPORT

A method of optically locating and guiding a vehicle relative to an airport having standardized signage where the method includes generating an image of at least a portion of the airport from an optical sensor mounted on the vehicle, determining the location of the vehicle, and providing guidance for operation of the vehicle.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

An initial explanation of an airport environment having standardized signage will be useful in understanding the inventive concepts. Airport signage, including signs, markings and lighting, are standardized by the International Civil Aviation Organization.FIGS. 1-5illustrate a variety of airport standardized signage; additional information regarding standardized signage may be found at http://www.faa.gov.

Beginning withFIG. 1, a taxiway2aligned with a runway4is illustrated as well as runway threshold markings6, runway designation markings8, runway aiming point markings10, runway touchdown zone markings12, runway centerline markings14, runway side stripe markings16, runway lighting18, taxiway markings including taxiway centerline20, taxiway edge marking22, taxiway lighting24, holding position markings26, holding position sign28, and holding position sign30.

FIG. 3illustrates a taxiway A at50, a taxiway B at52, a taxiway E at54, a runway designated as15at56as well as an area58on which is located an aircraft60, plow truck62, and luggage cart64all of which may be capable of optically locating themselves per embodiments of the invention. Various location signs66including taxiway location markings have been illustrated as well as taxiway directional markings or direction signs68. Such direction signs68identify the designation(s) of the intersecting taxiway(s) leading out of the intersection that a pilot would normally be expected to turn onto. Where appropriate, the direction signs68include arrow(s) indicating the direction of the turn. Direction signs68are normally located on the left prior to the intersection. Additional illustrated signage includes taxiway location signs70in combination with holding position signs72and74, runway safety area markers76and78.

FIG. 4illustrates a taxiway A at80, a taxiway E at82, both of which have been illustrated as including a centerline84. The signage86includes both a location sign and a direction sign. When the intersection includes only one crossing taxiway, the direction sign may have two arrows associated with the crossing taxiway as illustrated.FIG. 5illustrates a taxiway A at88, a taxiway F at90, a taxiway T at92, and a taxiway E at94, a location sign96and a destination and location sign98. On the destination and location sign98the taxiway designations and their associated arrows are arranged clockwise starting from the first taxiway on the left. For a location sign located with the direction signs, the location sign is placed so that the designations for all turns to the left will be to the left of the location sign and the designations for continuing straight ahead or for all turns to the right would be located to the right of the location sign. It will be understood thatFIGS. 1,2,3,4, and5merely illustrate a portion of the standard signage at an airport and that additional or varying signage may be utilized.

FIG. 6illustrates an exemplary vehicle that may be capable of optically locating itself. In the illustrated example, the vehicle includes an aircraft100having a cockpit102where a first user (e.g., a pilot) may be present in a seat104at the left side of the cockpit102and another user (e.g., a co-pilot) may be present at the right side of the cockpit102in a seat106. A flight deck108having various instruments110and multiple multifunction flight displays112may be located in front of the pilot and co-pilot and may provide the flight crew with information to aid in flying the aircraft100. The flight displays112may include either primary flight displays or multi-function displays and may display a wide range of aircraft, flight, navigation, and other information used in the operation and control of the aircraft100including that the flight displays112may be electronic flight bag displays. The flight displays112may be capable of displaying color graphics and text to a user. The flight displays112may be laid out in any manner including having fewer or more displays and need not be coplanar or the same size. A touch screen display or touch screen surface114may be included in the flight display112and may be used by one or more flight crew members, including the pilot and co-pilot, to interact with the systems of the aircraft100. It is contemplated that one or more cursor control devices116and one or more multifunction keyboards118may be included in the cockpit102and may also be used by one or more flight crew members to interact with the systems of the aircraft100.

An optical sensor120may be mounted to the aircraft100and has been schematically illustrated as being located at an outside forward portion of the aircraft100. It will be understood that the optical sensor120may be mounted anywhere on the aircraft100, internal or external, and is preferably forward looking so that it may generate images of the environment located in front of the aircraft100. By way of further example, multiple optical sensors120may be used including that wing-mounted sensors may be utilized. By way of non-limiting example, the optical sensor120may include a camera, which may be mounted on a forward portion of the aircraft100in a fixed location. Exemplary cameras include a CCD camera, a CMOS camera, a digital camera, a video camera, an infrared camera, or any other type of suitable camera for observing the external environment of the aircraft100. In this manner, the optical sensor120may be capable of generating an image including at least one of a still image or a video image and outputting an image signal for same. The generated image may be in any suitable spectrum for the anticipated signage, including at least one of an infrared spectrum, visible light spectrum, and ultraviolet spectrum. It should be appreciated that the use of a camera is exemplary only and that other types of optical sensors120may be employed. Regardless of the type of optical sensor120used, it is contemplated that the optical sensor120may detect standardized signage, including markings such as markings painted on a runway in the environment in front of the aircraft100. It is contemplated that the optical sensor120may provide any suitable type of image signal including images, video, etc. of at least a portion of environment in front of the aircraft100.

A computer or controller122may be operably coupled to components of the aircraft100including the flight displays112, touch screen surface114, cursor control devices116, multifunction keyboards118, and optical sensor120. The controller122may also be connected with other controllers (not shown) of the aircraft100. The controller122may include memory124and a processor126, which may be running any suitable programs. The memory124may include random access memory (RAM), read-only memory (ROM), flash memory, or one or more different types of portable electronic memory, such as discs, DVDs, CD-ROMs, etc., or any suitable combination of these types of memory. The controller122may also be connected with other controllers of the aircraft100over the aircrafts communication network. A computer searchable database of information may be stored in the memory124and accessible by the processor126or the controller122may be operably coupled to a database of information. For example, such a database may be stored on the same or alternative computer as the controller. It will be understood that the database may be any suitable database, including a single database having multiple sets of data, multiple discrete databases linked together, or even a simple table of data. For example, the database may include information related to standardized airport signage including standardized signs, standardized markings, and standardized lights. The controller122may also receive information from various sources including external memory, communication links such as a wireless communication link, and additional controllers or processors. For example, the aircraft100may receive a predetermined route for the aircraft100to traverse areas of the airport. Such a predetermined route may be input by the pilot or uploaded from an airline operations center. By way of non-limiting example, the predetermined route may include an airport taxi route from a gate at the airport to a runway that the aircraft is to takeoff on.

An image processing system128may utilize the database of standardized signage and an image processor. The image processing system128may be included in the aircraft100and may be operably coupled to the optical sensor120to receive the image signal and perform analysis on it. While the image processing system128is depicted as being a component of the controller122, it is contemplated that the image processing system128could be a physically separate entity from controller122. In the illustrated example, the controller122having the image processing system128may analyze the images signal from the optical sensor120without the utilization of a separate image processor. The image processing system128may be any suitable processing platform. Including that the image processing system128may be any combination of hardware and software that receives the image signal and processes or analyzes the image. For example, the image processing system128may include a software application that receives the image signal and processes it using object detection or recognition algorithms to detect and identify components of the environment in front of the aircraft100.

By way of alternative example, the object recognition algorithm may be implemented in a set of computer executable instructions stored in the memory124of the controller122and a separate image processor component may not be required. For example, Optical Character Recognition (OCR) including application-oriented OCR or customized OCR software may be used to identify the standard signage. Additionally, object recognition such as computer vision-based object recognition may be used to recognize objects within the generated image.

The aircraft may be operated by the pilot to travel a predetermined route. For example, the aircraft may be taxied from the area58down the taxiway B at52to the runway15at56. During such operation, the controller122may receive data from the optical sensor120from which the controller122and the image processing system128may determine information regarding the environment in front of the aircraft100. By way of non-limiting example, the aircraft's location may be determined from the recognized signage in the image generated by the optical sensor120. The controller122may access the memory124and the image processing system128may match the signage in the image with proper imagery data that may be stored in the memory124. In this manner, the controller122may determine the location of the aircraft100. The controller122may be configured to provide guidance to the pilot such that the pilot may operate the aircraft such that it continues to progress along the predetermined route. For example, if a taxiway location sign is identified, the controller122may determine the location of the aircraft100and may determine from its location what course of action should be taken to maintain the aircraft on its predetermined route. Many graphical and illustrative techniques may be used to indicate the location of the aircraft and provide guidance for operation of the aircraft. The guidance may be provided as any suitable indications using any suitable mechanism located in the cockpit102.

While a commercial aircraft has been illustrated it is contemplated that embodiments of the invention may be used in any type of aircraft, for example, without limitation, fixed-wing, rotary-wing, rocket, personal aircraft, and military aircraft. It will be understood that the technology used in the general aviation aircraft may be the equivalent of a webcam and tablet computer with suitable software and in larger business and transport aircraft the technology used may include existing computer platforms, enhanced vision cameras, and integration with the Flight Management System for runway selection. It is also contemplated that the indication may be provided by the tablet computer. Embodiments of the invention may be used for non-autonomous and autonomous vehicles.

FIG. 7illustrates an alternative exemplary vehicle that may be capable of optically locating itself. In the illustrated example, the vehicle includes a truck200. The truck200includes many of the same features as the aircraft100previously described and therefore, like parts will be identified with like numerals increased by100, with it being understood that the description of the like parts applies to this alternative embodiment, unless otherwise noted.

A cabin202where a user may be present in a seat204is included in the truck200. A dashboard208having various instruments210may be located in front of the user and may provide the user with information to aid in driving the truck200. As with the previous vehicle, the truck includes an optical sensor220, which may be mounted to the truck200, and has been schematically illustrated as being located at a forward portion of the truck200. It will be understood that the optical sensor220may be mounted anywhere on the truck200, internal or external, and is preferably forward looking so that it may generate images of the environment located in front of the truck200. In this manner, the optical sensor220may be capable of generating an image including at least one of a still image or a video image and outputting an image signal for same. It is contemplated that the optical sensor220may provide any suitable type of image signal including images, video, etc. of at least a portion of environment in front of the truck200.

A controller or computer222may be operably coupled to components of the truck200including the optical sensor220. The computer222may include memory224and a processor226, which may be running any suitable programs. A computer searchable database of information may be stored in the memory224and accessible by the processor226or the computer222may be operably coupled to a database of information that includes information related to standardized airport signage including standardized signs, standardized markings, and standardized lights. An image processing system may be included in the computer222and may be operably coupled to the optical sensor220to receive the image signal and perform analysis on it. Although the computer222has been illustrated as a laptop computer, any suitable computer222or controller may be used. It is contemplated that the computer222may include a suitable user interface216and user interface screen218. The computer222may also include any suitable system for creating an aural or visual indication.

The truck200operates much the same way as the aircraft100including that during operation the computer222may receive data from the optical sensor220from which the computer222and the image processing system228may determine information regarding the environment in front of the truck200. By way of non-limiting example, the truck's location may be determined from the recognized signage in the image generated by the optical sensor220. The computer222may access the memory224and the image processing system228may match the signage in the image with proper imagery data that may be stored in the memory224. In this manner, the computer222may determine the location of the truck200. A predetermined route for the truck may be communicated to the computer or otherwise be stored on the computer222. The computer222may compare the determined location of the truck200with the predetermined route and the computer222may provide indications including operational instructions regarding the same to the driver.

For example, if the truck200is meant to plow the taxiway B at52before plowing the runway22at56, and then taxiway A at50(this may define its predetermined route), the computer222may determine the location of the truck200and may compare its location to the predetermined route and may provide directional guidance to the user to maintain the truck on the predetermined route. Many graphical and illustrative techniques may be used to indicate the location of the truck200and guidance to maintain the truck on the predetermined route and such guidance may appear on the user interface screen218or may aurally be communicated to the user.

In this manner, it will be understood that any suitably equipped vehicle may optically locate itself relative to an airport having standardized signage and with respect to a predetermined route for the vehicle such that guidance may be provided for operation of the vehicle to make continued progress along the predetermined route. In accordance with an embodiment of the invention,FIG. 8illustrates a method300of optically locating a vehicle relative to an airport having standardized signage.

The method300may begin with receiving route information defining a predetermined route within the airport at302. The route information may include at least a destination for the vehicle. Alternatively, the predetermined route information may include the taxiways, runways, etc. that the vehicle is to travel. The predetermined route may be received in any suitable manner including that the predetermined route may be received from an airline operations center or other location or may be input by a user of the vehicle.

The method300may continue with generating an image of at least a portion of the airport at304. This may be done using any suitable optical sensor including a camera mounted on the vehicle and may include generating an image a taxiway of the airport, a runway of the airport, and surrounding areas. At306, at least some of the standardized signage in the generated image may be identified. This may be accomplished by processing the generated image on a computer aboard the vehicle. Identifying at least some of the standardized signage in the generated image may include identifying at least one of runway threshold markings, runway designation markings, runway aiming point markings, runway touchdown zone markings, runway centerline markings, runway side stripe markings, runway shoulder markings, taxiway location markings, taxiway directional markings, taxiway centerline markings, geographic position markings, and holding position markings.

At308, the location of the vehicle may be determined based on the identified standardized signage. For example, the computer onboard the vehicle may use information regarding standard airport signage, markings and lighting to determine the position of the vehicle relative to the airport or using the standardized signage identified in the generated image. Determining the location may include determining a progress of the vehicle along the predetermined route. By way of non-limiting example, a detected taxiway directional marking may be compared with data regarding the predetermine route for the vehicle. It is contemplated that determining the location of the vehicle may include determining the distance from the vehicle to the identified standardized signage. A situational position of the vehicle may also be determined based on the identified standardized signage.

It is contemplated that multiple images may be generated and that the location of the vehicle may be determined based on the signage identified in the multiple images. It is further contemplated that more than one sensor may be used such that multiple images may be generated by the sensors and that the location of the vehicle may be determined based on the signage identified in the multiple images. The multiple images may better allow for depth to be determined aiding in the determination of the location of the vehicle. By way of example, object recognition software may determine the taxiway or runway the vehicle is currently on as well as approaching crossing intersections.

At310, the location determined at308may be compared with the route information to aid in defining where the vehicle is compared to the predetermined route.

At312, guidance for operation of the vehicle may be provided. The guidance may include directions or other information to aid in the vehicle progressing from the determined location along the predetermined route. By way of non-limiting example, providing guidance may include providing an indication or alert, within the vehicle, related to a control action for the vehicle. More specifically, in the case of the aircraft, the indication or alert may be provided to the flight crew within a cockpit of the vehicle. Appropriate visual and/or aural guidance may be presented to maintain the vehicle on its predetermined route. At least one of an audible and visual indication may be provided. A variety of suitable indications may be provided based on the determined location of the vehicle. For example, indications may include that the vehicle is approaching a taxiway or runway on the ground or crossing a taxiway or runway on the ground. A visual or aural indication may identify if the vehicle is to maintain its course or turn onto an intersecting taxiway or runway.

If it is determined at310that the vehicle has somehow deviated from its predetermined route an indication regarding the discrepancy between the determined location and the predetermined route may be provided. In such an instance the guidance provided at312may include guidance to return the vehicle to its predetermined route. Alternatively, the guidance may include guidance regarding a new or alternative route that is based on the previously received route information and the determined location of the vehicle.

It will be understood that the method of guiding the vehicle is flexible and that the method300illustrated is merely for illustrative purposes. For example, the sequence of steps depicted is for illustrative purposes only, and is not meant to limit the method300in any way as it is understood that the steps may proceed in a different logical order or additional or intervening steps may be included without detracting from the embodiments of the invention. For example, the method may include continuously generating the image at304, continuously determining the location of the vehicle at308, and continuously providing guidance based thereon at312.

The camera image can also be supplemented with additional identifying features to highlight the detected signage to vehicle operators if the camera image is displayed to the operators. For example, the taxiway centerline marking may be identified and steering commands may be provided based thereon. Further, alerts may be provided if the vehicle is moved too far off the centerline of the taxiway.

By way of non-limiting example, the method of optically locating the vehicle may include generating an image of portions of the airport. For example,FIG. 9illustrates an image of a portion of an airport400including taxiway A at402, taxiway B at404, taxiway E at406, runway at408that may be taken by a vehicle traveling along a predetermined route. The above described embodiments may identify at least some of the standardized signage including location signs410, direction signs412, and centerline markings414.

It is contemplated that a distance the vehicle is from an approaching taxiway or runway may be determined from the identified signage. More specifically, the perspective of the signage in the generated image may be used to determine the distance the vehicle is from the approaching taxiway. An indication of the distance the vehicle is from the taxiway may then be provided within the vehicle along with any guidance to maintain the vehicle on its predetermined route such as guidance for turning the vehicle onto the approaching taxiway.

By way of additional non-limiting example, it is also contemplated that one or more hazards420may be identified in the generated image and that an alert of the identified hazard may be provided. For example, it is contemplated that indications may be given with respect to detected hazards on the runway such as vehicles or animals. In the illustrated example, a hazard420in the form of a truck is located on the taxiway B at404and an alert may be provided regarding same. For example, if the image is displayed to the operator of the vehicle, then the hazard420may be indicated with highlighting on the screen such as indicated at422. If the vehicle is to travel from taxiway B at404to taxiway E at406, then guidance such as an arrow at430may be provided and an indication of when to turn, for example in 500 feet as indicated at432, may be provided.

Furthermore, the physical airport signage and markings could be supplemented with infrared or ultraviolet mechanisms to convey additional information to assist in detection and identification. More specifically, the infrared or ultraviolet mechanisms could be recognized if the optical sensor technology used can discern the infrared and ultraviolet objects. It is contemplated that such mechanisms may not be human readable letters or numbers and may include shapes or digital encoding. Furthermore, these mechanisms may not be the current standard symbology in the standardized signage and may instead by symbology developed for locating the vehicle. The optical sensor image can also be supplemented with additional identifying features to highlight the detected taxiway and runway components if the image is displayed to the operator of the vehicle. Further still, the indications provided to the operator may highlight or display the centerline of the taxiway or runway during low visibility operation. External systems may use the centerline identification to further augment ground steering methodologies used by those systems.

The above described embodiments provide a variety of benefits including that the proposed system is self-contained, may be used at any airport, and may be used with or without the existing advisory methodologies and provides an added safety layer to the existing layers of prevention measures. A technical effect is that the location of the vehicle may be determined from recognized signage and monitor progress along that route, advise of upcoming turns, approaching sensitive areas such as runways, including that alerts may be provided to the vehicle operator in an effort to prevent unapproved runway incursions and to ensure vehicle progresses along its predetermined route. The above described embodiments would not require prior knowledge of the airport topology, construction, or structure and does not require radar, positioning systems, or detailed airport map databases that require continual update. The system also identifies existing low visibility taxi lighting and signage to follow those indications and highlight the path to the vehicle operator.

Furthermore, it is contemplated that embodiments of the invention may be used with a vehicle in the form of an autonomous vehicle or an unmanned vehicle. In the case of an unmanned vehicle, an image may be generated from an optical sensor mounted on the unmanned vehicle. The identification of at least some of the standardized signage may be done either onboard the unmanned vehicle or at a ground station. If the processing is done at the ground station, such as for example a computer at the ground stations, it is contemplated that the unmanned vehicle and the ground station may have any suitable communication abilities so that the image signal may be provided to the ground station. Further, the providing the guidance for operation of the vehicle may include providing guidance to a user on the ground or may include providing operational instructions to the unmanned vehicle.

Further, it will be understood that the inventive embodiments may be capable of identifying any suitable additional signage. For example, while not illustrated or described runway guard lights and stop bar lights may also be included and utilized by the inventive embodiments. The runway guard lights help highlight the runway hold point and the stop bar lights are controlled by the control tower at some airports and are turned off when it is okay to cross or enter a runway.