Abstract:
A method for augmenting a synthetic vision system of a vehicle includes receiving, with a processor, signals indicative of real-time sensor data of a terrain for the vehicle via one or more acquisition devices; creating, with the processor, a terrain mesh of the terrain in response to the receiving of the sensor data; correlating, with the processor, the terrain mesh with preloaded terrain data of the terrain; creating, with the processor, a multispectral image of the terrain in response to the correlating the terrain mesh with the preloaded data; and texturing, with the processor, the terrain mesh for display on a display device.

Description:
FIELD OF THE INVENTION 
       [0001]    The subject matter disclosed herein relates generally to the field of instrumentation for a vehicle, and to a system and method for augmenting an existing synthetic vision database for navigation and obstacle avoidance in a rotary-wing aircraft. 
       DESCRIPTION OF RELATED ART 
       [0002]    Typically, synthetic vision systems on aircraft rely on on-board databases of objects, e.g., terrain, obstacles, etc., to render an image of an environment, external to the aircraft, into visual form for display to a pilot on a primary flight display. The on-board object database can be an on-board static terrain database. However, typical synthetic vision systems render an image through on on-board synthetic vision databases that are infrequently updated or do not account for real-world changes of objects. Also, augmented synthetic vision systems, such as forward looking infrared (FLIR) technology, are not integrated with the on-board synthetic vision database. Prior art synthetic vision systems have attempted to integrate both types of systems but have resulted in an image overlay being displayed over the synthetic image. A system for augmenting an existing synthetic vision database on a vehicle with multispectral sensor data is desired. 
       BRIEF SUMMARY 
       [0003]    According to one aspect of the invention, a method for augmenting a synthetic vision system of a vehicle includes receiving, with a processor, signals indicative of real-time sensor data of a terrain for the vehicle via one or more acquisition devices; creating, with the processor, a terrain mesh of the terrain in response to the receiving of the sensor data; correlating, with the processor, the terrain mesh with preloaded terrain data of the terrain; creating, with the processor, a multispectral image of the terrain in response to the correlating the terrain mesh with the preloaded data; and texturing, with the processor, the terrain mesh for display on a display device. 
         [0004]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the receiving of the sensor data further comprises receiving real-time terrain information external to the vehicle. 
         [0005]    In addition to one or more of the features described above, or as an alternative, further embodiments could include correlating the terrain mesh for features of known obstacles in the sensor data with similar features of known obstacles in the preloaded terrain data. 
         [0006]    In addition to one or more of the features described above, or as an alternative, further embodiments could include correlating the terrain mesh for determining obstacles in the sensor data with missing obstacles in the preloaded terrain data. 
         [0007]    In addition to one or more of the features described above, or as an alternative, further embodiments could include correlating the terrain mesh with one or more probabilistic sensor models. 
         [0008]    In addition to one or more of the features described above, or as an alternative, further embodiments could include determining a weighting factor for one or more of the known obstacles in response to the correlating the terrain mesh. 
         [0009]    In addition to one or more of the features described above, or as an alternative, further embodiments could include creating the terrain mesh that represents a three-dimension (3-D) image of the terrain with topographical features below the vehicle. 
         [0010]    According to another aspect of the invention, a system for augmenting a synthetic vision system of a vehicle includes one or more sensors configured to receive signals indicative of terrain information for the vehicle; and a computer operably connected to the one or more sensors and configured to: receive signals indicative of real-time sensor data of a terrain for the vehicle via one or more acquisition devices; create a terrain mesh of the terrain in response to the receiving of the sensor data; correlate the terrain mesh with preloaded terrain data of the terrain; create a multispectral image of the terrain in response to the correlating the terrain mesh with the preloaded data; and texture the terrain mesh for display on a display device. 
         [0011]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to receive real-time terrain information external to the vehicle. 
         [0012]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to correlate the terrain mesh for features of known obstacles in the sensor data with similar features of known obstacles in the preloaded terrain data. 
         [0013]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to correlate the terrain mesh for determining obstacles in the sensor data with missing obstacles in the preloaded terrain data. 
         [0014]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to correlate the terrain mesh with one or more probabilistic sensor models. 
         [0015]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to determine a weighting factor for one or more of the known obstacles in response to the correlating the terrain mesh. 
         [0016]    In addition to one or more of the features described above, or as an alternative, further embodiments could include the processor configured to create the terrain mesh that represents a three-dimension (3-D) image of the terrain with topographical features below the vehicle. 
         [0017]    Technical effects of embodiments include rendering an environment external to a vehicle on synthetic vision display that uses multispectral information to augment information about the environment stored in a synthetic vision database for navigation and obstacle avoidance during travel of the vehicle, including travel in degraded visual environments. 
         [0018]    Other aspects, features, and techniques of the invention will become more apparent from the following description taken in conjunction with the drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 
         [0019]    The subject matter, which is regarded as the invention, is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which like elements are numbered alike in the several FIGURES: 
           [0020]      FIG. 1  is a perspective view of an exemplary aircraft for use with embodiments of the invention; 
           [0021]      FIG. 2  is a system architecture block diagram according to an embodiment of the invention; and 
           [0022]      FIG. 3  is a data flow diagram according to an embodiment of the invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0023]    Referring now to the drawings,  FIG. 1  illustrates a general perspective view of an exemplary vehicle in the form of a vertical takeoff and landing (VTOL) rotary-wing aircraft  100  for use with embodiments of the invention. In an embodiment, the aircraft  100  uses a synthetic vision algorithm  204  ( FIG. 2 ) for augmenting an on-board synthetic vision database for displaying 3-D terrain information to a pilot on a display device including detached and attached objects surrounding aircraft  100 . Attached objects include portions of natural and man-made terrain that are anchored to the ground, and detached objects include portions of natural or man-made objects that are not attached to the ground. For example, a bridge would include pillars that are considered attached objects and spans between pillars are considered detached objects. Other examples of detached objects include telephone or power lines, structural overhangs, tree canopies, and the like. The synthetic vision algorithm  204  ( FIG. 2 ) can receive real-time multispectral data for augmenting terrain data in an existing on-board synthetic vision database in order to enhance navigation and added safety in degraded visual environments. 
         [0024]    As illustrated, rotary-wing aircraft  100  includes an airframe  102  having a main rotor assembly  104  and an extending tail  106  which mounts an anti-torque system, such as a tail rotor assembly  108 . In embodiments, the anti-torque system may include a translational thrust system, a pusher propeller, a rotor propulsion system, or similar. The main rotor assembly  104  includes a plurality of rotor blades  110  mounted to a rotor hub  112  that rotates about axis A. Also, tail rotor assembly  108  includes a plurality of rotor blades  116  that rotates about axis B, which is orthogonal to the plane of rotation of blades  116 . Main rotor assembly  104  and tail rotor assembly  108  are driven to rotate by one or more engines  114  through one or more gearboxes (not shown). Although a particular rotorcraft is illustrated and described in the disclosed embodiments, other configurations and/or machines, such as manned or unmanned fixed wing and rotary wing aircraft, land-borne vehicles for fire, security, and search and rescue, ships, and submarines can also benefit from embodiments of the invention. For simplicity, embodiments are described with reference to “aircraft”. 
         [0025]      FIG. 2  illustrates a schematic block diagram of a system  200  on board aircraft  100  that is used to implement the embodiments described herein. In an embodiment, system  200  uses a synthetic vision algorithm  204  to augment an on-board synthetic vision database  210  through real-time multispectral sensors. Particularly, sensor data from a plurality of “image” acquisition devices  212  (hereinafter “acquisition devices  212 ”) are located on aircraft  100  for capturing data from a terrain or environment external to aircraft  100 . For clarity, “image” is used to represent sensor data that is acquired on-board aircraft  100  such as, e.g., Light Detection And Ranging (LIDAR), or radio waves, e.g., acquired from RAdio Detection And Ranging (RADAR). Real-time terrain information is used by flight display computer  202  for augmenting pre-loaded and/or predetermined terrain data of the environment in an on-board synthetic vision database  210 . In embodiments, the plurality of acquisition devices  212  can include long-range acquisition devices with a perception range of several miles such as, e.g., 3-D RADAR, telescope cameras, color video camera, multi-spectral camera, Time-of-flight 3D camera; medium range acquisition devices with a perception range from about 10 meters to about 1000 meters such as, e.g., pan-tilt-zoom (PTZ) cameras, single visible or infrared cameras, and medium range LIDAR; and short range acquisition devices with a perception range of less than 1 meter such as, e.g., short-range high definition LIDAR, or stereo cameras. Other acquisition devices  212  can include short wave infrared (SWIR) camera or long range infrared camera (LWIR) that can also be used in inclement weather conditions like fog, snow, or rain. 
         [0026]    Also shown in  FIG. 2 , system  200  includes flight display computer  202  that executes instructions for implementing synthetic vision algorithm  204 . Aircraft computer  202  receives sensor data on an environment external to aircraft  100  from the multispectral sensors associated with the plurality of acquisition devices  212 . Computer  202  includes a memory  208  that communicates with a processor  206 . The memory  208  may store synthetic vision algorithm  204  as executable instructions that is executed by processor  206 . The instructions mat be stored or organized in any manner and at any level of abstraction, such as in connection with the execution of synthetic vision algorithm  204 . Processor  206  may be any type of processor (such as central processor unit (CPU) or general processor unit (GPU)), including a general purpose processor, a digital signal processor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable grid array (FPGA), or the like. In an embodiment, processor  206  may include one or more image processors for processing the associated image data from the acquisition devices  212  using one or more processing algorithms to produce one or more processed signals. Also, in embodiments, memory  208  may include random access memory (RAM), read only memory (ROM), or other electronic, optical, magnetic, or any other computer readable medium onto which is stored the synthetic vision algorithm  204  described below. 
         [0027]    System  200  may include a synthetic vision database  210 . Database  210  can be used to store pre-loaded terrain/environment data that includes location and elevations of natural and man-made terrain at locations in which aircraft  100  is operating. As mentioned above in  FIG. 1 , stored terrain data can represent at least two types of items in the physical environment surrounding the aircraft: detached objects and attached objects. Database  210  may also store real-time terrain data (e.g., detached and attached objects) acquired by acquisition devices  212 . The data stored in database  210  may be based on one or more other algorithms or processes for implementing synthetic vision algorithm  204 . For example, in some embodiments, data stored in database  210  may be a result of processor  206  having subjected sensor data or preloaded data to one or more filtration processes. Database  210  may be used for any number of reasons. For example, database  210  may be used to temporarily or permanently store data, to provide a record or log of the data stored therein for subsequent examination or analysis, etc. In some embodiments, database  210  may store a relationship between data, such as one or more links between data or sets of data acquired through the various acquisition devices  212  on board aircraft  100 . System  200  may include a display device  214  coupled to computer  202 . Display device  214  may include any device or apparatus suitable for displaying various type of computer generated symbols and information representing for example, pitch, heading, flight path, air speed, altitude, landing information, waypoints, targets, obstacles, and terrain objects in an integrated form. 
         [0028]      FIG. 3  illustrates an exemplary data flow diagram  300  that is performed by processor  206  ( FIG. 2 ) for implementing synthetic vision algorithm  204  ( FIG. 2 ) according to an embodiment of the invention, and, as such,  FIG. 2  is also referenced in the description of data flow diagram  300  of  FIG. 3 . Particularly, data flow diagram  300  depicts a multispectral approach that is implemented by synthetic vision algorithm  204 , which uses data fusion to correlate/augment real-time terrain information (i.e., sensor data from multispectral sensors) with preloaded terrain information from an on-board synthetic vision database for display to a pilot for navigation, obstacle avoidance, and the like. 
         [0029]    Data flow diagram  300  begins at  302  where raw terrain data of the terrain is acquired as sensor data in real-time from one or more acquisition devices  212  for processing by processor  206 . In some non-limiting examples, algorithm  204  will select one or more acquisition devices  212  based on distance of aircraft  100  ( FIG. 1 ) from a target or obstacle. In an embodiment, SWIR or LWIR cameras can be used during inclement weather conditions like fog, snow, or rain. The sensor data for acquisition devices  212  may be low-quality due to variations in lighting, distance, and image enhancement algorithms such as edge preserving de-noising, contrast enhancement, and sharpening may be applied to the sensor data before further processing by processor  206 . 
         [0030]    In  304 , a terrain mesh of the sensor data is created. A terrain mesh is a representation of a 3-D terrain having topographical features of the surface (e.g., a 3-D heightmap of topographical features) over which the aircraft  100  is traveling. 
         [0031]    In  306 , the terrain mesh is correlated with pre-loaded data of the terrain in synthetic vision database  210 . The terrain mesh is correlated in order to refine sensor data acquired from acquisition devices  212  and/or to augment the terrain data in synthetic vision database  210  with information such as, e.g., terrain elevation and/or new obstacles not present in synthetic vision database  210 . In an example, a plurality of probabilistic sensor models are applied to the sensor data associated with each acquisition device  212 . Each probabilistic sensor model is associated with its own sensor data acquired from a respective acquisition device  212 . The probabilistic sensor model is used to correlate sensor data for features (e.g., terrain elevation) of known obstacles in the sensor data with similar features of known obstacles in preloaded terrain data from synthetic vision database  210 . In addition to the correlation of the features, or alternatively, probabilistic sensor model can also correlate new objects found in real-time sensor data with preloaded terrain data in synthetic vision database  210  such as, e.g., poles, wires, and structural overhangs that are not present in the preloaded data of synthetic vision database  210 . A weighting factor for features identified in the sensor data is determined. The weighting factor can be used by algorithm  204  to denoise sensor data for objects with respect to similar preloaded objects in synthetic vision database  210 . In embodiments, the probabilistic sensor model is continuously refined through sensor data acquired from known objects. 
         [0032]    In  308 , output from probabilistic sensor models are combined to produce a multispectral image of the terrain. In  310 , multispectral image is applied to the terrain mesh from  304  in order to texture the terrain mesh and display a texture map representing the terrain on display device  214 . For example, the multispectral image is applied to fill in additional details of objects in the terrain mesh including presenting objects that are not present in the preloaded data of synthetic vision database  210 . The processed terrain representation that is textured is presented to a pilot on display device  214 . 
         [0033]    Benefits of the embodiments of the invention described herein include enhanced navigation, obstacle avoidance, and added safety in degraded visual environments. The embodiments relate to rendering an environment external to a vehicle on a synthetic vision display that uses multispectral information to augment information about the environment stored in a synthetic vision database. 
         [0034]    The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. While the description of the present invention has been presented for purposes of illustration and description, it is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications, variations, alterations, substitutions or equivalent arrangements not hereto described will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. Additionally, while the various embodiments of the invention have been described, it is to be understood that aspects of the invention may include only some of the described embodiments. Accordingly, the invention is not to be seen as limited by the foregoing description, but is only limited by the scope of the appended claims.