Abstract:
A vehicle display system displays enhanced vision (EV) and synthetic vision (SV) images to an operator of a vehicle. The display system includes: an EV vision image sensor for generating EV images; an SV database containing information regarding terrain and objects of interest for a travel path of a vehicle; an SV image generating unit for generating SV images based on travel of the vehicle and information from the SV database; an EV image sensor control unit for controlling a field of view of the EV image sensor as a function of object of interest information from the SV database; and a display for displaying images generated by said EV image sensor and the SV image generating unit.

Description:
FIELD OF THE INVENTION  
       [0001]    The present invention relates to vehicle display systems, and more particularly to a vehicle display system and method with both enhanced vision system and synthetic vision system images. 
       BACKGROUND OF THE INVENTION  
       [0002]    An enhanced vision (EV) system, e.g., which uses an infrared (IR) and/or millimeter wave (MMW) video camera, provides a vehicle operator, such aircraft crew, with the ability to see in reduced visibility conditions. EV images, however, are often difficult to interpret and may suffer from deterioration under various weather conditions. Due for example to the sensing of a thermal signature rather than a visual optical view, EV images containing particular features/objects of interest may be overwhelmed by other image areas. Although image processing can improve image quality, image enhancement performed over the entire EV image may not lead to a better display of regions of interest. 
         [0003]    A synthetic vision (SV) system can provide useful information to the vehicle crew by rendering an image based on pre-stored database information, including terrain and objects/obstructions, so that the operator can visualize essential terrain and flight path information that may be outside of the view of the EV system or not clearly shown in EV images. Thus, SV images can provide the vehicle operator with an effective interface for vehicle control. SV image integrity, however, is limited by the integrity of the information pre-stored in the database. Incomplete and/or outdated database information can result in SV images of limited value. Although an operator display may include both an SV image display and an EV image display, e.g., as a fused image (such as overlaying a semi-transparent EV image onto an SV image) or as a side-by-side display, such an “enhanced synthetic vision system” display does not adaptively aid the vehicle operator in quickly recognizing particular regions of interest. 
       SUMMARY OF THE PRESENT INVENTION 
       [0004]    According to one aspect, the present invention is a vehicle display system for displaying enhanced vision (EV) and synthetic vision (SV) images to an operator of a vehicle. The system comprises: an EV image sensor for generating EV images; an SV database containing information regarding terrain and objects of interest, such as navigational targets, waypoints, obstructions, areas to avoid, etc., for a travel path of a vehicle; an SV image generating unit for generating SV images based on travel of the vehicle and information from the SV database; an EV image sensor control unit for controlling a field of view of the EV image sensor as a function of object of interest information from the SV database; and a display for displaying EV images generated by the EV image sensor and the SV image generating unit. 
         [0005]    According to another aspect, the present invention is a vehicle display system for displaying enhanced vision (EV) and synthetic vision (SV) images to an operator of a vehicle. The system comprises: an EV image sensor for generating EV images; an SV database containing information regarding terrain and objects of interest for a travel path of a vehicle; an SV image generating unit for generating SV images based on travel of the vehicle and information from the SV database; an EV image enhancement control unit for controlling an enhancement area for EV images generated by the EV image sensor as a function of object of interest information from the SV database; and a display for displaying EV images generated by the EV image sensor and the SV image generating unit. 
         [0006]    According to yet another aspect, the present invention is a method for displaying enhanced vision (EV) and synthetic vision (SV) images to an operator of a vehicle. The method comprises: generating EV images using an EV image sensor; accessing an SV database containing information regarding terrain and objects of interest for a travel path of a vehicle; generating SV images based on travel of the vehicle and information accessed from the SV database; controlling a field of view of the EV image sensor or an enhancement area for an EV image generated the EV image sensor as a function of object of interest information from the SV database; and a display for displaying images generated by the EV image sensor and the SV image generating unit. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    The present invention will become more fully understood from the detailed description given hereinafter and the accompanying drawings that are given by way of illustration only, and thus do not limit the present invention. 
           [0008]      FIG. 1  is a block diagram of a vehicle display system for generating SV and EV display images according to a first embodiment of the present invention; 
           [0009]      FIG. 2  is an exemplary vehicle display of EV and SV images according to principles of the present invention; 
           [0010]      FIG. 3  is a block diagram of an EV image sensor control unit utilized in the vehicle display system of  FIG. 1  according to the first embodiment of the present invention; 
           [0011]      FIG. 4  is a flow diagram illustrating a process for generating a vehicle display with EV and SV images according to the first embodiment of the present invention; 
           [0012]      FIGS. 5A and 5B  are exemplary vehicle displays with combined EV and SV images according to the first embodiment of the present invention; 
           [0013]      FIG. 6  is a block diagram of a vehicle display system for generating SV and EV display images according to a second embodiment of the present invention; 
           [0014]      FIG. 7  is a block diagram of an EV image enhancement control unit applied in the vehicle display system of  FIG. 6  according to the second embodiment of the present invention; 
           [0015]      FIG. 8  is a flow diagram illustrating a process for generating a vehicle display with EV and SV images according to the second embodiment of the present invention; and 
           [0016]      FIGS. 9A and 9B  are exemplary vehicle displays with combined EV and SV images according to the second embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION 
       [0017]    Embodiments of the present invention will now be described with reference to the accompanying drawings. Although embodiments described herein are specific to aircraft display systems, it should be recognized that principles of the present invention may be applied to other vehicle display systems.  FIG. 1  illustrates a vehicle display system  100  according to a first embodiment of the present invention. As illustrated in  FIG. 1 , the vehicle display system  100  includes: a synthetic vision (SV) system  120 ; an enhanced vision (EV) system  110 ; an SV database  170 ; flight management systems  180 ; enhanced vision (EV) image sensor(s)  130 ; a display  190 ; an indexing/registration unit  160 ; and an EV image sensor control unit  150 . Although various block diagram elements shown in  FIG. 1  are illustrated as discrete elements, this illustration is for use and explanation, and it should be recognized that certain elements may be combined in one or more physical devices, e.g., one or more microprocessor(s) with associated software. 
         [0018]    The enhanced vision system  110  generates an image for display on the display  190  based on the output of one or more of the EV image sensors  130 , e.g., IR and/or MMW video cameras. In one embodiment of the present invention, at least one EV image sensor  130  is a steerable EV camera with a narrow field of view. For example, the EV image sensor  130  in one implementation of this embodiment is a zoomed camera with a viewing angle of several degrees, allowing a closer look at a specific object of interest. The synthetic vision system  120  renders an image based on pre-stored terrain, objects of interest, obstructions, etc. and navigation information stored in the SV database  170  for output to the display  190 . The synthetic vision system  120  also generates the SV images based on information from flight management systems  180 , such as vehicle positioning, heading, attitude, flight plan, etc.  FIG. 2  illustrates an exemplary display output  191  on the display  190 , including an EV sensor image  192  and an SV image  194 . The images may be indexed at the time of camera installation, e.g., by aligning the EV image sensor  130  to ensure that the sensor and the SV view are indexed. Such a process may be periodically repeated to assure proper alignment during normal course of maintenance. In the exemplary display of  FIG. 2 , the EV image  192  is an IR image and the SV image  194  includes flight management data (heading, altitude, speed, etc.) superimposed on the synthetic rendering of terrain and an object of interest (runway). 
         [0019]      FIG. 3  is a block diagram illustrating elements of the image sensor control unit  150  according to the first embodiment of the present invention. As illustrated in  FIG. 3 , the image sensor control unit  150  includes a priority unit  152 ; integrity unit  154 ; and a sensor area optimizing unit  156 . The priority unit  152  determines the priority of certain objects of interest based on information from the SV system  120 . Since the EV image sensor  130  in this embodiment has a small field of view, it may be unlikely that multiple objects of interest can be viewed in the same image. Therefore, the priority level determines which object of interest will be viewed first by steering the EV image sensor  130  toward that object of interest. The object of interest may be within the SV display area or may be outside the SV display area. For example, this may become necessary in instances where the pilot must fly the aircraft and maintain proper control, using the SV display, while searching for an object of interest using the EV image sensor  130 . 
         [0020]    The integrity levels are assigned to object of interest data, for example using either an on-board database or uploaded data associated with a real-time datalink. The integrity level represents the confidence level for the object of interest data. For example, for an object of interest with high integrity, the area to zoom in on and search for the object of interest can be very limited, allowing quicker recognition of objects of interest. The integrity unit  154  determines the integrity level of objects of interest based on information from the SV system  120 . 
         [0021]    Next, operation of the vehicle display system  100  according to the first embodiment illustrated in  FIG. 1  and  FIG. 3  will be described with reference to the flow diagram of  FIG. 4 . Initially, the synthetic vision system  120  inputs flight and flight management system (FMS) information, including for example aircraft position, heading, attitude, flight plan, etc. from the flight management systems  180  (step S 202 ). In general, flight information refers to the state of the aircraft, such as present altitude, speed, pitch, roll and position, while the FMS information concerns more strategic flight information, such as planning, routing, etc. Based on this information, the priority unit  152  of the image sensor control unit  150  determines object priority. (step S 204 ). Such a priority will be application dependent and dynamic. For example, a threat such as impending terrain, obstruction, traffic, or enemy weapon location will be assigned high priority levels. For a task of landing an aircraft onto a runway, the runway will receive high priority when no immediate safety threat is present. 
         [0022]    The integrity unit  154  of the EV image sensor control unit  150  assigns integrity levels to objects of interest. (step S 206 ). Based on the assigned integrity, the sensor area optimizing unit  156  of the image sensor control unit  150  assigns a imaging area within the field of view of the synthetic vision system  120  (step S 208 ). The level of zooming is controlled by data integrity with human operator override. The EV image sensor control unit  150  outputs steering control signals to at least one EV image sensor  130  based on the assigned search area (step S 10 ). In this way, the EV image sensor  130  can be selectively controlled to effectively image an object or area of interest based on information from the SV system  120 . The size of the area for camera steering is a function of object integrity as determined by the integrity unit  154  of the EV image sensor control unit  150 , thereby providing a more useful display to the operator. The indexing/registration unit  160  indexes and aligns the images generated by the EV system  110  and the SV system  120  for output to the display  190  (steps S 212 , S 214 ). 
         [0023]      FIGS. 5A and 5B  illustrate exemplary images output to the display  190  in accordance with a first embodiment of the present invention. In the display image  291 , an EV display image  292  (e.g., video display) has a field of view that has been controlled as a function of the integrity level assigned to objects of interest of the SV system  120 . The EV display image  292  is indexed and registered with the SV image  292  to provide a useful interface for an aircraft operator. In  FIG. 5A , the object of interest has been assigned a lower integrity level, such that the focus area for the EV image is larger. In  FIG. 5B , the object of interest has been assigned a higher integrity level, such that the focus area for the EV image is smaller. 
         [0024]      FIG. 6  illustrates a vehicle display system according to a second embodiment of the present invention. As illustrated in  FIG. 6 , the vehicle display system  300  includes: a synthetic vision (SV) system  320 ; an enhanced vision (EV) system  310 ; an SV database  370 ; flight management systems  380 ; enhanced vision (EV) image sensor(s)  330 ; a display  390 ; an indexing/registration unit  360 ; and an EV image enhancement control unit  350 . Although various block diagram elements shown in  FIG. 6  are illustrated as discrete elements, this illustration is for use and explanation, and it should be recognized that certain elements may be combined in one or more physical devices, e.g., one or more microprocessor(s) with associated software. 
         [0025]      FIG. 7  is a block diagram illustrating elements of the EV image enhancement control unit  350  according to the second embodiment of the present invention. As illustrated in  FIG. 7 , the EV image enhancement control unit  350  includes: a priority unit  352 ; an integrity unit  354 ; and an enhancement area determining unit  356 . The priority unit  352  determines the priority of certain objects of interest based on information from the SV system  120 . For example, for EV image sensor  130  having a wide field of view (e.g., 30 degrees), the field of view may contain multiple objects of interest. According to this embodiment of the present invention, priority levels assigned to such objects of interest determine which area of the image generated by the EV image sensor  130  is to be enhanced first to bring out detailed features of the object of interest. The integrity unit  354  determines the integrity level of objects of interest based on information from the SV system  120 . 
         [0026]    Next, operation of the vehicle display system  300  of the second embodiment illustrated in  FIGS. 6 and 7  will be described with reference to the flow diagram of  FIG. 8 . Initially, the synthetic vision system  320  inputs flight and flight management system (FMS) information, including for example vehicle position, heading, attitude, flight plan, from the flight management systems  380  (step S 402 ). Based on this information, the priority unit  352  of the EV image enhancement control unit  450  determines object priority (step S 404 ). The integrity unit  354  of the EV image enhancement control unit  350  assigns integrity levels to objects of interest (e.g., targets) (step S 406 ). Based on the assigned integrity, the enhancement area determining unit  356  of the EV image enhancement control unit  350  assigns an enhancement area within the field of view of the an EV image sensor  330  (step S 408 ). The EV image enhancement control unit  350  outputs enhancement control signals to the EV system  310  so that image areas of the EV image can be selectively enhanced based on information from the synthetic vision system  320 . (step S 410 ). The size of the area for EV image enhancement is a function of object integrity as determined by the integrity unit  354  of the EV image enhancement control unit  350 , such that a relatively smaller area can be enhanced/emphasized/highlighted for high integrity objects of interest. The indexing/registration unit  360  aligns the images generated by the EV system  310  and the synthetic vision system  320  for output to the display  390  (step S 412 ) so that an enhanced EV images and SV images are output to the display  390  (step S 414 ). 
         [0027]      FIGS. 9A and 9B  illustrate exemplary display outputs to display  391  according to the second embodiment of the present invention. When applied to an aircraft environment, this embodiment of the present invention provides SV images  394  and EV images  392  that are specifically highlighted/enhanced around the area of the object of interest, whereby such EV image enhancement is performed as a function of object integrity so that a relatively smaller area is enhanced for high integrity objects thereby allowing flight crews to extract object/target information quicker and with improved certainty. In  FIG. 9A , the object of interest has been assigned a lower integrity level, such that the enhancement area for the object of interest in the EV view is larger. The enhancement may be a highlighted area (e.g., a color border around the object of interest) or digital image processing to improve the clarity/appearance of the image region containing the object of interest. In  FIG. 9B , the object of interest has been assigned a higher integrity level, such that the enhancement area for the object of interest is smaller. 
         [0028]    Although detailed embodiments and limitations to the present invention have been described above, it should be apparent that various modifications are possible without departing from the spirit and scope of the present invention.