Abstract:
A vehicle including a chassis, a drive system carrying the chassis, and a vision system carried by the chassis. The vision system having a stereo visible light camera producing a colorized 3D point cloud and a stereo long wave infrared camera producing 3D data. The vision system being configured to fuse the 3D data with the 3D point cloud thereby producing an enhanced 3D point cloud.

Description:
[0001]    This is a non-provisional application based upon U.S. provisional patent application Ser. No. 61/981,408 entitled “STEREO VISION FOR SENSING VEHICLES OPERATING ENVIRONMENT”, filed Apr. 18, 2014, which is incorporated herein by reference. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    1. Field of the Invention 
         [0003]    The present invention relates to vision systems for vehicles, and, more particularly, to stereo vision systems for vehicles and image processing of stereo images. 
         [0004]    2. Description of the Related Art 
         [0005]    Unmanned vehicle 3D perception systems have numerous military and commercial applications for performing in cluttered environments and detecting obstacles while mapping terrain. Several technical hurdles remain for this technology to be practical for widespread use. Current terrain detection solutions typically rely on active geometry sensing from LIDAR or RADAR sensors, all of which are easily detectable by enemy forces. Current stereo algorithms work using high signal to noise imagery and mainly use visible light cameras. LIDAR is a remote sensing technology that measures distance by illuminating a target with a laser and analyzing the reflected light. Although broadly LIDAR is used by many as an acronym of Light Detection And Ranging, the term LIDAR is also a commonly used military reference to “Laser Illuminated Detection And Ranging.” 
         [0006]    Current LIDAR range sensors provide a very high detailed, geometric view of the world and can be used to create 3D models of objects, but the use of LIDAR has severe limitations in certain applications, particularly in agriculture. The environment is often standing crop, which appears to LIDAR as obstacles, and the limitations at seeing through obscurants such as dust, snow, rain, smoke, etc. 
         [0007]    What is needed in the art is a vision system for a vehicle that efficiently extends the operating environment of the vehicle. 
       SUMMARY OF THE INVENTION 
       [0008]    The present invention provides an autonomous vehicle having a stereo vision system. 
         [0009]    The invention in one form is directed to a vehicle including a chassis, a drive system carrying the chassis, and a vision system carried by the chassis. The vision system having a stereo visible light camera producing a colorized 3D point cloud and a stereo long wave infrared camera producing 3D data. The vision system being configured to fuse the 3D data with the 3D point cloud thereby producing an enhanced 3D point cloud. 
         [0010]    The invention in another form is directed to a vision system, for use with a vehicle, having a stereo visible light camera producing a colorized 3D point cloud and a stereo long wave infrared camera producing 3D data. The vision system being configured to fuse the 3D data with the 3D point cloud thereby producing an enhanced 3D point cloud. 
         [0011]    The invention in yet another form is directed to a method of directing a vehicle using a vision system. The method includes the steps of: producing a colorized 3D point cloud with data from a stereo visible light camera; and fusing 3D data from a stereo long wave infrared camera with the 3D point cloud to thereby produce an enhanced 3D point cloud that is used to direct tasks of the vehicle. 
         [0012]    An advantage of the present invention is that objects that may be hidden by dust are resolved and detected so that the vehicle can react to the presence of the object. 
         [0013]    Another advantage is that the vehicle has a companion aerial vision system to gather data. 
         [0014]    Yet another advantage of the present invention is that the autonomous nature of the vehicle is leveraged to allow operation in otherwise marginal or hostile environments. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0015]    The above-mentioned and other features and advantages of this invention, and the manner of attaining them, will become more apparent and the invention will be better understood by reference to the following description of an embodiment of the invention taken in conjunction with the accompanying drawings, wherein: 
           [0016]      FIG. 1  illustrates an image taken with a Short Wavelength Infrared (SWIR) camera and a visible light camera image of the same scene, to illustrate a need for the fusion of data provided by the present invention; 
           [0017]      FIG. 2  is a Long Wave Infrared (LWIR) image with thermally radiant objects being clearly visible; 
           [0018]      FIG. 3  illustrates an infrared stereo algorithm data pipeline of the present invention; 
           [0019]      FIG. 4  is an aerial image captured on the DJI 550 platform; 
           [0020]      FIG. 5  is another aerial image captured on the DJI 550 platform; 
           [0021]      FIG. 6  is an image of items laying about a table; 
           [0022]      FIG. 7  is the image of  FIG. 6  that is processed by the present invention to show edge features highlighted; 
           [0023]      FIG. 8  is a stereo infrared image of a road scene; 
           [0024]      FIG. 9  is the same scene as  FIG. 8  with the sparse features being identified and used by the present invention to populate the 3D data cloud; 
           [0025]      FIG. 10  illustrates the use of segmenting point clouds using 3d and texture, with the left image being a view of the scene, with the right image showing a segmented scene projected onto an image plane with the object shaded in different intensities; 
           [0026]      FIG. 11  illustrates a plane interpolation algorithm of the present invention being used to resolve depth of wall planes that lack texture; 
           [0027]      FIG. 12  also illustrates a plane interpolation algorithm being used to resolve depth of wall planes that lack texture; 
           [0028]      FIG. 13  is a schematized view of a vehicle having a vision system of the present invention associated therewith; and 
           [0029]      FIG. 14  is a block diagram of the vision system of the present invention. 
       
    
    
       [0030]    Corresponding reference characters indicate corresponding parts throughout the several views. The exemplification set out herein illustrates one embodiment of the invention, in one form, and such exemplification is not to be construed as limiting the scope of the invention in any manner. 
       DETAILED DESCRIPTION OF THE INVENTION 
       [0031]    Referring now to the drawings, there is illustrated (in  FIG. 13  in schematized form) a vehicle  10  having a chassis  12 , a drive system  14  and a vision system  16 . Vision system  16  provides information to systems of vehicle  10 , such as a controller that controls drive system  14  so that vehicle  10  can be operated in an autonomous fashion. 
         [0032]    As represented in  FIG. 14 , vision system  16  includes an IR stereo system, such as a stereo LWIR camera  18  and/or NIR sensor  20 , which provide valuable information for classifying objects as vegetation, and for seeing through vegetation for more substantial and important targets, such as animals, people, machines, etc. Vision system  16  additionally includes a stereo RGB camera  22 . A controller  24  can receive data from stereo LWIR camera  18 , NIR sensor  20  and stereo RGB camera  22  to undertake the processing of image data received therefrom using algorithms of the present invention. With this increased sensing capability, many farming operations become accessible, and safe for vehicle automation. The present invention, among other things, provides a feature and object detection system that maintains functionality in low light scenarios and can detect obstacles within vegetation. Allowing for improved detection of obstacles for agriculture/mining applications, which allow for more effective safety and autonomy of systems. 
         [0033]    Near Infrared (NIR) and Short Wavelength Infrared (SWIR) cameras operate on the edge of the visible light spectrum from the wavelength of 0.75 to 3 μm. The infrared radiation in this spectrum interacts with objects in a similar manner as the visible wavelengths. This makes the images from the NIR and SWIR similar to the visible light images in resolution and detail. The main difference is that the infrared images are not in color. Energy from within these wavelengths must usually be reflected in order to obtain good imagery. This means that there must be some external illumination and at night, these cameras typically require some type of artificial illumination.  FIG. 1  illustrates a SWIR image from sensor  20  and a picture of the same scene from camera  22 , which is the visible light image to show the ability to detect objects with a combination of images. 
         [0034]    Long Wavelength Infrared cameras  18  operate on the far end of the infrared spectrum from the wavelength of 8 to 15 μm. The Long Wavelength Infrared (LWIR) cameras band is called the thermal band. Cameras in this range can be completely passive. This means that they require no external illumination. In  FIG. 2  there is shown a LWIR image with thermally radiant objects being clearly visible. The infrared cameras react to thermal energy, so this makes them optimal for use at night and conditions like dust, fog, or smoke. The LWIR camera  18  is highly tolerable to such conditions. 
         [0035]    Although there has been significant work in the area of visible light stereo depth perception, there has been little research into infrared spectrum stereo vision. This may be partially due to inaccessibility of sensors in previous years due to cost, as well as experimentation on older sensors that failed to perform using dense stereo computation due to high noise levels. With the cost reductions of IR sensors in recent years there have been further attempts at computing range from infrared pairs; most common methods have been for pedestrian detection and have used a priori models of objects (humans) and blob detection to perform depth computation. The present invention uses sparse algorithms to determine a set of feature correspondences based on edge features and a segmentation/interpolation algorithm is used to fill in depth regions between sparse points. This algorithm is more adaptable per specific camera and allows cameras with noisier images to still be functional, where dense algorithms will fail. The algorithm is very suitable to overhead imagery where edge data is prominent and especially in urban environments where terrain is structured and surfaces between edges tend be to linear. 
         [0000]    
       
         
               
             
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 System capabilities 
               
             
          
           
               
                 Features 
                 Summary 
               
               
                   
               
               
                 Terrain Sensing and Mapping based on 
                 Rely on edge based features, interpolate 
               
               
                 passive stereo imagery with low signal to 
                 segmented regions 
               
               
                 noise ratio 
               
               
                 Can use typical aerial data from urban and 
                 Data collection done locally using ground 
               
               
                 rural settings 
                 system and/or quad rotor assigned thereto. 
               
               
                 Use of many existing sensors for range, 
                 Sensor data fusion 
               
               
                 field of view, resolution, frame rate and 
               
               
                 power 
               
               
                 Used on various scales of Unmanned 
                 Sensor performance of various levels which 
               
               
                 Vehicles, with a target of low cost and 60 
                 can be integrated with the vision system 
               
               
                 degree field of view 
               
               
                 Uses low light sensors and machine vision 
                 Use vision algorithms with embedded stereo 
               
               
                 stereo algorithms 
                 processor 
               
               
                 Produces accurate real time terrain data 
                 Use a vehicle vision system with algorithms 
               
               
                   
                 that operate at 1 Hz refresh rate 
               
               
                 Robust software system 
                 Algorithms have a high noise tolerance and 
               
               
                   
                 scalability, modular to camera selection and 
               
               
                   
                 stereo baseline 
               
               
                   
               
             
          
         
       
     
         [0036]      FIG. 3  illustrates an infrared stereo algorithm data pipeline having steps that include an image grabbing function  30 , a pre-filter  32 , a feature extraction  34 , a matching and consistency validation  36 , segmentation and dense computation  38 , a post filter  40  and logging  42  capability. 
         [0037]    The vision system of the present invention can be used with existing robots (ground or aerial) to model terrain and detect features/objects. 
         [0038]    Sensors were studied to determine relevant sensors that could be used in a stereo camera system, with an emphasis on tradeoffs for quality, cost, weight etc.
   Sensors are selected for the vision system using existing hardware and baseline sensors.   The sensors allow data collection: to collect IR imagery of terrain data   Feature extraction and evaluation, determine which features can be used for sufficient pixel correspondence in infrared image pairs   The present invention uses a stereo algorithm using infrared imagery to resolved depth data of terrain, with the algorithm being tuned to perform with the best results at real time   These features are explained in more detail in the following sections. The features and summary of requirements is contained in Table 1.   
 
         [0044]    A survey was performed of commercially available IR camera modules and the selection of prime candidates for use in the three desired scale classes was done. It was important to identify sensors which allow for stereo vision processing; that is the cameras allow for hardware synchronization between cameras, have a global shutter to handle motion, allow for digital interfacing, have rugged properties for military and off-road vehicle usage, allow for short exposure time to minimize motion blur and allow for proper calibration between cameras (cameras must have very similar properties), and not require extensive external components for cooling or illumination. Further, signal to noise characteristics were used to help quantize camera quality for comparison. 
         [0045]    The stereo algorithm functions in the desired environment, using sensors selected and hardware bread-boarded because of the ease of accessibility for collecting data. The present invention uses an embedded processor platform suitable for real time stereo algorithms that are used to interface to the IR camera and collect terrain images. 
         [0046]    Representative data sets that match the requirements for targets and expected environments were collected. Data sets were gathered around testing grounds (buildings, varying brush types, varying tree leaf densities, coniferous and deciduous trees). Each data set was captured from a moving vehicle such that detection range could be investigated. 
         [0047]    For the aerial photography, a quad-rotor platform was used (see  FIGS. 4 and 5  of aerial images capture on the DJI 550 platform). This facilitates the stereo imagery capture of the lighter stereo system. RTK-corrected GPS data is captured on each target of interest. Having the geo-tagged target data, the present invention measures the accuracy of the stereo point clouds and the processing algorithms. 
         [0048]    In addition to aerial analysis, the technology of unmanned ground vehicles (UGVs) is leveraged. Several automated tractors are used as platforms for rural data acquisition, and an automated Ford F-150 was used for urban collection. While some of the urban settings may not easily lend themselves to automated testing (for safety concerns), all of the rural testing is automated. This eliminates variability in the test data from human path-following inconsistencies. By collecting multiple data sets, there is increased statistical significance of the findings, and the automation decreases the variability between each data set. Automating the testing also helps increase safety, especially for testing in low-light/night conditions. 
         [0049]    Correspondence feature extraction and matching algorithm: The main software development of the present invention assessed which vision feature type is most effective for pixel correspondence of noisy IR images. Log-Gabor edge type features are sufficient for good correspondence, but newer features such as a Ferns algorithm, Binary Robust Independent Elementary Features (BRIEF), and Speeded Up Robust Features (SURF) improve accuracy/speed/coverage tradeoffs of processed terrain images collected. The present invention has a stereo algorithm framework (see  FIG. 6  of an image—that is processed in  FIG. 7  to show edge features highlighted) that allows for methodologies to be rapidly integrated and uses interfaces to OpenCV (Open Source Computer Vision) such that desired feature types are available for analysis. 
         [0050]    Sparse features can be used to detect obstacles in Infrared imagery using thermal cameras (see  FIGS. 8 and 9 ). Sparse feature correspondence algorithms of the present invention use known camera constraints to remove outlier matches and ensure consistency between images. The algorithms find correspondence in left/right infrared images. The horizontal lines in  FIG. 9 , indicate matches between images, in this example Speeded Up Robust features are used, and consistency is checked between frames. Appling the known camera intrinsic and the camera baselines, the 3D location of each of the circles at the end of the lines can be determined. 
         [0051]    Sparse to Depth Computation, segmentation and hole filling: Because a sparse stereo algorithm won&#39;t completely generate an estimate of depth for every pixel, a secondary hole-filling algorithm is used. For this task, a segmentation algorithm is used to partition overhead terrain and a reconstruction algorithm performs in real time on an embedded processor. A Mean shift segmentation algorithm and a linear plane fit interpolation algorithm are used to fill empty regions ( FIG. 10 ). In  FIG. 10  segmenting point clouds using 3d and texture are illustrated with the left image being a view of the scene, with the right image showing a segmented scene projected onto an image plane with the object shaded in different intensities. In  FIGS. 11 and 12  a plane interpolation algorithm is used to resolve depth of wall planes that lack texture. 
         [0052]    The sensing and perception systems on vehicle  10  are to collect data such that the vehicle can interact with its environment. One of the data types/formats that appears to be most useful is colorized 3D point clouds. This is typically achieved by overlaying a camera image on top of range data. The combination of image analysis (edge/blob detection) with detecting features in the 3D range data allows for better reliability in the detection and identification of features. 
         [0053]    RGB cameras and laser sensors have significant problems seeing through obscurants such as dust or smoke. In agriculture dust is a common problem. Sensing and detecting clouds of dust can be of interest, but we are typically more interested in the features hidden behind a dust cloud. The present invention provides sensing systems that can penetrate dust clouds and provide information on the objects/features hidden behind dust clouds. While Radar is very good at penetrating dust, it typically only detects macro features (large scale features) so it is normally very difficult to get higher resolution information that provides useful information on smaller features. Typically the wavelength of the energy is proportional to how well it can penetrate obscurants. There is a relationship between obscurant particle sizes and how well a certain wavelength can penetrate, so higher wavelengths (e.g. LWIR) are typically better at penetrating obscurants. By using a stereo LWIR camera data is used to fill in the 3D data where dust obscures certain fields of vision of other sensing devices (RGB cameras, laser scanners, etc.). 
         [0054]    There is also a known ratio between energy levels in the NIR/SWIR and red spectrum ranges for detection of chlorophyll. The present invention can use this to detect foliage, which is abundantly present in many agricultural environments. The present invention measures and compares near infrared and red spectrum energy level ratios. This can be used to help differentiate plants from other objects (rock vs shrub). For this reason it is useful to have an NIR sensor included in the sensing system of the present invention. 
         [0055]    In summary the present invention would have and provide: 
         [0056]    A Stereo RGB camera (provide colorized 3D point cloud) 
         [0057]    A Stereo LWIR camera (fill in 3D data where obscurants/dust is present, and detection of warm blooded creatures) 
         [0058]    An NIR sensor (for foliage/plant detection) 
         [0059]    The algorithms fuse data from these 3 sensors to provide an Enhanced 3D point cloud that will allow the software to make decisions with higher levels of confidence. 
         [0060]    While this invention has been described with respect to at least one embodiment, the present invention can be further modified within the spirit and scope of this disclosure. This application is therefore intended to cover any variations, uses, or adaptations of the invention using its general principles. Further, this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains and which fall within the limits of the appended claims.