Abstract:
Advanced driver assistance systems need to be able to operate under real time constraints, and under a wide variety of visual conditions. The camera lens may be partially or fully obstructed by dust, road dirt, snow etc. The invention shown extracts high frequency components from the image, and is operable to classify the image as being obstructed or non-obstructed.

Description:
CLAIM OF PRIORITY 
       [0001]    This application claims priority under 35 U.S.C 119(e)(1) to Provisional Application No. 62274525 filed Jan. 4, 2016. 
     
    
     TECHNICAL FIELD OF THE INVENTION 
       [0002]    The technical field of this invention is image processing, particularly to detect if the view of a fixed focus camera lens is obstructed by surface deposits (dust, road dirt, etc). 
       BACKGROUND OF THE INVENTION 
       [0003]    The fixed focus cameras used for Advanced Driver Assistance Systems (ADAS) are subject to many external conditions that may make the lens dirty from time to time. Car manufacturers are starting to design intelligent self-cleaning cameras that can detect dirt and automatically clean the lens using air or water. 
         [0004]    One of the difficulties encountered in the prior art is the reliable detection of foreign objects such as dust, road dirt, snow, etc., obscuring the lens while ignoring large objects that are part of the scene being viewed by the cameras. 
       SUMMARY OF THE INVENTION 
       [0005]    The solution shown applies to fixed focus cameras, widely used in automotive for ADAS applications. The problem solved by this invention is distinguishing a scene obscured by an obstruction, such as illustrated in  FIG. 1 , from a scene having large homogeneous areas, such as illustrated in  FIG. 2 . In accordance with this invention the distinction is made based upon the picture data produced by the camera. Obstructions created by deposits on a lens surface, as shown in  FIG. 1 , will appear blurred and will have predominantly low frequency content. A high pass filter may therefore be used to detect the obstructions. 
         [0006]    A machine-learning algorithm is used to implement classification of the scene in this invention. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    These and other aspects of this invention are illustrated in the drawings, in which: 
           [0008]      FIG. 1  shows a partially obstructed scene due to an obstruction on the lens; 
           [0009]      FIG. 2  shows the same scene without an obstruction of the lens; 
           [0010]      FIG. 3  shows a block diagram of the functions performed according to this invention; 
           [0011]      FIG. 4  shows the scene of  FIG. 2  divided into a grid of blocks; 
           [0012]      FIG. 5  is a graphical representation of a feature vector; 
           [0013]      FIG. 6  is a graphical representation of a sample cost function for the case of a one dimensional feature vector; and 
           [0014]      FIG. 7  shows a processor operable to implement this invention. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0015]    The steps required to implement the invention are shown in  FIG. 3 . The input image is first divided into a grid of N×M blocks in step  301 .  FIG. 4  illustrates the scene of  FIG. 2  divided into a 3×3 set of blocks. 
         [0016]    In step  302  the high frequency content of each block is computed by using horizontal and vertical high pass filters. This produces a total of 2×M×N values. 
         [0017]    The reason for separately processing 3×3 (9) different regions of the image instead of the entire image is to calculate the standard deviation of the values across the image. The classified of this invention uses both mean and standard deviation values. Employing only the mean value could be sufficient to detect scenarios where the entire view is blocked but cannot prevent false-positive cases where one part of the image is obstructed and other parts are perfectly fine. The mean value cannot measure the high-frequency&#39;s contrast between different regions whereas the standard deviation can. 
         [0018]    Step  303  then calculates the mean and the standard deviation for each high pass filter, across M×N values to form a 4 dimensional feature vector. Step  304  is an optional step that may augment the features vector an additional P component. This additional component may be meta information such as image brightness, temporal differences, etc. 
         [0019]    Step  305  then classifies the scene as obscured or not obscured using a logistic regression algorithm having the feature vector as its input. This algorithm is well suited for binary classifications such as pass/fail, win/lose, or in this case blocked/not blocked. 
         [0020]    This algorithm performs well where the two classes can be separated by a decision boundary in the form of a linear equation. Classification is shown in  FIG. 5 , where: 
         [0021]    If θ 0 +θ 1 ·x 1 +θ 2 ·x 2≧ 0
       then the (x 1 ,x 2 ) sample belongs to the X class  501  (image blocked) illustrated in  FIG. 5 ,
 
and
       
 
         [0023]    If θ 0 +θ 1 ·x 1 +θ 2 ·x 2 &lt;0
       then the (x 1 ,x 2 ) sample belongs to the O class  502  (image clear) illustrated in  FIG. 5 .       
 
         [0025]    In this invention the line is parametrized by θ=[θ 0 , θ 1 , θ 2 ] since the feature vector has two components x 1  and x 2 . The task of the logistic regression is to find the optimal θ, which will minimize the classification error for the images used for training. In the case of scene obstruction detection, the feature vectors have 4 components [x 1 , x 2 , x 3 , x 4 ] and thus the decision boundary is in form of an hyperplane with parameters [θ 0 , θ 1 , θ 2 , θ 3 , θ 4 ]. 
         [0026]    The training algorithm determines the parameter θ=[θ 0 ,θ 1 ,θ 2  . . . ] by performing the following tasks: 
         [0027]    Gather all feature vectors into a matrix X and the corresponding classes into a vector Y. 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         X 
                         = 
                         
                           { 
                           
                             
                               
                                 
                                   X 
                                   1 
                                   0 
                                 
                               
                               
                                 
                                   X 
                                   1 
                                   1 
                                 
                               
                               
                                 … 
                               
                               
                                 
                                   X 
                                   1 
                                   
                                     M 
                                     - 
                                     1 
                                   
                                 
                               
                             
                             
                               
                                 
                                   X 
                                   2 
                                   0 
                                 
                               
                               
                                 
                                   X 
                                   2 
                                   1 
                                 
                               
                               
                                 … 
                               
                               
                                 
                                   X 
                                   2 
                                   
                                     M 
                                     - 
                                     1 
                                   
                                 
                               
                             
                             
                               
                                 
                                   X 
                                   3 
                                   0 
                                 
                               
                               
                                 
                                   X 
                                   3 
                                   1 
                                 
                               
                               
                                 … 
                               
                               
                                 
                                   X 
                                   3 
                                   
                                     M 
                                     - 
                                     1 
                                   
                                 
                               
                             
                             
                               
                                 
                                   X 
                                   4 
                                   0 
                                 
                               
                               
                                 
                                   X 
                                   4 
                                   1 
                                 
                               
                               
                                 … 
                               
                               
                                 
                                   X 
                                   4 
                                   
                                     M 
                                     - 
                                     1 
                                   
                                 
                               
                             
                           
                           } 
                         
                       
                     
                   
                   
                     
                       
                         = 
                         
                           [ 
                           
                             
                               X 
                               0 
                             
                              
                             
                               X 
                               1 
                             
                              
                             
                                 
                             
                              
                             … 
                              
                             
                                 
                             
                              
                             
                               X 
                               
                                 M 
                                 - 
                                 1 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                 
               
               
                 
                   = 
                   
                     [ 
                     
                       
                         X 
                         0 
                       
                        
                       
                         X 
                         1 
                       
                        
                       
                           
                       
                        
                       … 
                        
                       
                           
                       
                        
                       
                         X 
                         
                           M 
                           - 
                           1 
                         
                       
                     
                     ] 
                   
                 
               
             
             
               
                 
                   Y 
                   = 
                   
                     
                       [ 
                       
                         
                           y 
                           0 
                         
                          
                         
                           y 
                           1 
                         
                          
                         
                             
                         
                          
                         … 
                          
                         
                             
                         
                          
                         
                           y 
                           
                             M 
                             - 
                             1 
                           
                         
                       
                       ] 
                     
                      
                     
                         
                     
                      
                     where 
                      
                     
                         
                     
                      
                     
                       y 
                       k 
                     
                      
                     
                         
                     
                      
                     is 
                      
                     
                         
                     
                      
                     0 
                      
                     
                         
                     
                      
                     or 
                      
                     
                         
                     
                      
                     1. 
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
     
         [0028]    Find θ=[θ 0 , θ 1 , θ 2 , θ 3 , θ 4 ] that minimizes the cost function: 
         [0000]    
       
         
           
             
               
                 
                   
                     J 
                      
                     
                       ( 
                       Θ 
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       M 
                     
                      
                     
                       
                         ∑ 
                         
                           k 
                           = 
                           0 
                         
                         
                           M 
                           - 
                           1 
                         
                       
                        
                       
                         Cost 
                          
                         
                           ( 
                           
                             
                               
                                 h 
                                 Θ 
                               
                                
                               
                                 ( 
                                 
                                   X 
                                   k 
                                 
                                 ) 
                               
                             
                             , 
                             
                               y 
                               k 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
             
             
               
                 
                   
                     with 
                      
                     
                       : 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       Cost 
                        
                       
                         ( 
                         
                           
                             
                               h 
                               Θ 
                             
                              
                             
                               ( 
                               
                                 X 
                                 k 
                               
                               ) 
                             
                           
                           , 
                           
                             y 
                             k 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         
                           - 
                           
                             y 
                             k 
                           
                         
                          
                         
                           log 
                            
                           
                             ( 
                             
                               
                                 h 
                                 Θ 
                               
                                
                               
                                 ( 
                                 
                                   X 
                                   k 
                                 
                                 ) 
                               
                             
                             ) 
                           
                         
                       
                       - 
                       
                         
                           ( 
                           
                             1 
                             - 
                             
                               y 
                               k 
                             
                           
                           ) 
                         
                          
                         
                           log 
                            
                           
                             ( 
                             
                               1 
                               - 
                               
                                 
                                   h 
                                   Θ 
                                 
                                  
                                 
                                   ( 
                                   
                                     X 
                                     k 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                         
                          
                         
                             
                         
                          
                         and 
                       
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         h 
                         Θ 
                       
                        
                       
                         ( 
                         
                           X 
                           k 
                         
                         ) 
                       
                     
                     = 
                     
                       1 
                       
                         1 
                         + 
                         
                           e 
                           
                             
                               - 
                               
                                 Θ 
                                 T 
                               
                             
                              
                             
                               x 
                               k 
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
         [0029]      FIG. 6  shows the graphical representation of a sample cost function J(θ) for the case of a one dimensional feature vector. 
         [0030]    Gradient descent is one of the techniques to find the optimum θmin which minimizes J(θ). 
         [0031]    If for θmin we have Jθmin=0, this means the error rate for the classifier, when applied to the training data set, is 0%. However most of the time J(θmin)&gt;0, which means there is some miss-classification error that can be quantified. 
         [0032]    Next the algorithm&#39;s miss-classification error (also called accuracy) is calculated by applying the classifier rule to every feature vector of the dataset and comparing the results with the true result. 
         [0033]    The final classification is done as follows: 
         [0034]    If θ 0 +θ 1 ·x 1 +θ 2 ·x 2 ≧0
       then the image is blocked,
 
and
       
 
         [0036]    If θ 0 +θ 1 ·x 1 +θ 2 ·x 2 &lt;0
       then the image is clear.       
 
         [0038]      FIG. 7  illustrates an example system-on-chip (SOC)  700  suitable for this invention. SOC  700  includes general purpose central processing unit (CPU)  701 , digital signal processor (DSP)  702 , graphics processing unit (GPU)  703 , video input ports  704 , internal memory  705 , display controller subsystem  706 , peripherals  707  and external memory controller  708 . In this example, all these parts are bidirectionally connected to a system bus  709 . General purpose central processing unit  701  typically executes what is called control code. Control code is what gives SOC  700  its essential character generally in the way it interacts with the user. Thus CPU  701  controls how SOC  700  responds to user inputs (typically received via peripherals  707 ). DSP  702  typically operates to process images and real-time data. These processes are typically known as filtering. The processes  FIG. 3  are performed by DSP  702 . GPU  703  performs image synthesis and display oriented operations used for manipulation of the data to be displayed. Video input ports  704  receive the input images from possibly plural cameras. Video input ports  704  typically also includes suitable buffering of the image data prior to processing. Internal memory  705  stores data used by other units and may be used to pass data between units. The existence of memory  705  on SOC  700  does not preclude the possibility that CPU  701 , DSP  702  and GPU  703  may include instruction and data cache. Display controller subsystem  706  generates the signals necessary to drive the external display used by the system. Peripherals  707  may include various parts such as a direct memory access controller, power control logic, programmable timers and external communication ports for exchange of data with external systems (as illustrated schematically in  FIG. 7 ). External memory controller  708  controls data movement into and out of external memory  710 . 
         [0039]    A typical embodiment of this invention would include non-volatile memory as a part of external memory  710 . The instructions to control SOC  700  to practice this invention are stored the non-volatile memory part of external memory  710 . As an alternate, these instruction could be permanently stored in non-volatile memory part of external memory  710 .