Abstract:
Advanced driver assistance systems need to be able to recognize and to classify traffic signs under real time constraints, and under a wide variety of visual conditions. The invention shown employs binary masks extracted by color space segmentation, with a different binary mask generated for each sign shape. Temporal tracking is employed to add robustness to the detection system. The system is generic, and is trainable to the signs used in various countries.

Description:
CLAIM OF PRIORITY 
       [0001]    This application claims priority under 35 U.S.C. 119(e)(1) to Indian Provisional Application No. 201641000153 filed Jan. 4, 2016 
       TECHNICAL FIELD OF THE INVENTION 
       [0002]    The technical field of this invention is image processing. 
       BACKGROUND OF THE INVENTION 
       [0003]    Traffic sign recognition (TSR) is a technology which makes vehicles capable of recognizing the traffic signs appearing in the vicinity of the driving path. TSR systems form an important part of the ADAS (advanced driver assistance systems) that is currently being deployed in the cars of today. It is a classic example of rigid object detection. TSR systems depend on forward facing image sensors. Current TSR systems are aimed to assist the driver in the driving process. But in future, TSR systems will play a very crucial role in the functioning of autonomous cars. 
         [0004]    Computers face a lot of challenges in identifying traffic signs in images due to the following reasons:
       Within-class variability. The same traffic sign in the real world can give rise to different images due to:
           Different viewing positions and different distances between the camera and traffic sign positions,   Photometric effects: positions of multiple different light sources, their color, distribution of shadows, and view obstruction by objects present near the traffic signs.   Between-class similarity: different classes of traffic sign look may very much alike.   Background objects in cluttered urban environments also pose a challenge,   Motion blur in images   Faded, bent, dirty, sign boards   Adverse weather conditions like rain and snow.
 
Traffic signs may also be slightly different from country to country. For example, speed limit traffic signs in some European countries are round with red circle boundary, while in the US they are rectangular in shape.
   
               
 
       SUMMARY OF THE INVENTION 
       [0013]    A real time Traffic Sign Recognition (TSR) system is described comprising of a preprocessing stage to identify image regions containing a traffic sign, a localization stage accurately locate the sign within the image, a categorization stage to categorize the located sign into one of the sign categories, and a temporal smoothening stage remove noise and false detections due to noise. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]    These and other aspects of this invention are illustrated in the drawings, in which: 
           [0015]      FIG. 1  illustrates the s block diagram of the TSR system; 
           [0016]      FIG. 2  illustrates the segmentation stage of the TST system; 
           [0017]      FIG. 3  shows the flow chart of the ERST step; 
           [0018]      FIG. 4  illustrates the computed feature planes; 
           [0019]      FIG. 5  shows various positions of the models inside the image; 
           [0020]      FIG. 6  illustrates a depth=2 decision tree; 
           [0021]      FIG. 7  illustrates a block diagram of the temporal smoothening engine; and 
           [0022]      FIG. 8  shows a flow chart of the temporal smoothening engine. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0023]    A four stage TSR algorithm is shown as described below. It is also shown pictorially in  FIG. 1 . 
         [0024]    Stage  1 : Preprocessing Stage  101   
         [0025]    Identify the approximate image regions having traffic signs without missing any traffic sign in input images. 
         [0026]    Stage  2 : Accurate Localization Stage  102   
         [0027]    Stage  2   a : Extract features from input images in  103   
         [0028]    Stage  2   b : Accurate localization of the region of traffic sign within the image using classifier in  104 . 
         [0029]    Stage  3 : Classification Stage  105   
         [0030]    The windows localized by stage  2  are categorized into one of the categories. 
         [0031]    Stage  4 : Temporal Smoothening  106   
         [0032]    This stage is meant to remove the noisy detections and noisy classification that have been obtained from stage  3 . 
         [0033]    The preprocessing stage works on the input image and is aimed at reducing the complexity of TSR system by reducing the amount of data that is processed by subsequent stages. It is implemented in two steps: 
         [0034]    Extract color cues to find possible locations of traffic signs 
         [0035]    A shape detector uses these color cues to identify image locations having traffic signs. 
         [0036]    As shown in  FIG. 2 , the RGB input image to the preprocessing stage is spatially subsampled by a factor of four in both directions in  202 , and converted to YUV in  203 . This is to keep the complexity of segmentation stage under manageable limits for an embedded system implementation. 
         [0037]    Contrast stretching is done in  204  by using histogram equalization on the Y plane. This improves the performance of the algorithm in many low contrast input images. 
         [0038]    Red, Blue, Yellow and White binary masks are extracted by thresholding in YUV color space (1 mask for each color) in  205 . 
         [0039]    Morphological opening (erosion followed by dilation) is applied in  206  for each of these binary masks. 
         [0040]    The masks are combined in  207   
         [0041]    The binary masks are used by extended radial symmetry transform (ERST) in  208 . ERST detects circle, triangle, square and octagon in the input images by performing voting for the gradients present in regions of mask. 
         [0042]      FIG. 3  shows the flow chart of the ERST. 
         [0043]    In  301  a gradient map for entire image in grey scale is computed using Sobel operator. 
         [0044]    In  302 , the binary masks obtained from color space thresholding act as (color) cues for this stage. 
         [0045]    The gradients that are less than threshold are zeroed out in  303  and are not considered for later stages. 
         [0046]    The voting is performed in a 3D accumulator array(x,y,r)  304 . One 3D accumulator array is maintained for each shape (circle, square, triangle, and octagon). 
         [0047]    Voting (incrementing procedure of accumulator cells) is performed only for the gradient (edge) points for which the binary value in the mask is non-zero. 
         [0048]    After voting finishes for the entire image in  305 , the top ‘N’ peaks in each accumulator are used in  306  to determine the position and radius of the circle/polygon at that point. 
         [0049]    Feature extraction Stage  2   a  is performed by:
       For each input image, an image pyramid is prepared. The number of image scales used is dependent on:   Maximum and minimum traffic sign size to be detected   Input image dimensions   Complexity, accuracy tradeoff considerations   Aggregate Channel Features (ACF) planes are computed for every scale of each image, as shown in  FIG. 4 .   ACF is a collection of 10 feature planes of 3 channels comprising of original pixels of YUV space, 1 Gradient magnitude channel and 6 orientations channels of “histogram of oriented gradients (HOGs)”.   Each of HOG orientation channel used as a part of ACF is computed from a cell size of 4×4 pixels without any overlap between cells and without block normalization.       
 
         [0057]    Traffic sign localization Stage  2   b  is performed by: 
         [0058]    An ADA boost (Adaptive Boosting) classifier is used for this localization. Boosting is an approach to machine learning based on the idea of creating a highly accurate prediction rule by combining many relatively weak and inaccurate rules. 
         [0059]    1024 number of decision trees of depth  2  act as weak classifiers for ADA boost. A single weak classifier is depicted in  FIG. 6 . 
         [0060]    Features computed from 32×32 pixel blocks of images (known as a model) are used as inputs to the classifier. The model is made to step by 4 pixels (both horizontal and vertical) on each image and each scale, as shown in  FIG. 5 . At each position of the model 501 a feature vector of size 640 pixels is computed using the feature planes. 
         [0061]    Feature vectors obtained in this manner from training images are used for training the ADA boost classifier. Training is done in 4 stages with 32, 128, 256, 1024 weak classifiers used in each stage. Boot strapping is used in each stage to strengthen the hypothesis. 
         [0062]    The feature vector of size 640 pixels is fed to the ADA boost classifier. The ADA boost returns a real number which is binary thresholded to decide if TS is present or not. Note that localization procedure is only a binary decision procedure where it is decided if a traffic sign is present or not. Actual classification (categorization to specific class) is done in the next stage. 
         [0063]    Traffic sign classification Stage  3  is done by:
       The windows that are marked as containing traffic signs in Stage  2   b  are passed to next stage for categorization. Stage  2   b  is designed with maximum sensitivity in mind, i.e. no valid traffic sign should be missed but few false positives are acceptable. These false positives are filtered out by Stage  3 .   The feature vector used in Stage  3  is of size 2992 pixels and is used as an input to the Linear Discriminant Analysis (LDA) classifier.   LDA relies on minimization of Mahalanobis distance between a feature vector and the mean vector of various classes. The Mahalanobis distance of an observation       
 
         [0000]        x =( x   1   , x   2   , x   3   , . . . , x   N ) T          from a group of observations with mean         
         [0000]      μ=(μ 1 , μ 2 , μ 3 , . . . , μ N ) T  
       and covariance matrix S is defined as:       
 
         [0069]    Minimization of Mahalanobis distance is mathematically equivalent to minimization of the below function 
         [0000]        g   i ( x )= w   i   t   x+w   i0    
         [0000]    where g i (x)→cost function for class ‘i’
   w i →weight vector for class ‘i’   w i0 →bias for class ‘i’   x is vector of size 2992 pixels.   w i  and w i0  are pre-computed (during training) and are different for different classes. For a given feature vector x, g i (x) is computed for each class and the feature vector is associated with the class that gives the minimum value of the function g(x).   
 
         [0074]    Temporal smoothening Stage  4  is performed by: 
         [0075]    Removing the noisy detections and noisy classification that have been obtained from the earlier stages. This stage is present only when the input is a sequence of images that form a part of single video. 
         [0076]    The temporal smoothening engine is conceptually depicted in  FIG. 7 . The inputs to temporal smoothening engine are: 
         [0077]    The descriptors of detection windows  701  (position and dimensions) obtained from stage  2 . 
         [0078]    Class id&#39;s  702  that are associated with each of these detection windows obtained from stage  3 . 
         [0079]    The temporal smoothening engine internally maintains a history of the detection windows. This history is empty at the start of the sequence of pictures and is updated after every picture. The decision logic block inside the engine looks at the inputs and the history before finalizing the windows and its associated class. 
         [0080]    It uses the Jaccard coefficient to measure degree of similarity between windows detected in the current picture and the windows stored in the history. Jaccard coefficient J(A, B), between two windows A and B is defined as follows, 
         [0000]    
       
         
           
             
               J 
                
               
                 ( 
                 
                   A 
                   , 
                   B 
                 
                 ) 
               
             
             = 
             
               
                 
                    
                   
                     A 
                     ⋂ 
                     B 
                   
                    
                 
                 
                    
                   
                     A 
                     ⋃ 
                     B 
                   
                    
                 
               
               . 
             
           
         
       
     
         [0081]    The numerator term denotes the area under intersection and denominator denotes the area in the union of the two windows. 
         [0082]    The details of the temporal smoothening engine are shown in  FIG. 8 . Inputs to the temporal smoothing engine are the detection windows (det_win [])  801  output from stage  2  of the 
         [0083]    TSR algorithm, and the class id (id[])  802  for each detection window. In  803 , hist[] is the state memory that is built when a new picture is processed. The Jaccard coefficient is computed in  804  for every pair of windows, with one window selected from hist[] and the second from det_win[]. In  805  det_idx is set to zero, and in  806  the we find the hist[best_match_hist_idx] that gives the J, J_max when paired with det_win[det_idx]. If in  807  J_max is &gt;0.5, hist[best_match_hist_idx is stored into det_win[det_idx], and id[det_idx]is associated with the same entry of hist[] in  808 . If J_max is = or &lt; than 0.5 in  807 , det_win[best_match_det_idx] is added to hist[] as a new entry, and id[best_match_det_idx] is stored with the same entry of hist[] in  809 . In  810  we determine if all entries of det_win[] have been processed. If not, det_idx is incremented in  811 , and the flow returns to  806 . If all entries have been processed, all hist[] entries that have not been updated are deleted in  812 . 
         [0084]    The output of temporal smoothening engine in  813  and  814  is used as the final output of the TSR system.