Traffic sign detection and recognition with convolutional neural networks

Traffic sign detection and recognition in street scene images of a vehicle camera allow to localize the traffic signs in a street scene image and to classify them with regard to their semantic meaning for the car driver. In this contribution, a method is described to detect traffic signs in a street scene image by evaluating image patches, sampled by a sliding window approach, with a convolutional neural network detector. Robust shape fitting is performed on the image patch of a positive detection to obtain the exact position of the traffic sign shape in the patch. A second convolutional neural network is applied to the image patches centred on the fitted shapes to classify the meaning of these traffic signs. The networks are trained and tested with samples of traffic signs and other street scene objects from the GTSRB and GTSDB datasets. The results have shown that models for traffic sign detection and recognition can be trained with an overall accuracy of more than 90 % obtained for a test set. The position of a traffic sign, known from shape fitting, has been shown to be an important a-priori knowledge to select the appropriate image patch to ensure a high accuracy of the subsequent traffic sign recognition. 1 Safer roads by traffic sign warnings Modern advanced driver assistance systems are an important contribution to increase the safety on roads. An assistance system using traffic sign recognition can warn a car driver against a speed limit by means of optical hints (EURO NCAP 2013), for example. As well as a human driver, an assistance system needs to extract information from the environment around the car. Cars can therefore be equipped with sensors to perceive the environment. As the upcoming drive way and the traffic signs applicable for that way are in front of the vehicle, a front-looking sensor is suitable for traffic sign detection and recognition (e.g. LARSSON & FELSBERG 2011). Due to small costs compared to a Radar or LiDAR sensor (ETKBMW.COM), often an optical monocamera is the first choice for this task. Such a camera observing the environment acquires typically an image of the street scene including the street, traffic signs, buildings, traffic participants and further static and dynamic objects (Fig. 1). There exist algorithms for object detection which use the complete image of the street scene as input (e.g. REDMON et al. 2016). Other algorithms require an image patch centered on the object which should be detected (red rectangle in Fig. 1) as input (e.g. SERMANET & LECUN 2011). In the second case, the input image patch can be obtained by a sliding window approach sampling from the street scene image, for example. Traffic sign detection and traffic sign recognition have different objectives (ZHU et al. 2016): the objective of traffic sign detection is to decide whether the input image, or the image patch, shows 1 Technical University of Munich, Photogrammetry & Remote Sensing, Arcisstraße 21, D-80333 Munich, email: [alexander.hanel, stilla]@tum.de A. Hanel & U. Stilla 356 a traffic sign or another arbitrary object. In contrast, the objective of traffic sign recognition is typically to classify the meaning of a traffic sign, given an image patch from which it is known to show a traffic sign (e.g. German Traffic Sign Recognition Benchmark dataset, STALLKAMP et al. 2012). Fig. 1: Example image of a street scene recorded by a front-looking mono-camera. The image is an excerpt from the Cityscapes dataset (CORDTS et al. 2016), which can be used to train machine learning algorithms in the domain of advanced driver assistance systems, for example for traffic sign recognition In a recent work, ZHU et al. (2016) propose a method for simultaneous detection and recognition of traffic signs using a neural network-based deep learning approach. Their method provides the bounding box, a pixel-wise mask and the class label for each detection. Several examples (in supplementary material provided by these authors) show that the bounding boxes of partly occluded traffic signs tend to be smaller than the sign in the image is. Using the pixels of the mask to extract the shape contour of the traffic sign will therefore be prone to errors. HANEL & STILLA (2018) have recently proposed a method to calibrate a vehicle camera on public roads, where the exact shape of a traffic sign in the image has to be known precisely. Their method derives image points along the shape of a sign and the corresponding metric 3D position, for example known from governmental regulations, as reference points for calibration. In this contribution, a method for traffic sign detection and recognition is proposed, which delivers the correct shape of the sign in the image simultaneously. A sliding window approach is used to sample image patches from a street scene image. The samples are evaluated by a traffic sign detector, which is realized by a neural network-based deep learning approach. Deep approaches have shown to provide a higher accuracy than shallow learning approaches, as for example ZHU et al. 2016 mention. Mean shift clustering is used to reduce multiple detections of the same traffic. Such multiple detections with slightly different positions and sizes are typical for a sliding window approach (COMASCHI et al. 2013). The shape of the traffic sign in the image is obtained by fitting an appropriate geometric primitive (e.g. an ellipse for circular traffic signs) with RANSAC followed by a least-squares adjustment. The extracted shape is used to refine the image position of the patch, which is then used by another deep learning approach to classify the meaning of the traffic sign. 38. Wissenschaftlich-Technische Jahrestagung der DGPF und PFGK18 Tagung in München – Publikationen der DGPF, Band 27, 2018 357 2 Stepwise traffic sign detection and recognition By the sliding window approach, image patches are taken from an image with a higher geometric resolution (called ‘complete image’ in the following). In the scope of this contribution, the complete image is typically a street scene image. Each patch shows a different part of the complete image; the patch is shifted over the complete image and the complete image is resized to consider traffic signs at different image positions and with different sizes in the image. The deep network architecture (Fig. 2) of the traffic sign detector uses such an image patch as input sample. In contrast to architectures, which use the complete image as input, the sliding window approach allows to reduce the computational effort when tracking traffic signs in an image series by evaluating only the local neighbourhood around the predicted image position of a traffic sign in a subsequent image in contrast to evaluating the complete subsequent image. The traffic sign detector classifies each sample in either the class traffic sign or other objects. The architecture of our network as described in the following is a variation of the traffic sign detector architecture described by WU et al. (2013): the input sample with a fixed resolution of 32 x 32 pixels and RGB color channels is first convolved with different kernels to extract features. The “ReLu” activation function ensures the non-linearity property of the network to prevent that the network behaves just like a single-layer perceptron. Max pooling in the second and third layer reduces the resolution of the feature maps created by the previous convolutions. Dropout is applied to the third layer to avoid overfitting. The following layers are designed to classify the input patch based on the extracted features: the output of the feature extraction layers is flattened, followed by three fully-connected (FC) layers in which the number of neurons is reduced step-by-step. The last FC layer has two output neurons, one for each of the two classes to distinguish. Softmax loss is used. In contrast to by WU et al. (2013), we use no branching (for details on branching see the description of the traffic sign recognition architecture), but increase the depth of the classification part. ZHU et al. (2016) have mentioned that deeper networks perform better. According to these authors, branching could also increase the performance, but at the cost of a higher computational effort. The detector model is trained with a set of positive and negative samples. Image patches showing one traffic sign are called positive samples, while image patches showing arbitrary objects like buildings or vegetation are called negative samples. The class of an image patch with unknown content is predicted after training by feeding the patch together with the trained model into the Fig. 2: Architecture of the convolutional neural network used to distinguish between traffic signs and other objects shown in an image patch taken from a street scene image. A. Hanel & U. Stilla 358 neural network. Output of the prediction are the probabilities for both classes; the image patch is assigned to the class with the higher probability. Image patches from slightly different positions and sizes in the complete image are likely to show the same object. Therefore, it can be expected that several of such neighboring patches will be detected as traffic sign. To link all detections belonging to the same traffic sign, the detections are grouped to a single detection by mean shift clustering (FUKUNAGA & HOSTETLER 1975). Each detection is hereby represented by the central point of the image patch. Mean shift clustering groups the central points of all detections into several groups using the Euclidean distance as criterion to distinguish different groups. The size of the image patch belonging to a single detection is determined by the mean size of the patches of all contributing detections. An advantageous property of mean shift clustering is that it does not require the number of clusters to be determined manually, being therefore able to handle a varying number of traffic signs in different complete images. Fig.