# An Efficient Color Space for Deep-Learning Based Traffic Light Recognition

Authors: Hyun-Koo Kim; Ju H. Park; Ho-Youl Jung
Journal: Journal of Advanced Transportation
 (2018)

---

## Abstract

Traffic light recognition is an essential task for an advanced driving assistance system (ADAS) as well as for autonomous vehicles. Recently, deep-learning has become increasingly popular in vision-based object recognition owing to its high performance of classification. In this study, we investigate how to design a deep-learning based high-performance traffic light detection system. Two main components of the recognition system are investigated: the color space of the input video and the network model of deep learning. We apply six color spaces (RGB, normalized RGB, Ruta’s RYG, YCbCr, HSV, and CIE Lab) and three types of network models (based on the Faster R-CNN and R-FCN models). All combinations of color spaces and network models are implemented and tested on a traffic light dataset with 1280×720 resolution. Our simulations show that the best performance is achieved with the combination of RGB color space and Faster R-CNN model. These results can provide a comprehensive guideline for designing a traffic light detection system.


---

## Body

## 1. Introduction

Over the past few years, various advanced driving assistance system (ADAS) have been developed and commercialized. In particular, most automotive companies are now doing their best to launch autonomous vehicles as soon as possible. According to the society of automotive engineers (SAE), the international standard defining the six levels of driving automation requires the autonomous driving to achieve level 3 and higher [1]. Obviously, the traffic light recognition is an essential task for ADAS as well as for autonomous vehicle.For traffic light recognition, various methods have been proposed. These can be analyzed from three aspects such as color space, feature extraction, and verification/classification. Different color spaces, namely, gray scale [2, 3], RGB [4, 5], normalized RGB [6], Ruta’s RGB [7], YCbCr [8, 9], HSI [10], HSV [11, 12], HSL [13], and CIE Lab [14], have been used. Moreover, some studies [15–18] have used more than one color spaces. For feature extraction, Haralick’s circularity measure [19], Sobel edge detection [20], circle Hough transform [21], 2D Gabor wavelets [22], Haar-likes [23], histogram of oriented gradients (HOG) [24], and geometric features [25] have been applied. For verification/classification, various conventional classifiers have been used, e.g., k-means clustering [26], template matching [27], 2D independent component analysis (ICA) [28], linear discriminant analysis (LDA) [29], decision-tree classifier, k-nearest neighbor (kNN) classifier [30], adaptive boosting algorithm (Adaboost) [31], and support vector machine (SVM) [32]. Recently, some basic deep-learning networks such as LeNet [33], AlexNet [34], and YOLO [35, 36] have been applied to traffic light recognition. Other approaches using visual light road-to-vehicle communication have been developed. LED-typed traffic lights broadcast the information, then photo-diode [37, 38] or high-frame-rate image sensor [39, 40] receives the optical signal. In this paper, we mainly focus on vision-based traffic light recognition using deep-learning.In the last couple of years, deep-learning has achieved a remarkable success in various artificial intelligence research areas. In particular, deep-learning has become very popular in vision-based object recognition due to its high performances of classification. One of the first advances is OverFeat that applies the convolutional neural network (CNN) [34] to multiscale sliding window algorithm [41]. Girshick et al. proposed a region with CNN (R-CNN), which achieves up to almost 50% improvement on the object detection performance [42]. In R-CNN, object candidate regions are detected and features are extracted using CNN, while objects are classified using SVM. Girshick proposed a Fast R-CNN, which uses selective search to generate object candidates and applies fully connected neural network to classify the objects [43]. However, the selective search algorithm slows down the object detection system performance. Redmon et al. proposed YOLO, which uses a simple CNN approach to achieve real-time processing by enhancing detection accuracy and reducing computational complexity attaining [35]. Ren et al. proposed a Faster R-CNN which replaces the selective search by region proposal network (RPN) [44]. The RPN is a fully convolutional network that simultaneously predicts the object bounds and object/objectless scores at each position. This method makes it possible to implement the end-to-end training. Recently, two notable deep-learning network models were proposed, single shot detector (SSD) and region-based fully convolutional networks (R-FCN). SSD uses multiple sized convolutional feature maps to achieve a better accuracy and higher speed than YOLO [45]. R-FCN is a modified version of Faster R-CNN, which consists of only convolutional networks [46]. It is to be noted that the feature extraction is included in deep-learning detection network in the cases of Fast R-CNN, YOLO, Faster R-CNN, SSD, and R-FCN frameworks. The above-mentioned deep-learning methods have been widely applied to detect objects such as vehicle and pedestrian [47–51]. However, only a few deep-learning based network models have been applied to traffic light detection system [52–55].From the viewpoints of color representation, various color spaces of input video data have been used in conventional traffic light recognition methods. However, only a few color spaces have been applied in deep-learning based methods. Because color information plays an important role in the performance of traffic light detection, it is necessary to select the color space carefully in deep-learning based methods. In this study, we focus on how to design a high-performance deep-learning based traffic light recognition system. To find color space most suitable to deep-learning based traffic light recognition, six color spaces such as RGB, normalized RGB, Ruta’s RYG, YCbCr, HSV, and CIE Lab are investigated. For deep-learning network models, three models based on the Faster R-CNN and R-FCN are applied. All combinations of color spaces and network models are implemented and compared.The rest of this paper is organized as follows. Second section discusses the previous research works on traffic light detection system. In third section, we describe various color spaces and deep-learning network models. All combinations of color spaces and network models have been designed in this study. In fourth section, we explain the configurations such as parameter and data set for the performance evaluation. Fifth section presents the simulation results. Final section draws the conclusions.


## 2. Related Works

In this section, we briefly introduce the work done so far on traffic light detection. These works are categorized into two groups, namely, deep-learning based and conventional classification methods, depending on whether the deep-learning is used or not. They are investigated mainly from the viewpoints of color representation and verification/classification. The analysis is summarized in Table1.Table 1
Color space and verification/classification used in previous traffic light detection.
Ref. # Color space Verification / Classification [2], [3] Gray-scale Template matching [4] RGB K-means clustering, Circularity check [5] RGB Region growing, Color segmentation [6] Normalized RGB Color segmentation, Circle Hough transform [7] Ruta’s RGB SVM [8] YCbCr Adaboost [9] YCbCr Decision-tree classifier [10] HSI Gaussian mask, Existence-Weight Map [11] HSV Template matching [14] CIE Lab Fast radial symmetry transform [12] HSV SVM [13] HSL SVM [15] Normalized RGB, RGB Color clustering [16] Normalized RGB, RGB Fuzzy logic clustering [17] RGB, YCbCr Nearest neighbor classifier [18] RGB, HSV LDA, kNN, SVM [53] CIE Lab SVM, LeNet, AlexNet [52] HSV SVM, Simple CNN [54] RGB YOLO v1 [55] RGB YOLO 9000


### 2.1. Conventional Classification Based Methods

In general, conventional traffic light recognition methods mainly consist of two steps, candidate detection, and classification. Various color representations have been used. Charette and Nashashibi did not use any color information [2, 3]. They proposed to use the gray-scale image as input data. After the top-hat morphological filtering, adaptive template matching with geometry and structure information was applied to detect the traffic lights. Park and Jeong used color extraction and k-means clustering for candidate detection [4]. The average and standard deviation of each component in RGB color space were then calculated and used. Here, Haralick’s circularity was used for verification. Yu et al. used the difference of each pair of components in RGB space to extract the dominant color [5]. They applied region growing and segmentation for candidate detection. For verification, the information of shape and position was used. Omachi and Omachi used normalized RGB for color segmentation [6]. The edge detection and circle Hough transform were applied for verification of traffic light. Kim et al. used Ruta’s RGB based color segmentation for the detection of traffic light candidates at night [7]. Some geometric and stochastic features were extracted and used in SVM classifier. Kim et al. used YCbCr color-based thresholding and shape filtering for candidate detection [8]. Here, Haar-like features and Adaboost were used for classification. Kim et al. used YCbCr color segmentation for candidate detection [9]. Here, candidate blobs with red and green lights were detected by thresholding Cb and Cr components. Various shape and modified Haar-like features were extracted and used in decision-tree classifier. Siogkas et al. used the CIE Lab color space [14], where the multiplications of L and a components (RG), and L and b components (YB) are used to enhance the discrimination of red and green regions. They used fast radial symmetry transform and persistency to identify the color of traffic lights. Cylindrical color spaces such as HSI, HSV, and HSL have also been used [10–13]. Hwang et al. used HSI color-based thresholding, morphological filtering, and blob labeling for candidate detection [10]. For verification, they used convolution of the candidate region with Gaussian mask using existence-weight map. HSV color space was also used, where the histograms of hue and saturation components are used for candidate extraction [11]. Probabilistic template matching was applied for classification. Recently, it has been reported that the detection performance of traffic lights can be improved by using the 3D geometry map that are prebuild from GPS, INS/IMU, and range sensors such as stereo camera or 2D/3D range lidar [12, 13]. Jang et al. used Haar-like feature based Adaboost with 3D map information for candidate detection [12]. HOG and HSV color histogram were applied to SVM classifier. Moreover, traffic light candidates are detected by using HOG features based linear SVM classifier with the uncertainty of 3D prior that constrains the search regions [13]. For classification, image color distribution in HSL color space is used.Some researchers used two color spaces [15–18]. Omachi and Omachi used RGB and normalized RGB color spaces to find candidates, and circle Hough transform was applied for verification [15]. Combination of RGB and normalized RGB was also used for color segmentation based on fuzzy logic clustering, where some geometric and stochastic features were used as primary clues to discriminate traffic lights from others [16]. Cai et al. used RGB and YCbCr color spaces for candidate extraction and classification, respectively [17]. Gabor wavelet transform and ICA based features were extracted and applied to the nearest neighbor classifier. Furthermore, red, yellow and green traffic light regions are detected by thresholding based HSV color segmentation and geometrical features [18]. HOG features were extracted in RGB space and used to determine whether arrow sign is on the light or not. Three different classification algorithms such as LDA, kNN, and SVM were applied, respectively.Since the selection of color space plays the most important role in traffic light detection performance, past studies have explored all the possible options. Clearly, it is important to identify the best among all these color spaces.


### 2.2. Deep-Learning Based Methods

Deep-learning has been also used for traffic light detection and classification [52–55]. At first, deep learning was applied only in the classification of traffic lights, where candidates were detected by conventional method [52, 53]. Saini et al. used HSV color space-based color segmentation, aspect ratio, and area-based analysis and maximally stable extremal region (MSER) to localize the candidates [52]. HOG features and SVM were used for verification, whereas simple CNN was used for classification. Lee and Park used CIE Lab color space-based segmentation to find the candidate regions [53]. To reduce false regions, they use SVM with size, aspect ratio, filling ratio and position. The classification was performed by two cascaded CNN which consists successively of LeNet and AlexNet. LeNet quickly differentiates between traffic lights and background. AlexNet classifies traffic light types. Recently, deep-learning has been applied both to candidate detection and classification. Behrendt et al. [54] and Jensen et al. [55] applied YOLO-v1 [35] and YOLO-9000 [36] for traffic light detection/classification.As discussed, only a few color spaces have been applied in deep-learning based traffic light detection. Because color information plays an important role in the performance of detection, it is necessary to select the color space carefully. It is also required to apply more sophisticated and efficient deep-learning network models to traffic light detection and classification.


## 2.1. Conventional Classification Based Methods

In general, conventional traffic light recognition methods mainly consist of two steps, candidate detection, and classification. Various color representations have been used. Charette and Nashashibi did not use any color information [2, 3]. They proposed to use the gray-scale image as input data. After the top-hat morphological filtering, adaptive template matching with geometry and structure information was applied to detect the traffic lights. Park and Jeong used color extraction and k-means clustering for candidate detection [4]. The average and standard deviation of each component in RGB color space were then calculated and used. Here, Haralick’s circularity was used for verification. Yu et al. used the difference of each pair of components in RGB space to extract the dominant color [5]. They applied region growing and segmentation for candidate detection. For verification, the information of shape and position was used. Omachi and Omachi used normalized RGB for color segmentation [6]. The edge detection and circle Hough transform were applied for verification of traffic light. Kim et al. used Ruta’s RGB based color segmentation for the detection of traffic light candidates at night [7]. Some geometric and stochastic features were extracted and used in SVM classifier. Kim et al. used YCbCr color-based thresholding and shape filtering for candidate detection [8]. Here, Haar-like features and Adaboost were used for classification. Kim et al. used YCbCr color segmentation for candidate detection [9]. Here, candidate blobs with red and green lights were detected by thresholding Cb and Cr components. Various shape and modified Haar-like features were extracted and used in decision-tree classifier. Siogkas et al. used the CIE Lab color space [14], where the multiplications of L and a components (RG), and L and b components (YB) are used to enhance the discrimination of red and green regions. They used fast radial symmetry transform and persistency to identify the color of traffic lights. Cylindrical color spaces such as HSI, HSV, and HSL have also been used [10–13]. Hwang et al. used HSI color-based thresholding, morphological filtering, and blob labeling for candidate detection [10]. For verification, they used convolution of the candidate region with Gaussian mask using existence-weight map. HSV color space was also used, where the histograms of hue and saturation components are used for candidate extraction [11]. Probabilistic template matching was applied for classification. Recently, it has been reported that the detection performance of traffic lights can be improved by using the 3D geometry map that are prebuild from GPS, INS/IMU, and range sensors such as stereo camera or 2D/3D range lidar [12, 13]. Jang et al. used Haar-like feature based Adaboost with 3D map information for candidate detection [12]. HOG and HSV color histogram were applied to SVM classifier. Moreover, traffic light candidates are detected by using HOG features based linear SVM classifier with the uncertainty of 3D prior that constrains the search regions [13]. For classification, image color distribution in HSL color space is used.Some researchers used two color spaces [15–18]. Omachi and Omachi used RGB and normalized RGB color spaces to find candidates, and circle Hough transform was applied for verification [15]. Combination of RGB and normalized RGB was also used for color segmentation based on fuzzy logic clustering, where some geometric and stochastic features were used as primary clues to discriminate traffic lights from others [16]. Cai et al. used RGB and YCbCr color spaces for candidate extraction and classification, respectively [17]. Gabor wavelet transform and ICA based features were extracted and applied to the nearest neighbor classifier. Furthermore, red, yellow and green traffic light regions are detected by thresholding based HSV color segmentation and geometrical features [18]. HOG features were extracted in RGB space and used to determine whether arrow sign is on the light or not. Three different classification algorithms such as LDA, kNN, and SVM were applied, respectively.Since the selection of color space plays the most important role in traffic light detection performance, past studies have explored all the possible options. Clearly, it is important to identify the best among all these color spaces.


## 2.2. Deep-Learning Based Methods

Deep-learning has been also used for traffic light detection and classification [52–55]. At first, deep learning was applied only in the classification of traffic lights, where candidates were detected by conventional method [52, 53]. Saini et al. used HSV color space-based color segmentation, aspect ratio, and area-based analysis and maximally stable extremal region (MSER) to localize the candidates [52]. HOG features and SVM were used for verification, whereas simple CNN was used for classification. Lee and Park used CIE Lab color space-based segmentation to find the candidate regions [53]. To reduce false regions, they use SVM with size, aspect ratio, filling ratio and position. The classification was performed by two cascaded CNN which consists successively of LeNet and AlexNet. LeNet quickly differentiates between traffic lights and background. AlexNet classifies traffic light types. Recently, deep-learning has been applied both to candidate detection and classification. Behrendt et al. [54] and Jensen et al. [55] applied YOLO-v1 [35] and YOLO-9000 [36] for traffic light detection/classification.As discussed, only a few color spaces have been applied in deep-learning based traffic light detection. Because color information plays an important role in the performance of detection, it is necessary to select the color space carefully. It is also required to apply more sophisticated and efficient deep-learning network models to traffic light detection and classification.


## 3. Deep-Learning Based Traffic Light Detection

In this section, we present a deep-learning based traffic light detection system that consists of the preprocessing, deep-learning based detection, and postprocessing as shown in Figure1. In preprocessing, the input video data is transformed to other color space. Six color spaces are considered. The deep-learning based detection uses ensemble of feature extraction network model and detection one. Here, we consider three kinds of network models based on the Faster R-CNN and R-FCN which can perform localization and classification. In postprocessing, redundant detection is removed by using the nonmaximum suppression (NMS) technique [56, 57].Figure 1
Method overview of deep-learning based traffic light detection.We focus on the combination of the color spaces and the ensemble networks which can achieve high performance of the traffic light detection. The color spaces and ensemble network models to be considered are described below.


### 3.1. Color Spaces

In the vision-based object detection and classification, it is necessary to determine the color space in which the characteristic of the object appears well. The RGB color space is defined by the three chromatics, red (R), green (G), and blue (B) [58]. For robustness under changes in the lightning condition, normalized RGB has often been used. The normalized RGB denoted as Rn, Gn, and Bn are obtained by R/S, G/S, and B/S, respectively, where S=R+G+B [59]. At low illumination, it is difficult to distinguish between the normalized RGB colors [60]. To overcome this difficulty, Ruta et al. proposed the new red and blue color transform for traffic sign detection [60]. Since the traffic lights are red, green, and yellow, we modify Ruta’s color representation for the same. Ruta’s red, green, and yellow, denoted as fR, fy, and fG are obtained as below.(1)fR=max⁡0,min⁡Rn-Gn,Rn-Bn(2)fY=max⁡0,min⁡Rn-Bn,Gn-Bn(3)fG=max⁡0,min⁡Gn-Rn,Gn-BnThe YCbCr color space is obtained from the RGB [61].Y component is luma signal, and Cb and Cr are chroma components. The color space can also be represented in cylindrical coordinates such as HSV color space [62]. The hue component,H, refers to the pure color it resembles. All tints, tones, and shades of red have the same hue. The saturation, S, describes how white the color is. The value component, V, also called lightness, describes how dark the color is. The CIE Lab color space consists of one component for luminance, L, and two color components, a and b [63]. It is known that the CIE Lab space is more suitable to many digital image manipulations than RGB color space.In this paper, the six kinds of color spaces are considered in preprocessing of the traffic light detection system as shown in Figure1. Each color representation is applied and its performance is compared.


### 3.2. Deep-Learning Based Ensemble Networks

It is known that the end-to-end trainable deep-learning models are more efficient than other models in general object detection [35, 36, 44–46], because it allows a sophisticated training by sharing the weights between feature extraction and detection. YOLO [35, 36], Faster R-CNN [44], SSD [45], and R-FCN [46] have been developed for the end-to-end model. In our traffic light detection system, we only consider the end-to-end deep-learning network models that can perform feature extraction and detection.According to COCO [64], a dataset is divided into three groups depending on the size of the object to be detected; small (area<322), medium (322≤area≤962), and large (962≤area), where area denotes the number of pixels the object occupies. Therefore, the detection performance of a system can be different for different object sizes. Traffic lights are relatively smaller in size than other objects such as vehicle and pedestrian. For example, almost 90 % of traffic lights in our evaluation dataset belong to small-size group (area<322) as shown in Table 2. Therefore, it is necessary to determine a deep-learning network model which is suitable for small-size object detection.Table 2
The number (%) of small, medium, and large sizes traffic lights dataset.
Types # of small ( % ) # of medium ( % ) # of large ( % ) Total green 7,192 (86.83) 1,091 (13.17) 0 (0.00) 8,283 red 4,694 (95.37) 226 (4.59) 2 (0.04) 4,922 yellow 652 (92.22) 55 (7.78) 0 (0.00) 707 red left 1,429 (82.17) 308 (17.71) 2 (0.12) 1,739 green left 225 (75.50) 62 (20.81) 11 (3.69) 298 off 1,031 (89.42) 122 (10.58) 0 (0.00) 1,153 Total 15,223 (89.01) 1,864 (10.90) 15 (0.09) 17,102Huang et al. applied various network models to general object detection using COCO dataset and their performances are compared [65]. Fourteen kinds of meta-architectures with feature extractors and network models are analyzed. Five feature extractors such as VGGNet [66], MobileNet [67], Inception-v2 [68], Resnet-101 [47], and Inception-Resnet-v2 [69] are compared. Three kinds of network models based on the Faster-RCNN, R-FCN, and SSD are compared. They show that SSD (similar to YOLO) has higher performance for medium and large sized objects, but significantly lower performance than Faster R-CNN and R-FCN for small objects. They show that three ensemble networks such as Faster-RCNN with Inception-Resnet-v2, Faster R-CNN with Resnet-101, and R-FCN with Resnet-101 have higher performances than others for the small-size object detection. This is the reason why these three ensemble networks are applied in our traffic light detection method.In Faster R-CNN [44], the selective search is replaced by very small convolutional network called RPN to generate regions of interest (RoI). To handle the variations in aspect ratio and scale of objects, Faster R-CNN introduces the idea of anchor boxes. At each location, three kinds of anchor boxes are used for scale 128×128, 256×256, and 512×512. Similarly, three aspect ratios 1:1, 2:1, and 1:2 are used. RPN predicts the probability of being background or foreground for nine anchor boxes at each location. The remaining network is similar to the Fast-RCNN model. It is known that Faster-RCNN is 10 times faster than Fast-RCNN while maintaining a similar accuracy level [44].R-FCN [46] is a region-based object detection framework leveraging deep fully convolutional networks. In contrast to other region-based detectors such as Fast R-CNN and Faster R-CNN that apply per-region subnetwork hundreds of times, the region-based detector of R-FCN uses fully convolutional network that applies on the entire image. Instead of RoI pooling at the end layer of Faster R-CNN, R-FCN uses position-sensitive score maps and position-sensitive RoI pooling layer to address a dilemma between translation-invariance in image classification and translation-variance in object detection.The fully convolutional image classifier backbones, such as Resnet-101 [47] and Inception-Resnet-v2 [69], can be used for object detection. Resnet [47] is a residual learning framework to make the training easy for deeper neural network. It is reported that the residual network with 101 layers (Resnet-101) has the best performance for object classification [47]. Inception-Resnet-v2 [69] is a hybrid inception version which combines residual network and inception network.In this paper, three kinds of deep-learning based ensemble network models are considered for the traffic light detection (see Figures2–4). The first network model, Faster-RCNN with Inception-Resnet-v2, consists of Inception-Resnetv2 for feature extraction, RPN for candidate extraction, and RoI pooling of Fast R-CNN for classification. The second one, Resnet-101 with Faster R-CNN, consists of Resnet-101 for feature extraction, RPN, and RoI pooling. R-FCN with Resnet-101 consists of Resnet-101, RPN, and position-sensitive score map and position-sensitive RoI pooling for classification. Each network model is applied and its performance is compared.Figure 2
Block diagram of Faster R-CNN network model with Inception-Resnet-v2.Figure 3
Block diagram of Faster R-CNN network model with Resnet-101.Figure 4
Block diagram of R-FCN network model with Resnet-101.


## 3.1. Color Spaces

In the vision-based object detection and classification, it is necessary to determine the color space in which the characteristic of the object appears well. The RGB color space is defined by the three chromatics, red (R), green (G), and blue (B) [58]. For robustness under changes in the lightning condition, normalized RGB has often been used. The normalized RGB denoted as Rn, Gn, and Bn are obtained by R/S, G/S, and B/S, respectively, where S=R+G+B [59]. At low illumination, it is difficult to distinguish between the normalized RGB colors [60]. To overcome this difficulty, Ruta et al. proposed the new red and blue color transform for traffic sign detection [60]. Since the traffic lights are red, green, and yellow, we modify Ruta’s color representation for the same. Ruta’s red, green, and yellow, denoted as fR, fy, and fG are obtained as below.(1)fR=max⁡0,min⁡Rn-Gn,Rn-Bn(2)fY=max⁡0,min⁡Rn-Bn,Gn-Bn(3)fG=max⁡0,min⁡Gn-Rn,Gn-BnThe YCbCr color space is obtained from the RGB [61].Y component is luma signal, and Cb and Cr are chroma components. The color space can also be represented in cylindrical coordinates such as HSV color space [62]. The hue component,H, refers to the pure color it resembles. All tints, tones, and shades of red have the same hue. The saturation, S, describes how white the color is. The value component, V, also called lightness, describes how dark the color is. The CIE Lab color space consists of one component for luminance, L, and two color components, a and b [63]. It is known that the CIE Lab space is more suitable to many digital image manipulations than RGB color space.In this paper, the six kinds of color spaces are considered in preprocessing of the traffic light detection system as shown in Figure1. Each color representation is applied and its performance is compared.


## 3.2. Deep-Learning Based Ensemble Networks

It is known that the end-to-end trainable deep-learning models are more efficient than other models in general object detection [35, 36, 44–46], because it allows a sophisticated training by sharing the weights between feature extraction and detection. YOLO [35, 36], Faster R-CNN [44], SSD [45], and R-FCN [46] have been developed for the end-to-end model. In our traffic light detection system, we only consider the end-to-end deep-learning network models that can perform feature extraction and detection.According to COCO [64], a dataset is divided into three groups depending on the size of the object to be detected; small (area<322), medium (322≤area≤962), and large (962≤area), where area denotes the number of pixels the object occupies. Therefore, the detection performance of a system can be different for different object sizes. Traffic lights are relatively smaller in size than other objects such as vehicle and pedestrian. For example, almost 90 % of traffic lights in our evaluation dataset belong to small-size group (area<322) as shown in Table 2. Therefore, it is necessary to determine a deep-learning network model which is suitable for small-size object detection.Table 2
The number (%) of small, medium, and large sizes traffic lights dataset.
Types # of small ( % ) # of medium ( % ) # of large ( % ) Total green 7,192 (86.83) 1,091 (13.17) 0 (0.00) 8,283 red 4,694 (95.37) 226 (4.59) 2 (0.04) 4,922 yellow 652 (92.22) 55 (7.78) 0 (0.00) 707 red left 1,429 (82.17) 308 (17.71) 2 (0.12) 1,739 green left 225 (75.50) 62 (20.81) 11 (3.69) 298 off 1,031 (89.42) 122 (10.58) 0 (0.00) 1,153 Total 15,223 (89.01) 1,864 (10.90) 15 (0.09) 17,102Huang et al. applied various network models to general object detection using COCO dataset and their performances are compared [65]. Fourteen kinds of meta-architectures with feature extractors and network models are analyzed. Five feature extractors such as VGGNet [66], MobileNet [67], Inception-v2 [68], Resnet-101 [47], and Inception-Resnet-v2 [69] are compared. Three kinds of network models based on the Faster-RCNN, R-FCN, and SSD are compared. They show that SSD (similar to YOLO) has higher performance for medium and large sized objects, but significantly lower performance than Faster R-CNN and R-FCN for small objects. They show that three ensemble networks such as Faster-RCNN with Inception-Resnet-v2, Faster R-CNN with Resnet-101, and R-FCN with Resnet-101 have higher performances than others for the small-size object detection. This is the reason why these three ensemble networks are applied in our traffic light detection method.In Faster R-CNN [44], the selective search is replaced by very small convolutional network called RPN to generate regions of interest (RoI). To handle the variations in aspect ratio and scale of objects, Faster R-CNN introduces the idea of anchor boxes. At each location, three kinds of anchor boxes are used for scale 128×128, 256×256, and 512×512. Similarly, three aspect ratios 1:1, 2:1, and 1:2 are used. RPN predicts the probability of being background or foreground for nine anchor boxes at each location. The remaining network is similar to the Fast-RCNN model. It is known that Faster-RCNN is 10 times faster than Fast-RCNN while maintaining a similar accuracy level [44].R-FCN [46] is a region-based object detection framework leveraging deep fully convolutional networks. In contrast to other region-based detectors such as Fast R-CNN and Faster R-CNN that apply per-region subnetwork hundreds of times, the region-based detector of R-FCN uses fully convolutional network that applies on the entire image. Instead of RoI pooling at the end layer of Faster R-CNN, R-FCN uses position-sensitive score maps and position-sensitive RoI pooling layer to address a dilemma between translation-invariance in image classification and translation-variance in object detection.The fully convolutional image classifier backbones, such as Resnet-101 [47] and Inception-Resnet-v2 [69], can be used for object detection. Resnet [47] is a residual learning framework to make the training easy for deeper neural network. It is reported that the residual network with 101 layers (Resnet-101) has the best performance for object classification [47]. Inception-Resnet-v2 [69] is a hybrid inception version which combines residual network and inception network.In this paper, three kinds of deep-learning based ensemble network models are considered for the traffic light detection (see Figures2–4). The first network model, Faster-RCNN with Inception-Resnet-v2, consists of Inception-Resnetv2 for feature extraction, RPN for candidate extraction, and RoI pooling of Fast R-CNN for classification. The second one, Resnet-101 with Faster R-CNN, consists of Resnet-101 for feature extraction, RPN, and RoI pooling. R-FCN with Resnet-101 consists of Resnet-101, RPN, and position-sensitive score map and position-sensitive RoI pooling for classification. Each network model is applied and its performance is compared.Figure 2
Block diagram of Faster R-CNN network model with Inception-Resnet-v2.Figure 3
Block diagram of Faster R-CNN network model with Resnet-101.Figure 4
Block diagram of R-FCN network model with Resnet-101.


## 4. Configuration for Evaluation

In this section, we introduce the dataset and data augmentation method for traffic light detection, parameter tuning, and measurement metrics.


### 4.1. Dataset and Data Augmentation

For the simulations, we use Bosch Small Traffic Lights Dataset (BSTLD) offered by Behrendt et al. [54]. To use the same types of traffic lights both for training and test, we use only training data set of BSTLD which consists of 5,093 images. Among them, 2,042 images containing 4,306 annotated traffic lights are randomly selected and used as the test data set. The training data set consists of 6,102 images containing 12,796 annotated traffic lights. For testing set, 3,051 images are obtained from BSTLD training set and the others are generated using the following data augmentation techniques.(i)
Additional Noise and Blur. Random addition of Gaussian, speckle, salt and pepper noise, and generation of an image with signal-dependent Poisson noise.(ii)
Brightness Changes in the Lab Space. Addition of random values to luminance (lightness) component.(iii)
Saturation and Brightness Changes in the HSV Space. Additive jitter which is generated at random by means of exponentiation, multiplication and addition of random values to the saturation and value channels.Both training and testing data sets consist of 1,280×720 size images with annotations including bounding boxes of traffic lights as well as the current state of each traffic light. An active traffic light is annotated by one of six kinds of traffic light states (green, red, yellow, red left, green left, and off) as shown in Figure5. Detail descriptions of training and testing data sets are summarized in Table 3.Table 3
Descriptions of training and testing sets.
Dataset # of images # of annotated traffic lights Ratio Training set 6,102 12,796 (Total classes: 6 ea) 75 % (1) green (6,152) (2) red (3,730) (3) yellow (526) (4) red left (1,294) (5) green left (240) (6) off (854) Testing set 2,042 4,306 (Total classes: 6 ea) 25 % (1) green (2,131) (2) red (1,192) (3) yellow (181) (4) red left (445) (5) green left (58) (6) off (299)Figure 5
Types of traffic lights.


### 4.2. Parameter Tuning for Training

All three ensemble networks are trained until a maximum of 20,000 epochs using the pretrained weights are obtained from the COCO dataset [64]. The Faster R-CNN and R-FCN networks are trained by stochastic gradient descent (SGD) with momentum [70, 71], where the batch size is 1 and the momentum optimizer value is 0.9. We manually tune the learning rate schedules individually for each feature extractor. In our implementation, the tuning parameters of learning rate for SGD with momentum optimizer are set as follows:(i)
Initial learning rate: 0.0003(ii)
Learning rate of 0 ≤ Step < 900,000: 0.0003(iii)
Learning rate of 900,000 ≤ Step < 1,200,000: 0.00003(iv)
Learning rate of 1,200,000 ≤ Step: 0.000003As suggested by Huang et al. [65], we limit the number of proposals to 50 in all three networks to attain similar speeds of traffic light detection.


### 4.3. Measurement Metrics

To evaluate the performances of traffic light detection, we use measurement metrics such as average precision (AP), mean average precision (mAP), overall AP, and overall mAP that have been widely used in VOC challenge [72, 73] and the COCO 2015 detection challenge [74].AP is precision averaged across all values of recall between 0 and 1. Here, AP is calculated by averaging the interpolated precision over eleven equally spaced interval of recall value[0, 0.1, 0.2, …0.9, 1.0] [75]. To evaluate the performance for two or more classes, the average of AP, mAP is calculated by averaging APs over every class. We also use overall AP and overall mAP that are obtained by averaging APs and mAPs, respectively, over the IoU=[0.5, 0.55, 0.60, …, 0.90, 0.95], where IoU stands for interval of intersection over union [73].


## 4.1. Dataset and Data Augmentation

For the simulations, we use Bosch Small Traffic Lights Dataset (BSTLD) offered by Behrendt et al. [54]. To use the same types of traffic lights both for training and test, we use only training data set of BSTLD which consists of 5,093 images. Among them, 2,042 images containing 4,306 annotated traffic lights are randomly selected and used as the test data set. The training data set consists of 6,102 images containing 12,796 annotated traffic lights. For testing set, 3,051 images are obtained from BSTLD training set and the others are generated using the following data augmentation techniques.(i)
Additional Noise and Blur. Random addition of Gaussian, speckle, salt and pepper noise, and generation of an image with signal-dependent Poisson noise.(ii)
Brightness Changes in the Lab Space. Addition of random values to luminance (lightness) component.(iii)
Saturation and Brightness Changes in the HSV Space. Additive jitter which is generated at random by means of exponentiation, multiplication and addition of random values to the saturation and value channels.Both training and testing data sets consist of 1,280×720 size images with annotations including bounding boxes of traffic lights as well as the current state of each traffic light. An active traffic light is annotated by one of six kinds of traffic light states (green, red, yellow, red left, green left, and off) as shown in Figure5. Detail descriptions of training and testing data sets are summarized in Table 3.Table 3
Descriptions of training and testing sets.
Dataset # of images # of annotated traffic lights Ratio Training set 6,102 12,796 (Total classes: 6 ea) 75 % (1) green (6,152) (2) red (3,730) (3) yellow (526) (4) red left (1,294) (5) green left (240) (6) off (854) Testing set 2,042 4,306 (Total classes: 6 ea) 25 % (1) green (2,131) (2) red (1,192) (3) yellow (181) (4) red left (445) (5) green left (58) (6) off (299)Figure 5
Types of traffic lights.


## 4.2. Parameter Tuning for Training

All three ensemble networks are trained until a maximum of 20,000 epochs using the pretrained weights are obtained from the COCO dataset [64]. The Faster R-CNN and R-FCN networks are trained by stochastic gradient descent (SGD) with momentum [70, 71], where the batch size is 1 and the momentum optimizer value is 0.9. We manually tune the learning rate schedules individually for each feature extractor. In our implementation, the tuning parameters of learning rate for SGD with momentum optimizer are set as follows:(i)
Initial learning rate: 0.0003(ii)
Learning rate of 0 ≤ Step < 900,000: 0.0003(iii)
Learning rate of 900,000 ≤ Step < 1,200,000: 0.00003(iv)
Learning rate of 1,200,000 ≤ Step: 0.000003As suggested by Huang et al. [65], we limit the number of proposals to 50 in all three networks to attain similar speeds of traffic light detection.


## 4.3. Measurement Metrics

To evaluate the performances of traffic light detection, we use measurement metrics such as average precision (AP), mean average precision (mAP), overall AP, and overall mAP that have been widely used in VOC challenge [72, 73] and the COCO 2015 detection challenge [74].AP is precision averaged across all values of recall between 0 and 1. Here, AP is calculated by averaging the interpolated precision over eleven equally spaced interval of recall value[0, 0.1, 0.2, …0.9, 1.0] [75]. To evaluate the performance for two or more classes, the average of AP, mAP is calculated by averaging APs over every class. We also use overall AP and overall mAP that are obtained by averaging APs and mAPs, respectively, over the IoU=[0.5, 0.55, 0.60, …, 0.90, 0.95], where IoU stands for interval of intersection over union [73].


## 5. Simulation Results

In this section, we analyze the simulation results and detection examples. For the evaluation, we use measurement metrics such as overall mAP, overall AP, mAP, and AP. For analysis of the detection examples, we apply NMS.


### 5.1. Simulation Results

Every eighteen methods combined with six different color spaces and three network models are implemented and compared. Tables4 and 5 show the detection performances where every combination methods are listed in the left columns. In the tables, bold and underlined numbers indicate the top-ranked method, bold for the second ranked and underlined for the third ranked.Table 4
Detection performances (overall mAP and overall AP) of combination methods on test set.
Combination Method Overall mAP ( % ) Overall AP ( % ) Ensemble Network Model Color Space total small non small green red yellow red left green left off Faster R-CNN with Inception-Resnet-v2 RGB 20.40 15.85 36.15 33.46 23.81 4.75 34.69 17.59 8.08 Normalized RGB 19.81 15.16 38.10 32.15 22.29 6.06 38.28 11.43 8.65 Ruta’s RYG 18.07 13.54 33.33 28.58 20.05 2.39 35.30 17.98 4.11 YCbCr 16.50 12.71 31.31 29.51 15.25 4.67 31.17 14.33 4.07 HSV 19.70 15.41 37.06 29.23 16.91 6.74 36.00 23.54 5.77 CIE Lab 17.64 13.31 34.30 26.62 18.27 5.41 34.63 15.82 5.09  Faster R-CNN with Resnet-101 RGB 19.24 14.67 37.91 31.21 20.73 3.79 36.92 14.34 8.44 Normalized RGB 17.57 13.54 32.86 29.70 18.20 4.87 33.67 11.99 6.98 Ruta’s RYG 14.72 11.21 28.42 26.55 16.62 4.71 26.27 10.14 4.05 YCbCr 12.36 9.49 25.02 24.03 10.02 2.83 26.36 8.84 2.05 HSV 15.76 11.11 32.24 25.07 14.77 5.64 23.06 17.99 8.01 CIE Lab 10.90 7.63 23.73 19.98 13.79 3.67 20.43 5.28 2.28  R-FCN with Resnet-101 RGB 16.63 11.85 37.27 28.47 13.00 4.92 30.19 18.32 4.85 Normalized RGB 14.50 10.95 29.97 23.57 14.41 2.50 27.87 14.59 4.08 Ruta’s RYG 14.21 10.33 26.66 20.89 9.08 3.01 32.75 13.77 5.72 YCbCr 13.06 9.44 25.05 21.43 10.01 2.50 24.49 14.63 5.28 HSV 14.66 10.59 29.52 25.40 9.99 3.17 28.39 15.23 5.78 CIE Lab 12.24 9.06 23.51 14.58 12.43 1.93 27.87 11.80 4.85Table 5
Detection performances (mAP@0.5 and AP@0.5) of combination methods on test set.
Combination Method mAP@0.5 ( % ) AP@0.5 ( % ) Ensemble Network Model Color Space total small non small green red yellow red left green left off Faster R-CNN with Inception-Resnet-v2 RGB 38.48 31.27 57.79 70.56 52.12 8.49 59.11 27.13 13.44 Normalized RGB 38.24 31.42 59.87 70.43 52.09 10.98 63.94 17.39 14.60 Ruta’s RYG 35.94 29.16 52.99 65.02 49.77 06.03 57.87 28.76 8.16 YCbCr 35.55 29.32 51.83 68.68 41.30 9.53 58.91 26.07 8.83 HSV 35.13 28.82 56.76 58.55 38.50 12.94 57.89 32.88 10.04 CIE Lab 32.19 25.45 53.05 54.26 41.84 8.47 55.71 24.00 8.84  Faster R-CNN with Resnet-101 RGB 37.24 30.25 61.45 65.23 47.68 6.82 63.11 24.37 16.23 Normalized RGB 34.24 28.32 51.82 64.11 43.46 8.20 57.30 19.63 12.72 Ruta’s RYG 31.96 26.14 50.63 61.55 41.21 13.01 50.04 18.11 7.85 YCbCr 26.17 21.44 42.64 56.82 27.16 5.52 47.45 15.57 4.48 HSV 30.30 22.69 54.00 52.50 34.45 11.02 41.49 27.88 14.44 CIE Lab 24.71 18.86 41.38 46.99 33.59 6.56 46.18 9.48 5.48  R-FCN with Resnet-101 RGB 34.88 27.33 62.19 64.76 36.48 10.07 55.44 30.93 11.57 Normalized RGB 32.16 26.18 52.80 58.86 38.17 5.99 54.03 26.46 9.43 Ruta’s RYG 31.21 24.78 47.20 55.03 30.14 7.13 60.38 23.28 11.29 YCbCr 30.42 23.18 50.18 57.18 29.10 5.45 49.49 30.42 10.87 HSV 30.05 23.25 51.61 56.33 28.48 7.58 50.58 26.10 11.25 CIE Lab 27.33 21.63 44.38 46.86 32.74 4.41 48.95 21.54 9.50The first two network models, Faster R-CNN model with Inception-Resnet-v2 and Faster R-CNN model with Resnet-101, have roughly better performances than R-FCN model with Resnet-101 in terms of mAP. In all three networks, RGB and normalized RGB have high performance than other colors. There is no method having good performance over every type of traffic light.In Table4, from the view point of color space, RGB, normalized RGB, and HSV spaces have higher mAP in Faster-RCNN with Inception-Resnet-v2. RGB and normalized RGB have good performance in Faster R-CNN with Resnet-101. In the case of yellow traffic light, normalized RGB and HSV in Faster R-CNN with Inception-Resnet-v2 and HSV in Faster R-CNN model with Resnet-101 have higher performance than other methods, but most methods have limited overall mAPs depending on sizes of traffic light, small and nonsmall. The medium and large size data are combined into nonsmall set, because our dataset has very limited number of large size traffic light data. The top-ranked three methods such as RGB based Faster R-CNN with Inception-Resnetv2, normalized RGB based Faster R-CNN with Inception-Resnet-v2, and HSV based Faster R-CNN with Inception-Resnet-v2 retain their good performances in the small-size object.Table5 shows the detection performances in terms of AP and mAP when IoU is fixed to 0.5. The top-ranked two methods such as RGB based Faster R-CNN with Inception-Resnet-v2 and normalized RGB based Faster R-CNN with Inception-Resnet-v2 have also better performance than others, even though the ranking order is slightly changed depending on the object size. Ruta’s RYG and HSV in Faster-RCNN model with Resnet-101 and HSV in Faster R-CNN with Inception-Resnet-v2 have significantly high performance for yellow traffic light. Similar to Table 4, CIE Lab color space has relatively poor performance regardless of the network models. As shown in Table 5, the performances of mAP depending on the size are similar to Table 4.


### 5.2. Detection Examples

After the traffic light detection procedure, we use NMS to remove the redundant detections. At final test process, IoU threshold of NMS is fixed to 0.5. The traffic light detection examples of the top-ranked two methods are shown in Figures6 and 7. Six example images are selected to show detection results for six types of traffic lights. The traffic lights with object score being greater than 0.5 are detected and classified. True positives are indicated by the corresponding traffic light symbol. False positives and false negatives are noted by FP and FN, respectively. As shown in Figure 6, the top-ranked method, RGB color-based Faster R-CNN with Inception-Resnet-v2, has twenty-six true positives, three false positives, and four false negatives in six images. Figure 7 shows that normalized RGB color-based Faster R-CNN with Inception-Resnet-v2 has twenty-four true positives, two false positives, and seven false negatives. Both methods cannot detect the yellow traffic lights well.Figure 6
Traffic light detection examples of top-ranked RGB based Faster R-CNN model with Inception-Resnet-v2.Figure 7
Traffic light detection examples of the second-ranked normalized RGB based Faster R-CNN model with Inception-Resnet-v2.


### 5.3. Summary

Based on the performance analysis, the Faster R-CNN model is more suitable to traffic light detection than R-FCN. Inception-Resnet-v2 shows better performance for feature extraction than Resnet-101 in Faster R-FCN framework. From view point of color space, the use of RGB has highest performance in all ensemble networks. The normalized RGB is also a good color space for Inception-Resnet-v2 model.


## 5.1. Simulation Results

Every eighteen methods combined with six different color spaces and three network models are implemented and compared. Tables4 and 5 show the detection performances where every combination methods are listed in the left columns. In the tables, bold and underlined numbers indicate the top-ranked method, bold for the second ranked and underlined for the third ranked.Table 4
Detection performances (overall mAP and overall AP) of combination methods on test set.
Combination Method Overall mAP ( % ) Overall AP ( % ) Ensemble Network Model Color Space total small non small green red yellow red left green left off Faster R-CNN with Inception-Resnet-v2 RGB 20.40 15.85 36.15 33.46 23.81 4.75 34.69 17.59 8.08 Normalized RGB 19.81 15.16 38.10 32.15 22.29 6.06 38.28 11.43 8.65 Ruta’s RYG 18.07 13.54 33.33 28.58 20.05 2.39 35.30 17.98 4.11 YCbCr 16.50 12.71 31.31 29.51 15.25 4.67 31.17 14.33 4.07 HSV 19.70 15.41 37.06 29.23 16.91 6.74 36.00 23.54 5.77 CIE Lab 17.64 13.31 34.30 26.62 18.27 5.41 34.63 15.82 5.09  Faster R-CNN with Resnet-101 RGB 19.24 14.67 37.91 31.21 20.73 3.79 36.92 14.34 8.44 Normalized RGB 17.57 13.54 32.86 29.70 18.20 4.87 33.67 11.99 6.98 Ruta’s RYG 14.72 11.21 28.42 26.55 16.62 4.71 26.27 10.14 4.05 YCbCr 12.36 9.49 25.02 24.03 10.02 2.83 26.36 8.84 2.05 HSV 15.76 11.11 32.24 25.07 14.77 5.64 23.06 17.99 8.01 CIE Lab 10.90 7.63 23.73 19.98 13.79 3.67 20.43 5.28 2.28  R-FCN with Resnet-101 RGB 16.63 11.85 37.27 28.47 13.00 4.92 30.19 18.32 4.85 Normalized RGB 14.50 10.95 29.97 23.57 14.41 2.50 27.87 14.59 4.08 Ruta’s RYG 14.21 10.33 26.66 20.89 9.08 3.01 32.75 13.77 5.72 YCbCr 13.06 9.44 25.05 21.43 10.01 2.50 24.49 14.63 5.28 HSV 14.66 10.59 29.52 25.40 9.99 3.17 28.39 15.23 5.78 CIE Lab 12.24 9.06 23.51 14.58 12.43 1.93 27.87 11.80 4.85Table 5
Detection performances (mAP@0.5 and AP@0.5) of combination methods on test set.
Combination Method mAP@0.5 ( % ) AP@0.5 ( % ) Ensemble Network Model Color Space total small non small green red yellow red left green left off Faster R-CNN with Inception-Resnet-v2 RGB 38.48 31.27 57.79 70.56 52.12 8.49 59.11 27.13 13.44 Normalized RGB 38.24 31.42 59.87 70.43 52.09 10.98 63.94 17.39 14.60 Ruta’s RYG 35.94 29.16 52.99 65.02 49.77 06.03 57.87 28.76 8.16 YCbCr 35.55 29.32 51.83 68.68 41.30 9.53 58.91 26.07 8.83 HSV 35.13 28.82 56.76 58.55 38.50 12.94 57.89 32.88 10.04 CIE Lab 32.19 25.45 53.05 54.26 41.84 8.47 55.71 24.00 8.84  Faster R-CNN with Resnet-101 RGB 37.24 30.25 61.45 65.23 47.68 6.82 63.11 24.37 16.23 Normalized RGB 34.24 28.32 51.82 64.11 43.46 8.20 57.30 19.63 12.72 Ruta’s RYG 31.96 26.14 50.63 61.55 41.21 13.01 50.04 18.11 7.85 YCbCr 26.17 21.44 42.64 56.82 27.16 5.52 47.45 15.57 4.48 HSV 30.30 22.69 54.00 52.50 34.45 11.02 41.49 27.88 14.44 CIE Lab 24.71 18.86 41.38 46.99 33.59 6.56 46.18 9.48 5.48  R-FCN with Resnet-101 RGB 34.88 27.33 62.19 64.76 36.48 10.07 55.44 30.93 11.57 Normalized RGB 32.16 26.18 52.80 58.86 38.17 5.99 54.03 26.46 9.43 Ruta’s RYG 31.21 24.78 47.20 55.03 30.14 7.13 60.38 23.28 11.29 YCbCr 30.42 23.18 50.18 57.18 29.10 5.45 49.49 30.42 10.87 HSV 30.05 23.25 51.61 56.33 28.48 7.58 50.58 26.10 11.25 CIE Lab 27.33 21.63 44.38 46.86 32.74 4.41 48.95 21.54 9.50The first two network models, Faster R-CNN model with Inception-Resnet-v2 and Faster R-CNN model with Resnet-101, have roughly better performances than R-FCN model with Resnet-101 in terms of mAP. In all three networks, RGB and normalized RGB have high performance than other colors. There is no method having good performance over every type of traffic light.In Table4, from the view point of color space, RGB, normalized RGB, and HSV spaces have higher mAP in Faster-RCNN with Inception-Resnet-v2. RGB and normalized RGB have good performance in Faster R-CNN with Resnet-101. In the case of yellow traffic light, normalized RGB and HSV in Faster R-CNN with Inception-Resnet-v2 and HSV in Faster R-CNN model with Resnet-101 have higher performance than other methods, but most methods have limited overall mAPs depending on sizes of traffic light, small and nonsmall. The medium and large size data are combined into nonsmall set, because our dataset has very limited number of large size traffic light data. The top-ranked three methods such as RGB based Faster R-CNN with Inception-Resnetv2, normalized RGB based Faster R-CNN with Inception-Resnet-v2, and HSV based Faster R-CNN with Inception-Resnet-v2 retain their good performances in the small-size object.Table5 shows the detection performances in terms of AP and mAP when IoU is fixed to 0.5. The top-ranked two methods such as RGB based Faster R-CNN with Inception-Resnet-v2 and normalized RGB based Faster R-CNN with Inception-Resnet-v2 have also better performance than others, even though the ranking order is slightly changed depending on the object size. Ruta’s RYG and HSV in Faster-RCNN model with Resnet-101 and HSV in Faster R-CNN with Inception-Resnet-v2 have significantly high performance for yellow traffic light. Similar to Table 4, CIE Lab color space has relatively poor performance regardless of the network models. As shown in Table 5, the performances of mAP depending on the size are similar to Table 4.


## 5.2. Detection Examples

After the traffic light detection procedure, we use NMS to remove the redundant detections. At final test process, IoU threshold of NMS is fixed to 0.5. The traffic light detection examples of the top-ranked two methods are shown in Figures6 and 7. Six example images are selected to show detection results for six types of traffic lights. The traffic lights with object score being greater than 0.5 are detected and classified. True positives are indicated by the corresponding traffic light symbol. False positives and false negatives are noted by FP and FN, respectively. As shown in Figure 6, the top-ranked method, RGB color-based Faster R-CNN with Inception-Resnet-v2, has twenty-six true positives, three false positives, and four false negatives in six images. Figure 7 shows that normalized RGB color-based Faster R-CNN with Inception-Resnet-v2 has twenty-four true positives, two false positives, and seven false negatives. Both methods cannot detect the yellow traffic lights well.Figure 6
Traffic light detection examples of top-ranked RGB based Faster R-CNN model with Inception-Resnet-v2.Figure 7
Traffic light detection examples of the second-ranked normalized RGB based Faster R-CNN model with Inception-Resnet-v2.


## 5.3. Summary

Based on the performance analysis, the Faster R-CNN model is more suitable to traffic light detection than R-FCN. Inception-Resnet-v2 shows better performance for feature extraction than Resnet-101 in Faster R-FCN framework. From view point of color space, the use of RGB has highest performance in all ensemble networks. The normalized RGB is also a good color space for Inception-Resnet-v2 model.


## 6. Conclusions

In this paper, we present a deep-learning based traffic light detection system that consists mainly of color space transform and ensemble network model. Through the simulations, it is shown that Faster R-CNN with Inception-Resnet-v2 model is more suitable to traffic light detection than others. Regardless of the network models, RGB and normalized RGB color spaces have high performance. However, most methods have limited performance to detect yellow traffic lights. It is observed that yellow lights are often misclassified into red lights because the amount of yellow lights is relatively much smaller in training dataset. The performance can be improved, if yellow light training data is large enough as other colors. The results can help developers to choose appropriate color space and network model when deploying deep-learning based traffic light detection.



---

*Source: 2365414-2018-12-06.xml*