# Traffic Light and Arrow Signal Recognition Based on a Unified Network
## 3 Approach
### Unified Network  
Three networks were employed in our approach, namely YOLOv3 in the first stage, YOLOv3-tiny, and LeNet in the second stage. If the networks are trained separately, the resultant weights can only be used to optimize the results for each stage. If they can share feature maps in one unified network, better results could be achieved from end-to-end training. Moreover, the training speed and inference would also increase.  
In a unified network, the three subnets share feature maps and, thus, the inputs and frameworks of the second and third subnets change. Shared feature maps replace images as the input. Feature extraction is removed from the subnets, and only their prediction function is retained. First, the third-layer feature maps of YOLOv3 that are generated through the FPN are extracted, and these maps are then cropped to retain the areas with traffic lights that are indicated by the YOLOv3 detection results. These areas are then converted to a fixed size through interpolation and adopted as the input feature maps for YOLOv3-tiny. After the inputs are subjected to the convolutional layer and FPN, YOLOv3-tiny then predicts the positions of traffic lights in feature maps of varying scales. At this point, the unified network has predicted the positions of traffic lights and their respective signals. The remaining task for the network is to judge and predict whether a traffic light has arrow lights. If a traffic light has arrow lights, the network must predict the arrow light type. Similar to the previous procedure, the second-layer feature maps produced by YOLOv3-tiny through the FPN are extracted and cropped to retain the areas of arrow lights as indicated by the detection results of YOLOv3-tiny. These areas are then converted to a fixed size and used as the input feature maps for LeNet. After the maps undergo convolution, global average pooling, and softmax layer processing, the network is able to predict arrow light types. Therefore, the unified network can recognize the position of traffic lights, the position of light signals on these traffic lights, and light signal types on the input images. Figure 5 illustrates the unified network architecture.  
Figure 4: Detection network input and output. (**a**) Network input, and (**b**) network output.  
The loss function is calculated by summing the losses from YOLOv3's traffic light prediction, YOLOv3-tiny's light signal detection, and LeNet's arrow light recognition. By summing the losses of the three subnets, we modified the unified network to account for the overall loss of all performed tasks. The loss functions of YOLOv3-tiny and YOLOv3 remain unchanged, but cross entropy loss is applied for LeNet. Moreover, our detection network is based around segmented images. Hence, our training images are segmented for simulating LiDAR processing (see Figure 6 as an illustration). Each image of traffic lights is segmented thrice. Then, the traffic light positions are randomly placed in the segmented image. The dataset includes six primary classes: Green, Yellow, Red, Straight, StraightRight, and Close. We performed data augmentation by rotating the arrow light images to generate more training data. Figure 7 shows an example of data augmentation.  
Figure 5: Flowchart of the unified network.  
Figure 6: Training images. (**a**) Input image, and (**b**) images cropped using LiDAR data.