# Enhanced Image Captioning with Color Recognition Using Deep Learning Methods
## 4 Experiments and Results
### Model Training and Datasets  
In this stage, two models are trained for object recognition according to the context of use. One model uses the MSCOCO dataset (2014 version), and the other model uses a self-made traffic signal dataset. MSCOCO is an open-source dataset with multiple object features, popularly used to train algorithms for different purposes. In the MSCOCO dataset, there are 1.5 million objects, belonging to 80 object recognition categories and 91 image captioning categories. Comparisons between three open-source datasets are shown in Table 1.  
Figure 5: Proposed system architecture.  
Figure 6: Process flow diagram of the proposed system.  
Although the number of images in the MSCOCO dataset is not the most, as shown in Table 1, the number of bounding boxes is far more than in other commonly used datasets, and the numbers of various object sizes--such as small, medium, and large--are evenly distributed in comparison to other datasets, as shown in Table 2. The image content provided in MSCOCO is closer to daily life scenes. For the second object recognition model, a self-made traffic light dataset is used. Compared to the open-source dataset, the self-made dataset needs more effort to collect and organize the images. However, the self-training model can expand the types of objects that the model can recognize. In practice, a total of 100 images are self-collected for traffic light recognition. The collected images are first labeled, and the training is carried out with 70% training and 30% verification. In general, if the image data are insufficient, the model recognition rate could be reduced. Data augmentation techniques can be applied to produce a greater variety of samples, such as flip, rotation, and scaling. In this work, original images are considered references, and these images are zoomed in and out by 0.5 times, and also rotated 45 degrees to the left or right. As a result of previous augmentation processes, the self-made dataset now has 500 images; the training process is shown in Figure 7. From Figure 8, it can be clearly seen that the recognition result is significantly improved after using the data augmentation technique. In Figure 8a, the trained model can detect just one traffic light using only original sample data. In Figure 8b, two traffic lights can be detected, as a result of the recognition model being trained with the augmented data.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Category** & **Image Quantity** & **BBox Quantity** \\ \hline PASCAL VOC (07++12) & 20 & 21,503 & 62,199 \\ MSCOCO & 80 & 123,287 & 886,266 \\ ImageNet Det (2017 train) & 200 & 349,319 & 478,806 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparisons of open-source datasets.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Total** & **Small** & **Middle** & **Large** \\ \hline PASCAL VOC (07++12) & 62,199 & 6983 & 19,677 & 35,539 \\ MSCOCO & 886,266 & 278,651 & 311,999 & 295,616 \\ (2014 trainval) & 478,806 & 22,677 & 86,439 & 369,690 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Distribution of object size (BBox size) in open-source datasets.  
Figure 7: Flowchart of model training.