# Text Augmentation Using BERT for Image Captioning
## 4 Results and Discussion
### Results  
To compare the results we used BLEU [44], METEOR [45], ROUGE-L [46], CIDEr [47], and SPICE [48], which are widely used for image captioning models comparison. BLEU is metrics which is widely used for machine translation task. It uses n-grams precision to calculate a similarity score between reference and generated sentences. METEOR is also a metric that is based on n-grams, but it uses synonym matching functions along with exact word matching. ROUGE-L is based on a longest common subsequence statistics. The longest common subsequence problem takes into account sentence level structure similarity naturally and identifies longest co-occurring in sequence n-grams automatically. We especially focused on a CIDEr and SPICE metrics, as they are a human consensus metrics. CIDEr measures the similarity of the captions generated by the model and the captions that created by a person using n-grams and performing TF-IDF weighting each of them. SPICE is based on a semantics graph parsing is used to determine how well model has captured the attributes of objects and their relationships. It is important to note that all of the models trained based on their open source code show slightly worse result than that reported in corresponding papers. In Figure 1, you can see a comparison of augmentation using BERT with \(d=2\) for different values of replacement rate \(p\). It can be seen that, with a large value of \(p\) equal to 0.5 model performs worse than with smaller value equal to 0.1. This may be due to the fact that too much information is lost regarding the original caption and therefore the sentences become less grammatically correct and less human-like. On the other hand, model with a value of 0.1 performs better than all others. Such small changes adds some variety to the dataset without a damage to meaning and grammatical structure. The comparison of models trained using various augmentation techniques--synonyms and BERT--with a model trained on an original dataset is shown in Figure 2. The model trained on a dataset with synonymous augmentation is slightly better that the original model. A model that was trained on a dataset augmented with BERT shows significantly better results than both original model and the model trained on a synonymous augmented dataset. Models that are trained on a BERT augmented dataset with \(d=2\) and \(d=3\) and \(p=0.1\) (as the more promising replacement rate) are compared on a Figure 3. Comparison shows that the training on the more augmented dataset don't increases model performance. With a three-times increase in dataset size the model trains worse than with a two-times increase. This shows some boundaries of the proposed method and that the quality do not increase with the significant increase of the dataset more than two-times. Figure 1: Validation scores per epochs during a training phase on a datasets with Bidirectional Encoder Representations from Transformers (BERT) augmentation with \(d=1\) and \(p=0.1,0.3,0.6\). Figure 2: Validation scores per epochs during a training phase on a dataset without augmentation, on a dataset with BERT augmentation with \(d=2\) and on a dataset with synonyms augmentation with \(d=2\). Table 1 summarizes the final test scores for all of the trained [40] models. The model trained on the two-times increased dataset obtained using BERT with \(p=0.1\) augmentation shows the best results in almost all of the metrics significantly exceeding the model trained on an original dataset by 2.7 points for CIDEr and 0.2 points for SPICE. This proves the applicability of the proposed augmentation method for models designed for image captioning task quality improvement. Augmentation can be widely used to increase the quality of existing state-of-the-art approaches without any changes to that models. Table 2 summarizes the results for all three models trained with BERT augmentation with \(d=2\) and \(p=0.1\). All of the models trained with augmentation show better results than the corresponding models trained without augmentation.