# Image-Caption Model Based on Fusion Feature
## 3 Models and Methods
### Dataset, Evaluation, and Loss Function  
We performed experiments on the MS COCO2014 [17] dataset, which aims to improve the state-of-the-art in object recognition by placing the object-recognition problem in the context of a broader scene of understanding the problem and by collecting images containing common objects in natural environments. The dataset uses professional agencies to artificially describe the images with 5 or 15 reference descriptions per image, and the annotation set is typically saved in JSON format. The dataset has over 330,000 images, 200,000 annotated descriptions containing 91 classes of objects, and a total of 2.5 million tagged instances in 328,000 images, making it the largest semantic segmentation dataset available. BLEU (Bilingual Evaluation Understudy) [18] and CIDEr [19] were used to evaluate this algorithm. The BLEU algorithm rates the difference between the captions generated and those annotated manually, scoring outputs between 0 and 1. The algorithm has become one of image description's most widely used algorithms. For an image, the image description model is used to evaluate a set of 5 descriptors generated for the image and manually marked:  
\[CP_{n}(C,S)=\frac{\sum_{i}\sum_{k}min\big{(}h_{k}(c_{i}),max_{j\in m}h_{k} \big{(}s_{ij}\big{)}\big{)}}{\sum_{i}\sum_{k}h_{k}(c_{i})} \tag{17}\]  
\[b(C,S)=\Big{\{} \begin{matrix}1\\ e^{1-l_{S}/l_{C}}\end{matrix} ,\ \text{$if\ l_{C}>l_{S}$}\\,\ \text{$if\ l_{C}<l_{S}$}\end{matrix} \tag{18}\]  
\[BLEU_{N}(C,S)=b(C,S)exp(\sum_{n=1}^{N}\omega_{n}lgCP_{n}(C,S)) \tag{19}\]  
The number of occurrences of the \(n\)-tuple, \(\omega_{k}\), in the manual reference caption, \(S_{ij}\), and the caption, \(c_{i}\), to be evaluated is denoted as \(h_{k}\big{(}s_{ij}\big{)}\) and \(h_{k}(c_{i})\). Moreover, \(l_{S}\) is the length of the caption to be evaluated, and \(l_{C}\) is the length of the manual reference caption. The higher the BLEU score, the better the performance. The METEOR metric considers the effect of recall, which calculates the matching relationship between synonyms, root words, and affixes, and the evaluation results are more correlated with the results of the manual evaluation, which is calculated as follows:  
\[F_{means}=\frac{PR}{\alpha P+(1-\alpha)R} \tag{20}\]\[P = \frac{m}{c} \tag{21}\] \[R = \frac{m}{r}\] (22) \[Pen = \frac{\#chunks}{m}\] (23) \[METEOR = (1-Pen)\cdot F_{means} \tag{24}\]  
where \(R\) is the recall rate; \(P\) is the accuracy rate, \(m\) is the total number of matched pairs; \(c\) is the length of the candidate caption; \(r\) is the length of the reference caption; and \(Pen\) is the penalty factor, which is to consider the order between words, if in two sentences, the words that match each other are adjacent. They are defined as the same chunk, the total number of \(\#chunks\). ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of evaluation metrics proposed by Lin, mainly including ROUGE-N, ROUGE-L, ROUGE-S, ROUGE-W, and ROUGE-SU. Users can choose the appropriate evaluation metrics according to their needs. ROUGE-L evaluation metrics are generally used to evaluate the model's performance in the image-captioning field. ROUGE-L is mainly optimized for the problem that BLEU evaluation metrics ignore the recall rate. Similar to BLEU, ROUGE-L is also based on the \(n\)-gram evaluation metrics, which calculate the tuple overlap between generated captions and reference captions to measure the quality of captions. BLEU is the same, except that the recall factor is added to the algorithm, and its calculation formula is as follows:  
\[\text{ROUGE}-L=\frac{\left(1+\beta^{2}\right)R_{lcs}P_{lcs}}{R_{lcs}+\beta^{2} P_{lcs}} \tag{25}\]  
\[R_{lcs}=\frac{LCS(X,Y)}{m} \tag{26}\]  
$if\ l_{C}<l_{S}$0  
where \(X\) denotes the candidate title, \(Y\) denotes the reference title, \(LCS(X,Y)\) denotes the length of the longest common subsequence of the candidate caption and the reference caption, \(m\) denotes the length of the reference caption, and \(n\) denotes the length of the candidate caption.