## 5 Experiments
### Evaluation Metrics  
To evaluate the performance of proposed blocks and generated sentences, we employ a set of commonly used captioning metrics: BLEU, METEOR, ROUGE-L, CIDEr-D, and SPICE.  
BLEU [48]: This metric is usually used in machine translation. It measures the co-occurrences of \(n\)-grams (\(n=1\), 2, 3, 4) in generated sentences and ground truth sentences. The metrics are calculated by counting the mean of the \(n\)-grams precision scores as follows,  
\[\text{BLEU- }n=\text{BP}\times e^{\left(\sum_{k=1}^{N^{B}}w_{n}^{B}\log P_{n}^{B} \right)} \tag{7}\]  
where \(P_{n}^{B}\) and \(w_{n}^{B}\) are the precision and weight for \(n\)-grams, respectively. \(BP\) is a brevity penalty as Equation (8) is used to penalize sentences that are shorter than ground truth sentences.  
\[\text{BP}=\left\{\begin{array}{ll}1,&\text{if }l_{c}>l_{r}\\ e^{\left(1-l_{r}/l_{c}\right)},&\text{if }l_{c}\leq l_{r}\end{array}\right. \tag{8}\]  
where \(l_{c}\) and \(l_{r}\) are the lengths of the generated and ground truth sentences, respectively.  
METEOR [49]: These metrics evaluate an alignment between the generated and ground truth sentences. First, METEOR calculates the F-Score (\(F^{M}\)) based on the uni-gram precision \(P^{M}\) and recalls \(R^{M}\) as follows:  
\[F_{\text{mean}}=\frac{10P^{M}R^{M}}{R^{M}+9P^{M}} \tag{9}\]  
The METEOR is calculated as follows:  
\[\text{METEOR}=F_{\text{mean}}\left(1-\frac{0.5\times|Chunks|}{|uni-grams|}\right) \tag{10}\]  
where \(|Chunks|\) is the number of contiguous order matches of the generated and ground truth sentence. \(|uni-grams|\) is the number of matched uni-grams.  
ROUGE-L [50]: This metric calculates the recall to measure the longest common sequence (LCS) between the generated and ground truth sentences. The ROUGE-L is based on the precision \(P^{L}\) and recall \(R^{L}\) score of LCS as follows:  
\[R^{L} =\frac{LCS(X^{l},Y^{l})}{m^{l}} \tag{11}\] \[P^{L} =\frac{LCS(X^{l},Y^{l})}{n^{l}}\] (12) ROUGE-L \[=\frac{(1\times\beta^{2})\times R^{L}\times P^{L}}{R^{L}+\beta^{2} \times P^{L}} \tag{13}\]  
where \(m^{l}\) and \(n^{l}\) are the length of ground truth sentence \(X^{l}\) and generated sentences \(Y^{l}\). \(LCS(X^{l},Y^{l})\) is the longest common subsequence of \(X^{l}\) and \(Y^{l}\).  
CIDEr-D [51]: This metric gives different weights based on the frequency of \(n\)-grams. That is, CIDEr-D gives higher weights to rarely seen \(n\)-grams. It calculates a term frequency-inverse document frequency (TF-IDF) weight to \(n\)-grams. \(CIDEr_{n}\) for \(n\)-grams as follows,  
\[CIDEr_{n}(Y^{c},X^{c})=\frac{1}{m^{c}}\sum_{j=1}^{m^{c}}\frac{g_{n}(Y^{c}) \cdot g_{n}(X^{c}_{j})}{\|g_{n}(Y^{c})\|}\Big{\|}g_{n}(X^{c}_{j})\Big{\|} \tag{14}\]  
where \(m^{c}\) is the number of ground truth sentences, \(X^{c}_{j}\) is the \(j_{th}\) ground truth sentence, \(Y^{c}\) is the generated sentence. \(g_{n}\) is the TF-IDF weighting for n-gram. The CIDEr is calculated as follows,  
\[CIDEr(Y^{c},X^{c})=\sum_{n=1}^{N}w_{n}CIDEr_{n}(Y^{c},X^{c}) \tag{15}\]  
where \(w_{n}=1/N\) and \(N=4\).  
SPICE [52]: This metric uses the rule-based method to parse the sentence into a dependency tree and map the tree into a scene graph. Finally, it calculates the F-score of the objects, attributes, and relations. The SPICE focuses on evaluating the objects, attributes, and relations in generated sentences.