# Storytelling with Image Data: A Systematic Review and Comparative Analysis of Methods and Tools
## 8 Task 6: Quality Assessment and Dataset
### Automatic Assessment  
Determining the quality of created text remains a significant difficulty [117]. In machine translation, some metrics such as BLEU [125] and METEOR [126] are commonly utilized. In addition, for image captioning evaluation, CIDEr [128] and SPICE [127] are a kind of text summarization evaluation tool. However, each of these metrics has drawbacks for analyzing natural language performance, as there is a significant gap between automated metrics and human evaluation [117]. Recent research has focused on more natural evaluations of text creation tasks, such as structuredness, variety, and readability [112; 129; 130]. As follows, we discuss each metric in detail, and [131; 132] Table 5 indicates a summary of the evaluation metrics considered in this section. **Bilingual Evaluation Understudy (BLEU). ** BLEU [125] is one of the earliest and most widely used metrics used to measure the similarity between two texts and is established for the automatic evaluation of machine translation platforms [132]. It is a precision-based measure that calculates the n-gram overlap between the assumption and the reference [133]. BLEU represents the proportion of overlapping n-grams to the total number of n-grams in the assumption [133]. To be accurate, the numerator includes the total number of overlapping n-grams across all hypotheses, whereas the denominator comprises the total number of n-grams throughout all assumptions [133]. BLEU stipulates that a high-scoring explanation should meet the ground truth in length, i.e., a precise match of words and their arrangement [131]. An accurate match will receive a BLEU assessment score of 1 [131]. **Metric for Evaluation of Translation with Explicit Ordering (METEOR). ** METEOR [126] is another measurement for machine translation. It was designed to overcome the inadequacies of BLEU, and instead of requiring a precise lexical match, METEOR implements semantic matches [131]. It may be described as the arithmetic mean of accuracy and recall of unigram matches among texts [132]. The METEOR score depends on how effectively the produced and reference phrases are constructed [131]. Each phrase is represented as a collection of unigrams, and alignment is accomplished by mapping the unigrams of nominees and reference phrases [131]. A unigram in a nominee phrase (or reference phrase) would map to either a unigram in the reference phrase (or candidate phrase) or to zero throughout the mapping process [131]. Furthermore, it uses synonyms and the matching of phrases. METEOR solves various shortcomings of BLEU, such as the assessment of recall and the absence of direct word matching [132]. In cases when numerous alignment possibilities are available between two phrases, the alignment arrangement with the fewest crossings is selected [131]. After completing the alignment procedure, the METEOR score is determined [131]. **Recall-Oriented Understudy for Gisting Evaluation (ROUGE). ** In 2004, ROUGE [180] was first developed for evaluating summarization programs, and this assessment is performed by comparing overlapping n-grams, word sequences, and word pairs [132]. Using n-grams, the algorithm determines the recall value of the created sentences that match the reference phrases [131]. Comparable to BLEU, ROUGE is determined by altering the number of n-grams. ROUGE is based on recall rates, whereas BLEU is based on accuracy [131]. Since the ROUGE metric depends heavily on recollection, it favors extended sentences [132]. In addition to n-gram variations of ROUGEn, there is also ROUGEL (Longest Common Subsequence), ROUGES (Skip-Bigram Co-Occurrences Statistics), ROUGEW (Weighted Longest Common Subsequence), and ROUGESU variants (extension of ROUGES) [131]. **Semantic Propositional Image Captioning Evaluation (SPICE). ** SPICE [127] is a recently developed measure for measuring the similarity of picture captions [132]. It is determined by comparing the scenery graph tuples of the prospective phrase with those of all reference phrases [132].