X-ACE: Explainable and Multi-factor Audio Captioning Evaluation* (2024)

Abstract: Automated audio captioning (AAC) aims to generate descriptions based on audio input, attracting exploration of emerging audio language models (ALMs). However, current evaluation metrics only provide a single score to assess the overall quality of captions without characterizing the nuanced difference by systematically going through an evaluation checklist. To this end, we propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE identifies four main factors that constitute the majority of audio features, specifically sound event, source, attribute and relation. To assess a given caption from an ALM, it is firstly transformed into an audio graph, where each node denotes an entity in the caption and corresponds to a factor. On the one hand, graph matching is conducted from part to whole for a holistic assessment. On the other hand, the nodes contained within each factor are aggregated to measure the factor-level performance. The pros and cons of an ALM can be explicitly and clearly demonstrated through X-ACE, pointing out the direction for further improvements. Experiments show that X-ACE1 exhibits better correlation with human perception and can detect mismatches sensitively.

heading: 1 Introduction
text: Recognizing the pivotal role of auditory perception in human cognition, there is a trend among multimodal large language models (MLLMs) (Liu et al., 2023a; Dai et al., 2023) to broaden their scope into the audio-language domain. It has given rise to audio language models (ALMs) (Chu et al., 2023; Tang et al., 2023; Huang et al., 2023b), with a notable emphasis on automated audio captioning
* This work was supported in part by the National Natural Science Foundation of China under Grant U23B2053
† Corresponding author. 1 The open-source dataset and code are available on https:
//github.com/wangqian621/X-ACE
(AAC) for generating comprehensive descriptions of provided audio clips.
With the advancement of ALMs, there is an urgent need for a comprehensive and fine-grained assessment of their generated captions. Since current ALMs generally meet the requirements for depicting salient features of audio, nuanced yet crucial features are still ignored or misdescribed (Takeuchi et al., 2023). As depicted in Figure 1, captions inferred by ALMs have hallucinated in sound activities that do not exist ("rustling"), or confused the order of sounds (between "man speak" and "spray hiss") (Wu et al., 2023; Huang et al., 2023a), demonstrating the significant shortcomings. Metrics used to evaluate the quality of captions include the conventional ones such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) on word overlaps, and BERTScore (Zhang et al., 2020) and Sentence-BERT (Reimers and Gurevych, 2019) on semantic similarity. However,
12273
these metrics cannot be used to measure these shortcomings of existing work, since only a single score is provided to assess the overall quality of captions, thus failing to answer the question: "Where do mistakes commonly occur?
To this end, we explore a comprehensive direction of building an automatic evaluation paradigm for audio caption, and propose the explainable and multi-factor audio captioning evaluation (X-ACE) paradigm. X-ACE consists of four factors tailored for audio caption including sound event, source, attribute and relation, each of which reflects the quality of caption in terms of a specific perspective, rather than providing a single score. To explicitly model the association between these factors, an audio caption to be assessed is transformed into an audio graph, where each node denotes an entity in the audio caption and corresponds to a designed factor. To calculate the score for each factor, the nodes from the audio graph corresponding to this factor are extracted for comparison with the reference through designed graph matching. In this way, we can assess the quality of a caption from multiple perspectives, and enhance the interpretability of the evaluation process.
To facilitate the proposed evaluation paradigm and address the granularity deficiency in the current test dataset, this paper presents a novel dataset for assessment. As for the conventional dataset AudioCaps (Kim et al., 2019) widely used in assessment, each caption includes much less information than the audio itself, which leads to omissions of audio characteristics. To tackle this, a dataset AudioCapsF is constructed based on the AudioCaps, with finegrained annotations from domain experts. This dataset provides specific sound events along with corresponding sound sources and attributes, rather than descriptions in sentence form, promising completeness for further evaluation.
To demonstrate the correlation of the proposed evaluation paradigm with human judgement, XACE is compared with current automatic evaluation metrics including Sentence-BERT (Reimers and Gurevych, 2019), BERTScore (Zhang et al., 2020), and CIDEr (Vedantam et al., 2015). Experimental results of pair-wise tests show that X-ACE exhibits remarkable correlation with human subjective evaluation. Furthermore, extensive experiments also show that X-ACE exhibits better ability to detect the inconspicuous mismatches of a caption, as well as better capability of temporal relation reasoning.
Our main contributions can be summarized
as follows: (1) An explainable and multi-factor evaluation paradigm X-ACE is proposed for audio captioning, demonstrating better correlation with human judgments and mismatch detection ability. (2) A dataset AudioCaps-F with refined annotations is constructed to facilitate further research in this field. (3) A comprehensive and empirical evaluation of existing ALMs based on X-ACE has been conducted, underscoring and analysing the problems with current ALMs.

heading: 2 Related Work
text: Audio Captioning Metrics Presently, there are three types of metrics employed in evaluating AAC. 1) Word Overlapping Metrics: These include ROUGE (Lin, 2004), METEOR(Banerjee and Lavie, 2005), and BLEU (Lin, 2004), which measure the overlap of n-gram words between the generated caption and the reference.2) Semantic Similarity Metrics: Metrics like BERTScore (Zhang et al., 2020; Zhou et al., 2022), SentenceBERT (Reimers and Gurevych, 2019) assess the semantic similarity between captions. 3) Image Caption Metrics: CIDEr (Vedantam et al., 2015) focuses on n-grams of TF-IDF (Jones, 2004). SPICE (Anderson et al., 2016) transforms caption into a scene graph, and then calculating the graph F1 score. These metrics only offer a final score, overlooking crucial yet elementary error.
Aspect-level Evaluations In the domains of vision and language, evaluation methods now aim to discern performance across multiple aspects, rather than providing a single overall score. For image captioning, some hallucination evaluations for MLLMs (Li et al., 2023; Zhou et al., 2023) focus on visible object, AMBER (Wang et al., 2023) comprehensively assesses existence, attribute and relation hallucination. For video captioning, COAHA (Ullah and Mohanta, 2022) detects object and action hallucination. FactVC (Liu and Wan, 2023) classifies factual errors into categories (person, adjective, etc.). X-EVAL (Liu et al., 2023b) employs text evaluation on naturalness, coherence aspects. These methods systematically assess MLLMs but are inherited from visual or language concepts, making it hard to translation to the audio domain. Furthermore, they merely detect nonexistent entities, neglecting the critical omission.
Explainable Evaluations Recent demand for explainability in evaluation metrics has grown significantly. INSTRUCTSCORE (Xu et al., 2023) estab-
lishes a fine-grained explainable evaluation in natural language generation (NLG) task. EAPrompt (Lu et al., 2023), AUTOMQM (Fernandes et al., 2023) attempt to use error analysis on human-like translation evaluation. Metrics from VIEScore (Ku et al., 2023) explain the reason for scoring in image systhesis evaluation. InfoMetIC (Hu et al., 2023) reports fine-grained scores with explainable proof with specific incorrect words or image regions. However, they overly emphasize pinpointing the exact issues rather than categorizing them as a deficiency in a particular aspect. Furthermore, these identified issues fail to full coverage modal information, hindering the establishment of a unified and comprehensive metric.

heading: 3 X-ACE
text: In this section, we commence with defining the audio factors in Section 3.1, which are key components of X-ACE for assessment in Section 3.2.

heading: 3.1 Audio Factors
text: Definition of Audio Factors As depicted in Figure 1, errors in captions arise from different perspectives. In this paper, four factors covering nearly all error issues and audio information are defined including sound event, source, attribute and relation. First of all, sound event denotes the
specific sound activity occurring in an audio like "crying" and "speaking". We have observed that the most common and vital omission occurs in sound event. Models tend to describe prominent sounds while overlooking less significant ones or background noises. Secondly, source denotes the object producing the sound. By assessing this factor, we can detect subtle yet significant differences when the same event is emitted by different objects. Thirdly, attribute denotes auditory characteristics of the sound. This factor highlights nuanced audio features that often overlooked in descriptions. Last but not least, relation denotes temporal order between sounds, which is as crucial as spatial relation in the vision domain. In this way, four factors complement and interdepend on each other, collectively forming a comprehensive description of the audio content.
Definition of Audio Graph Based on audio factors, a caption is transformed into an audio graph, akin to scene graph (Johnson et al., 2015; Schuster et al., 2015) in vision. The graph in the center of Figure 2 comprises two tiers of nodes of parent and child nodes. On the one hand, a parent node is formed by an entity belonging to sound event, denoted as Ei representing the i-th sound event in an audio, where i = 0, .., I−1 and I denotes the number of sound events in this caption.
On the other hand, the corresponding child nodes include source Si,j and attribute Ai,j , representing the j-th source/attribute of the i-th sound event, and the child node Ri,k on the edge connecting the i-th event with the k-th event represents the temporal relation between them.
Factor Annotation To use audio graphs as labels and further enhance the completeness of references, a fine-grained dataset AudioCaps-F is constructed base on AudioCaps, which is widely used in AAC task. We hired domain experts to meticulously annotate entities corresponded to three factors grounded each audio, comprising sound events and their respective sources and attributes. Audio was annotated with an average of three annotators per sample. Subsequently, an automated program and manual review were employed for checking and refinement, the output format is shown in Appendix A. Different from the past practice of using only a few sentences of human descriptions as labels, this detailed annotation is the first attempt on the AAC task, supporting a more comprehensive and robust evaluation. The occurrence distribution of different audio factors are depicted in the Figure 3. The attributes are divided into nine subcategories for finer assessment on ALMs.

heading: 3.2 Evaluation Steps
text: Overview Our X-ACE pipeline, as illustrated in Figure 2, to evaluate a given caption inferred by an ALM, it is firstly decomposed into an audio graph using a factor extractor, which consists of an LLM extractor and a relation reasoner. In the assessment computation phase, the process diverges is divided into two streams. In one stream,
the predicted graph undergoes a greedy matching with the reference graph. This branch provides both factor-level and overall graph assessment. Simultaneously, in the other stream, a cross-modal similarity is calculated and then integrated with the graph score to produce a holistic evaluation.
Factor Extraction. To obtain factor nodes in our audio graph, we concatenate the given caption with a designed prompt and a template, and feed it into an LLM extractor. ChatGPT (GPT3.5-Turbo) serves as the main module for factor extraction. Furthermore, to consider user cost-effectiveness, we also offer Llama3-8b2, which is fine-tuned using output from ChatGPT to achieve the same function. This process results in the structured output of all sound events described in the caption, along with their sources and attributes. The details of the prompt instructions are provided in Appendix A.
Subsequently, a relation reasoner module is employed. It firstly locates sound events and extracts intermediate temporal relation Ri,i+1 between
2https://github.com/meta-llama/llama3
adjacent events, and then extends the inference to deduce the relation Ri,k between arbitrary events. To elaborate, we summarize relations into three types: X
before−→ Y , X and←→ Y , and X after−→ Y as depicted in the Table 1. To deduce the relation Ri,k between the i-th event and the k-th event, the recursive formula follows the chain reasoning rule outlined in the last row of Table 1 as:
Ri,k = G (Rk−1,k, Ri,k−1) , (1)
Ri,k = <before> represents the i-th event occurring before the k-th event, and
G(x, x̂) =    Na(x, x̂) <and> ∈ [x̂, x], x̂ ̸= x, x x̂ = x,
<unknown> others. (2)
Here, x̂ represents a previously inferred relation, and x represents the current relation between adjacent events, Na(x, x̂) outputs the one between x and x̂ which is not equal to <and>.
Greedy Graph Matching Existing binary matching method (Anderson et al., 2016) compares predicted and reference graphs as sets of tuples, scoring each tuple as 1/0, which may overlook potential candidates. To address this, we propose a matching method inspired from the greedy search algorithm, finding the best match candidate node for each node. We formulate matching probability P (x) of each node, which involves computing the maximum similarity between an anchor node x and the candidate set Y = {yi}Ni=0 at each level of graph, as illustrated in the following equation:
P (x) = max y∈Y
S(x, y), 0 ≤ P (x) ≤ 1. (3)
Here, S(x, y) represents the similarity between x and y. When calculating precision, the prediction node is considered as x, with the reference node as y. When calculating recall, the roles are reversed.
Contrary to previous methods (Gontier et al., 2023; Anderson et al., 2016) that treated nodes equally, here sources, attributes, and relations are dependent on their respective sound events, and their significance as child nodes relies on their parent node. Thus, we define the matching probability of a child node based on the matching probability of its parent node, as follows:
P (Ci,j) = P (Ci,j , Ei) = P (Ci,j |Ei)P (Ei), (4)
where Ei represents the i-th event as a parent node, and Ci,j (source or attribute) represents its j-th child node. Notably, Ci,j is contained within Ei, leading to P (Ci,j) = P (Ci,j , Ei).
As for the temporal relation between sound events, the formula is as follows:
P (Ri,k) = P (Ri,k, Ei, Ek)
= P (Ri,k | Ei, Ek)P (Ei)P (Ek) , (5)
where Ri,k denotes the temporal relation between the i-th event and the k-th event, and P (Ei) and P (Ek) are independent from each other.
Subsequently, the average matching probabilities Sfac for each factor are summarized, with E, A, S and R denoting the factor sets of sound events, attributes, sources, and relations, respectively:
Sfac = Avg [P (fac)] , fac ∈ [E,A,S,R]. (6)
If the input variable x in P (x) represents a hypothesis/reference, approximate precision P̂fac /recall R̂fac can replace score Sfac above. This yields the factor-level F-value Ffac, which serves as the score for the E/S/A/R-ACE metrics.
Lastly, the overall graph score SG is calculated leveraging the macro-Precision/Recall as follows:
Pmacro = 1
4
∑
fac
P̂fac, (7)
Rmacro = 1
4
∑
fac
R̂fac, (8)
SG = Fmacro = 2 · Pmacro ·Rmacro Pmacro +Rmacro . (9)
Cross-modal Similarity This module is employed to identify predictions that may correspond to a part of the audio but are not explicitly mentioned in the text. The text global vector Vt ∈ RDim is extracted from predicted caption using BERT (Devlin et al., 2019), and the audio global vector Va ∈ RDim is extracted from reference audio using the HT-SAT encoder (Chen et al., 2022). Subsequently, cross-modal similarity is derived from the cosine similarity as follows,
SC = V ⊤ a Vt. (10)
The holistic score of X-ACE is defined as:
SX−ACE = (SG + SC)/2. (11)
HC HI HM MM Total
Random judgement 45.8 45.7 51.1 52.0 50.0 BLEU_4 (Papineni et al., 2002) 55.2 85.8 77.3 50.7 61.5 SPICE (Anderson et al., 2016) 50.7 83.4 76.5 49.3 59.6 CIDEr (Vedantam et al., 2015) 56.7 96.0 89.1 61.0 70.8 METEOR (Banerjee and Lavie, 2005) 66.0 96.4 89.1 60.2 71.7 ROUGE_L (Lin, 2004) 61.1 91.5 81.1 52.4 64.8 BERTScore (Zhang et al., 2020) 61.1 97.2 91.2 65.4 74.3 Sentence-BERT (Reimers and Gurevych, 2019) 64.0 99.6 92.0 73.7 79.7

heading: 4 Evaluation on X-ACE

heading: 4.1 Correlation with Human Judgement
text: In our experiment on the AudioCaps dataset, a pairwise comparison was employed to measure the correlation between metrics and human judgments. For each pair of candidate captions, humans label which caption in the pair is closer to the given audio. The evaluation is categorized into four splits, as utilized in (Zhou et al., 2022): "HC": two humanwritten captions matching the audio, "HI": two human-written captions with only one matching the audio, "HM": a matching human-written caption, and a machine-generated caption, and "MM":
two machine-generated captions. As depicted in Table 2, X-ACE emerges as the state-of-the-art in correlation with human subjective judgment. XACE without cross-modal similarity module and extra annotation exhibits higher correlation in HC split, with further analysis detailed in Appendix C. We also offer an open-source model Llama3-8B, trained using outputs from ChatGPT. Our X-ACE composed of Llama3-8B as the factor extraction component, still closely matches human subjective perception. Interestingly, it even outperforms system built with ChatGPT under ablation.

heading: 4.2 Mismatch Detection
text: To see whether our factor-level score can sensitively detect types of nuanced mismatch. we automatically introduced different perturbation to clean captions as Table 3 to synthesize subtle mismatch. We then compare the fluctuations in factor-level score with other metrics before and after the perturbations to validate their ability to distinguish error types. To measure the degree of fluctuations on metric scores, we employed the Kruskal-Wallis (McKight and Najab, 2010) significance test. The value in the Table 4 is p̂ = −log(p), with p denotes significance level. A smaller p value, corresponding to a larger p̂, indicates a higher level of difference. When p is less than 0.05 (p̂ is greater than 3), it signifies a prominent difference in scores
before and after perturbing the captions. As showed in the Table 4, our factor-level score mostly get the greatest sensitivity towards the corresponding perturbation on captions, while maintaining other scores essentially the same. In particular, along with the change occur on the sound, the score of source score also changes. The results demonstrate that the factor score of X-ACE exhibits strong capabilities in detecting subtle mismatches and distinguishing different types of errors.

heading: 4.3 Temporal Relation Reasoning
text: To evaluate the temporal relation reasoning capability of metrics shown in Figure 4, individual sound events were concatenated to form three types of captions: X before−→ Y (reference), X after−→ Y (candidate 1), and Y after−→ X (candidate 2).
Based on prior knowledge, candidate 2 merits a higher score for its equivalence to the reference through reasoning. We observed the conflict rate between the metric scores and the prior facts, the results shows that X-ACE and R-ACE greatly outperform others, with their conflict rates lower than 30%. As the temporal relation reasoning abilities of the other metrics even inferior to random selection (50% conflict rate).

heading: 4.4 Ablation Studies
text: To eliminate interference of cross-modal similarity and annotation, X-ACE w/o. cm&anno serves as
our baseline X-ACEbase in ablation study.
The impact of each factors The ablation analysis was conducted to investigate the impact of removing individual factor-level scores. The results presented in Table 5 indicate that removing any single factor leads to a significant decrease in correlation, suggesting that all factors are indispensable for correlation with human perception. The removal of sound event leads to a notably decreased performance, which underscores the dominance of sound events within audio descriptions.
Greedy matching vs. binary matching We conducted a comparative experiment to evaluate the greedy graph matching method designed for XACE against the conventional binary matching (Anderson et al., 2016). As evident from the Table 6, our approach results in an overall performance improvement of 15.2%, demonstrating substantial enhancements across all splits, especially with 24.7% improvement in the MM split.

heading: 5 Evaluation of ALMs Using X-ACE

heading: 5.1 Empirical Evaluation of Current ALMs
text: In this section, we employed X-ACE for the first time to systematically evaluate current ALMs in audio captioning. We specifically analyzed the areas in which different ALMs excel or fall short in. The ALMs models selected for evaluation include
Qwen-Audio (Chu et al., 2023), SALMONN (Tang et al., 2023), Pengi (Deshmukh et al., 2023), and AudioGPT (Huang et al., 2023b). Extensive experiments to assess ALMs with other metrics and investigate the influence of certain variables on metric assessment are shown in Appendix D.
The performance of ALMs evaluated by X-ACE across different factors is depicted in Table 7. Salmonn emerges as the top-performing model, followed by Qwen-Audio, showcasing the best overall performance. The average performance of ALMs on the sound event factor is the highest, while scores for other factors dropped significantly. Notably, even if a model excels in a specific area (e.g., Salmonn-7b performs the best in sound events), the total score may still decrease due to a high incidence of omissions in other factor. Because X-ACE evaluates all factors collectively,
and the shortcomings of each factor affect the overall evaluation. Among models, Qwen-Audio, Pengi, and AudioGPT all exhibit subpar performance in attributes and relations. These models necessitate targeted enhancements grounded in their performance in specific factors, highlighting their inadequacy in characterizing audio attributes.

heading: 5.2 Analysis of Attribute Factor
text: As Table 7 clearly shows that the factor attribute constitutes a vulnerability for the popular ALMs, consequently, a finer evaluation of their performance across nine subcategories of attributes was undertaken. The examples of these subcategories and the fine-grained calculation of the A-ACE score can be referred in Appendix B. Figure 5 presents a comparison of precision and recall values, revealing that the primary issue in attributes
lies in insufficient detail rather than a high incidence of incorrect descriptions or hallucination. Pitch is mostly misdescribed, followed by tone and volume. Most ALMs generate accurate descriptions regarding the vocalization count and their location. Among them, Pengi achieves a high precision rate of nearly 90% in count, but it lacks description or contains errors in pitch. There is a noticeable inconsistency in the distribution of strengths and weaknesses among different models, necessitating targeted improvements.

heading: 6 Conclusion
text: The current assessment of ALMs only provide an overall score, presenting challenges for model refinement. In response, this paper introduces explainable and multi-factor evaluation paradigm X-ACE for AAC, defining sound event, source, attribute, and relation as four factors tailored for the audio description. Furthermore, we provided a dataset AudioCaps-F to enhance evaluation granularity. X-ACE exhibits remarkable alignment with human perception and shows a nuanced capacity to distinguish model errors. Our analysis, derived from outcomes of X-ACE, illuminates substantial variances among mainstream models in audio attribute, and temporal sequence description. While differences are less pronounced in sound events, considerable room for improvement exists across factors.
Limitation
Despite achieving satisfactory performance, our XACE metric also has limitations. Due to the fact that X-ACE requires entity reasoning from LLM during the factor extraction stage, it incurs greater time overhead compared to other metrics that do not rely on large models. We are also dedicated to speeding up inference and training smaller models to replace LLM, thereby achieving more efficient automatic evaluation. Additionally, the crossmodal metrics in X-ACE rely on the robustness of the corresponding cross-modal models, and due to cost constraints, annotations are limited to the AudioCaps dataset. We hope to expand this to other formats of datasets in the future.

heading: A Factor Construction
text: Annotation Construction This section provides a detailed demonstration of the annotation process. We engaged specialized annotators to manually annotate structured data for the given audio, including all sound events occurring in the audio along with their respective sources and attributes, as illustrated in Figure 8. We further categorize attributes into nine major classes. Annotators can use the attribute labels provided in Figure 9 as a reference for annotation. On average, each sound event is annotated with at least four attributes.
Factor extraction via LLM The prompts are input into an LLM extractor, as shown in Table 10, to generate structured data in JSON format. The format of data is then checked and refined via an automated program.

heading: B Model Details
text: Encoder The text encoder and audio encoder are trained on the audio-text retrieval task, using the parameters provided in (Mei et al., 2023). For the matching of each pair of entities in graph, GloVe (Pennington et al., 2014) is used to transform phrases into word vectors for similarity calculation.
Attribute subcategory score We conduct greedy graph matching on anchor attributes, then semantically map them to the closest attribute labels in the attribute tag library, categorizing them into corresponding subcategories, thereby obtaining scores in that subcategory.

heading: C Experiment
text: Ablation Study and Analysis We analyzed cross-modal similarity (cm) and refined annotation (anno) through ablation study as Table 11. Poorer performance in HC and MM splits is evident for cm only, for it lacks of fine-grained interaction and only distinguishes prominent differences (e.g. in HI and HM). Particularly notable in the HC split is that "w/o. anno" outperforms X-ACE. As for the two candidates in HC are reference captions of a given audio in the test data, and the annotation based on references contains all features extracted from these two sentences. Consequently, a self-reference issue arises in the evaluation process, leading to a slight decrease in fairness and correlation with human judgement. This issue does not impede our use of X-ACE to evaluate sentences generated by ALMs.
Temporal Relation Reasoning Setting We picked sound event pairs and generated 178 samples, each formatted as shown in Table 12.

heading: D Evaluate ALMs on Metrics
text: Overall Evaluations As shown in Figure 6, Salmonn and Qwen-Audio exhibit superior performance in AAC, while scores for Pengi and AudioGPT show great fluctuations. X-ACE curve aligns with the trend of SPICE, for they both encompass multiple factors. However, SPICE focuses on visible entities rather than audio characteristics. Notably, X-ACE is the only one to discern superiority of Salmonn-13b over 7b. For the 13b model expresses more diversely, which X-ACE can identify and match, resulting in higher scores.
Association between assessment and caption length It can be observed in Figure 7 that for descriptions with short lengths, such as phrases of only two words, Sentence-BERT, BERTScore, CIDEr, and SPICE surprisingly provide high scores.
X-ACE shows a slight increase followed by a plateau in scores as the caption length increases, with slight fluctuations. On the contrary, most other
metrics tend to decrease, as they do not encourage detailed descriptions to minimize hallucinations or mismatches. BLEU_1 exhibits a rapid rise
followed by a decline, the greatest fluctuation indicates that it is the most influenced by caption length. The scores of all metrics fluctuate greatly in the range of caption length from 15 to 25, where the metric assessments show little correlation with caption length.
Case Studies Case studies are conducted as shown in Fig. 8, where (a) displays sample comprising the ref (correctly described reference for the given audio), along with candidate A and B for evaluation, with blue highlighting the true mismatches. Analyzing the first sample, candidate A interchanged the attributes "distant" and "nearby" associated with two sounds as a bad caption. Candidate B only altered the sentence structure and employed synonym substitutions without changing the content as a good caption. As shown in (b), our A-ACE indicates a significant decrease in the attribute score for candidate A, while B shows a slight decline, demonstrating the sensitivity of our method in detecting attribute errors. Furthermore, in (c), comparing all metrics, the bold sections corresponding to X-ACE and A-ACE reveal results consistent with subjective evaluation, indicating that B outperforms A. It is worth noting that since the current case study utilizes only one sample set without establishing a sample library, the CIDEr score remains at 0 throughout.

heading: Future Work
text: We hope to expand this to other formats of datasets in the future