# Context Is King: Large Language Models' Interpretability in Divergent Knowledge Scenarios
## 5 Results and Discussion
### Correlations Between Metrics  
In this section, we analyze the correlations between various evaluation metrics to validate the effectiveness of our proposed methodology. First, we focus on how LLM-based evaluation metrics (LAV, Acc, and Cm) correlate with objective metrics, such as ASS, which is based on cosine similarity. By understanding these correlations, we aim to identify any alignment or divergence between subjective model-based assessments and objective vector-space metrics, thereby evaluating the overall usefulness of the proposed measures. Figure 4 shows the correlation between a weighted combination of the LLM-based metrics (0.5 for LAV, 0.25 for Acc, and 0.25 for Cm) and ASS, employing GPT-4 Turbo (Figure 4a) and Claude 3 Haiku (Figure 4b) as evaluators. The weighting was chosen to reflect the relative importance of each metric in capturing core aspects of model alignment with the context: LAV, as the primary indicator of correct answers in our analysis, is assigned the highest weight (0.5), while Acc and Cm each contribute 0.25 to balance precision with thoroughness. The figure presents the results for each of the 50 contexts in the Divergent Knowledge Dataset (DKD) for the four models analyzed, highlighting the correlation area between ASS and the weighted LLM-based metrics. These results illustrate the alignment between ASS and the combined LLM-based metrics (LAV, accuracy, and completeness), with GPT-4o demonstrating the highest overall alignment. The strong correlation observed between ASS and the weighted LLM-based metrics supports the validity of our proposed evaluation framework, suggesting that models performing well in generating semantically similar answers (high ASS) also score highly in the combined subjective metrics. In particular, the figure shows that GPT-4o and GPT-3.5 Turbo exhibit the highest correlation between ASS and the weighted metrics with both evaluators. This indicates that models that excel in adhering to divergent contexts are also effective at producing semantically aligned responses. GPT-4o's high scores across both measures underscore its advanced capability in handling context-specific information, while GPT-3.5 also achieves strong alignment, albeit with slightly less consistency than GPT-4o. Figure 4: Correlation between weighted LLM-based metrics (LAV, accuracy, and completeness) and Answer Semantic Similarity (ASS) across different models. Each point represents an individual evaluation of a context on the Divergent Knowledge Dataset, with colors indicating different models (Mikrtal-8x7B, GPT-3.5, GPT-4o, and Llama 3). A weighted combination (0.5 for LAV, 0.25 for accuracy, and 0.25 for completeness) was used to highlight alignment between subjective model-based metrics and the objective similarity measure (ASS). Analyzing the performance for Llama 3 and Mixtral 8x7B, we observe that both models exhibit lower ASS, reflecting a tendency for more concise or less contextually aligned responses. However, GPT-4 Turbo as the evaluuator tends to assign lower scores in the LLM-based metrics, while Claude 3 Haiku generally provides higher, although more variable, scores. This contrast suggests that [PERSON] may be overestimating subjective metrics (or GPT-4 Turbo undervaluing them), highlighting slight differences between the evaluation models. Furthermore, if we analyze the correlation between LAV precision and LLM-based accuracy, also employing GPT-4 Turbo as the evaluuator, we obtain the diagram shown in Figure 5. This diagram shows the correlation between both measures for the 50 contexts for the model that committed the most errors, Mixtral 8x7B. The figure clearly shows how the analysis of response accuracy correlates highly with response validity, confirming the sense of the proposed metrics. Some of these metrics rely on the evaluative LLM's ability to respond or score accurately, making them somewhat subjective. Therefore, we decided to manually review the results, especially those of LAV.