## 2 Methodology
### Semantic Textual Similarity  
To further evaluate the Semantic Textual Similarity (STS) between the ground truth data and the LLMs' outputs, we employ the Sentence Transformer library. Sentence transformers are a set of pre-trained models that generate dense vector representations of sentences, capturing their semantic meaning [28]. These models have been shown to outperform traditional word embedding methods in various natural language processing tasks, including semantic textual similarity [28].  
In our study, we utilize the \"all-mpnet-base-v2\" model, a state-of-the-art transformer model pre-trained on a large corpus of text data. This model has demonstrated excellent performance in encoding sentences into semantically meaningful vectors [38]. By leveraging the \"all-mpnet-base-v2\" model, we aim to capture the semantic proximity between the ground truth and each LLM's output effectively [39].  
The process of calculating STS using Sentence Transformers involves the following steps [39]:  
Encoding the ground truth and LLM output: The \"all-mpnet-base-v2\" model is used to generate dense vector representations for each sentence in the ground truth and the corresponding LLM output. These vectors are obtained by passing the sentences through the pre-trained model, which learns to map semantically similar sentences to nearby points in the vector space. Once the vector representations are obtained, we calculate the STS between the ground truth and LLM output vectors. STS measures the cosine of the angle between two vectors in a multi-dimensional space. It ranges from \(-1\) to \(1\), with higher values indicating greater semantic similarity. The semantic textual similarity between two vectors **a** and **b** is calculated as follows:  
\[\text{semantic textual similarity}=\frac{\textbf{a}\cdot\textbf{b}}{|\textbf{ a}||\textbf{b}|} \tag{7}\]where \(\mathbf{a}\cdot\mathbf{b}\) denotes the dot product of the vectors, and \(|\mathbf{a}|\) and \(|\mathbf{b}|\) represent their Euclidean norms.  
Figure 4 demonstrates how sentences with similar semantic content are positioned closer together in the embedding space, while unrelated sentences appear farther apart.  
As an illustrative example (Figure 4) using sentence embeddings from the \"all-MiniLM-L6-v2\" model, we explored semantic relationships centered around the sentence \"The new movie is awesome for learning a new language\". This analysis revealed clear patterns: notably, the sentence exhibited a robust similarity (similarity score: 0.9299) with \"The new movie is so exciting to learn a new language\", indicating a strong thematic alignment. In contrast, sentences like \"A man walks down the street\" (similarity score: \(-0.0948\)) and \"People are shopping in the mall\" (similarity score: \(-0.0188\)) showed lower or negative similarities, highlighting distinct semantic contrasts. This example showcases the model's ability to discern and quantify semantic nuances effectively across different contexts.  
In our framework, we used semantic similarity across all categories. The semantic similarity scores generated offer a quantitative assessment of how closely the LLM's output aligns with the ground truth. Scores near 1 denote a high level of semantic similarity, indicating that the LLM has successfully captured the meaning and context of the ground truth [29]. Conversely, lower scores suggest a semantic disparity between the two texts, indicating a greater divergence in meaning and context between the ground truth and the LLM's output [29].