## 4 Model-Specific Cosine Similarity  
Traditional evaluation metrics like BERTScore and BLEU are based on single-model or rule-based calculations, where the scores are fixed once the dataset is determined and cannot effectively evaluate model performance. These metrics do not capture how different models understand perturbed input, limiting their ability to assess model-specific effects. Additionally, since both BERTScore and BLEU rely on token matching, they are unable to detect perturbations caused by word-order changes. To address this limitation, we propose Model-Specific Cosine Similarity (MSCS), a novel evaluation framework that leverages multiple LLMs to assess model performance under input perturbations comprehensively. MSCS builds upon the well-established cosine similarity metric [26; 27], which measures semantic similarity between text representations. Unlike BERTScore and BLEU, which provide static evaluations, MSCS dynamically integrates the capabilities of different models, enabling fine-grained analysis of how perturbations affect input understanding across diverse LLM architectures. This approach captures model-specific variations in processing perturbed inputs, offering insights that traditional metrics cannot provide. The key steps of MSCS are as follows. (1) Text Representation: First, we use different LLMs and their corresponding tokenizers to convert the two texts to be compared into vector representations. Given an input text, the model tokenizes it and converts it into tokens. The formula for this step is the following. Figure 3: Flowchart of self-correction reflection process. \[t_{i}=\text{Tokenizer}_{k}(T_{i}),\quad t_{j}=\text{Tokenizer}_{k}(T_{j}) \tag{8}\]  
where \(T_{i}\) and \(T_{j}\) are the two input texts, and \(t_{i}\) and \(t_{j}\) are the corresponding tokens. (2) Hidden States Extraction: Once the text has been tokenized, the tokens are passed through the feedforward layer of the model. In this layer, the model outputs a series of hidden states at each sublayer. To capture the semantic information, we focus on the hidden states of the final layer, as it can comprehensively integrate semantic information after being processed by previous sublayers. As shown in Figure 4, it illustrates the overall flow of the MSCS calculation process. The formula for this step is the following. Figure 4: Flowchart of the MSCS, BERTScore, and BLEU calculation procedures. \[h_{i,j},h_{j}=\{\text{Model}_{k}(t_{i}),\text{Model}_{k}(t_{j})\}, \tag{9}\]  
where \(h_{i,i},h_{j}\) are the hidden states of the final embedding layer of Model\({}_{k}\). (3) Pooling: To obtain a fixed-length vector for each text, we apply mean pooling to the hidden states of the final layer. This involves averaging the hidden states across all tokens in the sequence \(h_{i},h_{j}\), resulting in vectors \(v_{i}v_{j}\) that represent the entire input text. The formula for this step is the following. \[v_{i}=\frac{1}{n}\sum_{p=1}^{n}h_{ip},\quad v_{j}=\frac{1}{n}\sum_{p=1}^{n}h_{ip} \tag{10}\]  
where \(n\) is the number of tokens of each text, and \(v_{i}\) and \(v_{j}\) are the vectors of the texts. (4) MSCS Calculation: We compute the MSCS between the two representation vectors. MSCS is defined as the cosine of the angle between two vectors and is computed as follows:  
\[\text{MSCS}=\frac{v_{i}\cdot v_{j}}{\|v_{i}\|\|v_{j}\|} \tag{11}\]  
where \(v_{i}\) and \(v_{j}\) are the vector representations of the two texts, and \(\|v_{i}\|\) and \(\|v_{j}\|\) are their Euclidean norms. For \(\forall Model_{k}\), MSCS ranges from \(-1\) to \(1\). When we calculate MSCS for the perturbed text and the text before perturbation (original text), we can judge the difference between the LLM's understanding of the text after perturbation and the understanding of the original text based on the results of MSCS. The closer the value is to \(1\), the more the model understands the two sentences, proving that the model basically ignores the impact of the perturbation.