# Large-Language-Model-Enabled Text Semantic Communication Systems
## 3 System Model and Mechanism
### Performance Metrics  
Performance metrics are essential for evaluating proposed methods in communication systems. In end-to-end communication, BER is commonly adopted as the training target by both transmitters and receivers, yet it often overlooks broader communication goals. BER may not accurately reflect the performance of semantic communication systems. Consequently, novel metrics such as Bilingual Evaluation Understudy (BLEU) and Word Error Rate (WER) have been proposed, focusing on word-level similarity between transmitter and receiver outputs. However, these metrics do not fully capture the similarity between entire sentences. To address this gap, metrics utilizing pre-trained models like BERT have emerged for evaluating semantic similarity. This paper selects evaluation metrics that encompass both traditional technical communication systems and emerging semantic communication paradigms, providing a comprehensive assessment of the proposed method. (1) BLEU: BLEU is a metric commonly used to evaluate the quality of machine-generated text against one or more reference texts. Originally developed for machine translation, it has been adapted in semantic communication systems where assessing the quality of generated outputs is crucial. BLEU calculates a score based on the precision of \(n\)-grams (continuous sequences of \(n\) items, typically words) between the generated output and the reference texts. It quantifies how closely the generated text matches the reference texts in terms of these \(n\)-grams, providing a numerical assessment of similarity and fluency. For a transmitted sentence \(s\) of length \(l_{s}\) and its decoded counterpart \(\hat{s}\) with length \(l_{\hat{s}}\), the BLEU score is calculated as follows:  
\[\log\text{BLEU}=\min(1-\frac{l_{\hat{s}}}{l_{s}},0)+\sum_{n=1}^{N}u_{n}\log p_{ n}, \tag{21}\]  
where \(u_{n}\) are weights assigned to \(n\)-grams, and \(p_{n}\) denotes the \(n\)-gram score:  
\[p_{n}=\frac{\sum_{k}\min(C_{k}(\hat{s}),C_{k}(s))}{\sum_{k}\min(C_{k}(\hat{s}))}, \tag{22}\]  
Here, \(C_{k}(\cdot)\) represents the frequency count function for the \(k\)-th elements in \(n\)-grams. BLEU captures contextual relationships to some extent but primarily evaluates superficial similarity of \(n\)-grams without considering semantic equivalence. Consequently, BLEU may assign a lower score even when the generated text is semantically aligned with the reference, due to differences in expressions or the use of synonyms. (2) Sentence similarity: For addressing the issue of polysemous words, [PERSON] et al. introduce a novel evaluation metric for sentence similarity [5]. Sentence similarity assesses the semantic equivalence between two sentences using a pre-trained model. Such models, exemplified by BERT [23], are natural language processing models trained extensively on diverse corpora. The semantic similarity between sentences \(s\) and \(\hat{s}\), both from the sender and receiver perspectives, is computed as follows:  
\[\text{match}(s,\hat{s})=\frac{\mathbf{B_{\phi}(s)\mathbf{B_{\phi}(s)^{T}}}}{ \|\mathbf{B_{\phi}(s)}\|\|\mathbf{B_{\phi}(s)}\|}, \tag{23}\]  
where \(\mathbf{B_{\phi}}\), based on BERT, represents a highly parameterized pre-trained model designed for extracting semantic information from text. (3) BER: BER is a widely adopted metric for assessing the performance of communication systems, quantifying the likelihood of correctly receiving a transmitted symbol. A low BER signifies the system's capability to accurately recover transmitted symbols. However, bit-level errors often inadequately reflect the performance of semantic transmission systems. For instance, occasional bit errors per word may yield a negligible BER, yet render the received text unintelligible due to the absence of correctly received words. Conversely, texts with notably high BER can remain understandable if crucial semantic elements are correctly deciphered.