# Large-Language-Model-Enabled Text Semantic Communication Systems
## 3 System Model and Mechanism
### System Model  
In this subsection, the application of LLMs for modeling the problems outlined in the previous subsection is expounded. LLMs undergo an unsupervised pre-training process aimed at predicting the probability of the next token in a training set, given preceding text. Tokens, which represent encoded forms of human language beyond individual characters, are derived from extensive natural language data and undergo the phase of tokenizer training, akin to a data compression process [37]. Frequently used methods such as Byte Pair Encoding (BPE) and WordPiece [38] serve as forms of data compression, aligning with the function \(\varphi\) discussed earlier. The embedding layer after tokenization further represents semantic information from this encoding. Consider a scenario where both the transmitter and receiver share identical background knowledge from a common knowledge base. If all possible sequences of transmitted information can be enumerated, then \(P(S)\) becomes computable, resolving the problem stated in the foregoing subsection. The objective of LLM training is to predict token probabilities after the tokenization of text, while dataset construction aims to comprehensively cover all human languages. Assuming text information, [PERSON] and [PERSON], needing communication, should possess the same knowledge base (\(X\) distribution equivalence) for effective language interaction. Assuming a comprehensive collection of their language usage into dataset \(\mathbb{D}\), and given adequate computational resources for both parties, algorithms like BPE or WordPiece tokenize \(\mathbb{D}\)'s text. The goal is to train a tokenizer with a vocabulary size \(V\). If \(m\) bits represent a token, then \(2^{m}\geq V\). Tokenization's compression effect--where UTF-8 encodes sequence \(X\) of length \(n\) into sequence \(W\) of length \(t\)--should satisfy the expression  
\[m*t\leq 8*n, \tag{12}\]  
Assuming \(V\) approaches infinity and the number of training iterations tends to infinity, tokenization achieves maximum data compression. For a dataset \(D\), this tokenization results in each token in the vocabulary becoming approximately equiprobable. Consequently, the encoding of transmitted information reaches its shortest form. If such tokenization is applied to train an LLM, the model may fail to converge, as the output \(P(S)\) remains constant. Therefore, by controlling the size of \(V\), one can regulate the degree of compression through tokenization, thereby managing the redundancy of information carried by transmitted bits. This effectively controls the coding rate of source-channel encoding [39]. Next, tokens are fed into LLMs for unsupervised pre-training. The model output can be approximated as  
\[LLM_{output}\approx P(w_{i}\mid w_{i-1},w_{i-2},\cdots,w_{max(1,i-N)}), \tag{13}\]  
where \(N\) denotes the context length of the LLM. As LLMs evolve, \(N\) tends to increase. Given the contextual dependencies inherent in natural language sequences, the tokens \(W=(w_{i},w_{i-1},w_{i-2},\cdots,w_{max(1,i-N)})\) are not independent. Thus, the equation addressing the causal system is  
\[P(W)=\prod_{i=1}^{t}\!\!P(w_{i}|w_{i-1},w_{i-2},\cdots,w_{max(1,i-N)}), \tag{14}\]  
The semantic knowledge base is established through the unsupervised pre-training phase of LLMs by Equation (14), which learns the joint probability distribution \(P(W)\) of token sequences from massive text corpora. The loss function for unsupervised pre-training of LLMs is  
\[\mathcal{L}=-\sum_{i=1}^{t}\log P(w_{i}\mid w_{i-1},w_{i-2},\ldots,w_{max(i-N,1 )}), \tag{15}\]  
It is observed that Equation (14) and the loss function Equation (15) are equivalent. This distribution serves two critical functions: (1) it guides the beam search decoder by ranking candidate sequences based on linguistic plausibility, and (2) it enables semantic error correction by assigning near-zero probabilities to implausible sequences. The knowledge base is static after pre-training and requires no fine-tuning for deployment.