# A DeBERTa-Based Semantic Conversion Model for Spatiotemporal Questions in Natural Language
## 3 Semantic Encoding Module
### Encoding Model Construction  
The semantic encoding of spatiotemporal queries in Chinese corpora can be viewed as a sequence labeling process, where training on annotated data enables automatic encoding of query semantics. Chinese spatiotemporal queries contain rich semantic information, with sentence structures exhibiting long-distance dependencies and diverse semantic encoding elements [27], resulting in phenomena where the same word may be encoded as different types of information in various contexts. To address these challenges, this paper proposes a DeBERTa-BiGRU-CRF model for spatiotemporal query semantic encoding, which is an improvement based on the dynamic pre-training model DeBERTa, and the model structure is shown in Figure 2. The model integrates a BiGRU network following the DeBERTa model to capture more contextual semantic features, followed by a CRF layer to obtain globally optimal label sequences. This architecture better understands semantic relationships between different words in sentences, captures long-distance dependencies, and resolves ambiguities. (1) Embedding Layer  
The embedding layer converts discrete text data into continuous, low-dimensional vector representations. Given an input text \(\text{X}=\{\text{X}_{1},\text{X}_{2},\text{X}_{3},...,\text{X}_{n}\}\), the embedding layer learns a corresponding embedding vector for each character \(\text{X}_{i}\). The output vector \(\text{Y}=\{\text{Y}_{1},\text{Y}_{2},\text{Y}_{3},...,\text{Y}_{n}\}\), where \(\text{Y}_{i}\) is the embedding vector of the corresponding word \(\text{X}_{i}\), serves as the input to subsequent layers. (2) DeBERTa Layer  
BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture that enhances the performance of various NLP (natural language processing) tasks [28]. It uses a bidirectional training approach, leveraging contextual information from both directions for understanding and processing. Compared to traditional models like recurrent neural networks (RNN) and convolutional neural networks (CNN), BERT can better capture long-range dependencies through the attention mechanism of Transformers [29], thus improving language modeling [30]. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
\begin{tabular}{c} **Semantic** \\ **Types** \\ \end{tabular} & **Word Class** & **Code** \\ \hline Location & Question & 1 & Place type & Noun & t \\ interrogative & word & 2 & Attribute & Noun & o \\ Choice & Question & 3 & Quality & Adjective & q \\ interrogative & word & 4 & Activity & Verb & s \\ Measurement & Question & 5 & Spatiotemporal & Preposition/Noun & r \\ interrogative & word & 6 & Numbers and & Numerals, & d \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table} 
Table 2: Elements of spatiotemporal semantic encoding. The DeBERTa (Decoding-enhanced BERT with disentangled attention) model is an improvement over the BERT model. Introduced by Microsoft in 2020, the DeBERTa model primarily utilizes the Transformer's encoder and demonstrates superior performance and robustness in spatiotemporal questions [31; 32]. The task of the DeBERTa layer is to acquire deep semantic features of natural semantic spatiotemporal questions through word embedding training, converting text into word vectors. The model structure is shown in Figure 3.  
The calculation process for the disentangled attention input representation in the DeBERTa model is shown in Equation (1). \[\rm{A_{i,j}=H_{i}H_{j}^{T}+H_{i}P_{j||i}^{T}+P_{ij}H_{j}^{T}+P_{ij}P_{j||i}^{T}} \tag{1}\]  
Figure 3: Model structure of DeBERTa. Figure 2: Model structure of DeBERTa-BiGRU-CRF. (N stands for multiple Transformer layers).