# A DeBERTa-Based Semantic Conversion Model for Spatiotemporal Questions in Natural Language
## 3 Semantic Encoding Module
### Encoding Model Construction
Here, \(H_{i}\) and \(H_{j}\) represent the content embeddings of the i-th and j-th words, respectively. \(P_{ijij}\) represents the relative positional embedding between words i and j. \(H_{i}H_{j}^{T}\) represents the computation from content to content, while \(H_{i}P_{ijij}^{T}\) and \(P_{ijij}H_{j}^{T}\) represent the computations from content to position and position to content, respectively. \(P_{ijij}P_{ijij}^{T}\) represents the computation from position to position. The formula for calculating the attention scores is as follows:  
\[Attention(Q,K,V)=Softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V \tag{2}\]  
Among them, Q, K, and V are all word-embedding representations. The variable d represents the dimensionality of the hidden states. The variable k denotes the maximum relative distance, and T indicates the transpose operation. (3) BiGRU Layer  
The task of the BiGRU layer is to extract more semantic features from the text. A Bidirectional Gated Recurrent Unit (BiGRU) is constructed based on two GRUs, each operating in opposite directions [33]. In the structure of a unidirectional GRU network, state information is propagated in a single direction; for example, a left-to-right GRU can only learn information from previous time steps. However, the output of a BiGRU network is determined by both GRUs, allowing it to capture information from both past and future time steps. Since the semantic information of a sentence is related to both its preceding and succeeding context, this paper employs BiGRU to capture more comprehensive contextual information. In BiGRU, the reset gate n selectively forgets parts of the information in the previous hidden state, while the update gate controls the degree to which the previous hidden state is updated and selects the candidate hidden state to be updated [34]. The computational propagation of BiGRU is described by Equations (3)-(6). \[z_{t} =\sigma(U_{z}h_{t-1}+W_{z}x_{t}+b_{z}) \tag{3}\] \[r_{t} =\sigma(U_{r}h_{t-1}+W_{r}x_{t}+b_{r})\] (4) \[\widetilde{h}_{t} =\tanh(U_{h}\left(r_{t}\otimes h_{t-1}\right)+W_{h}x_{t}+b_{h})\] (5) \[h_{t} =z_{t}\otimes\widetilde{h}_{t}+\left(1-z_{t}\right)\otimes h_{t-1} \tag{6}\]  
Here, \(z_{t}\) and \(r_{t}\) represent the update gate and reset gate at time t, respectively; U and W are the weight matrices; and b denotes the bias matrix. The variable \(\sigma\) is the activation function of sigmoid, \(x_{t}\) represents the input information at the current time step, \(\widetilde{h}_{t}\) is the state information of the new memory unit at the current time step, and \(h_{t}\) denotes the updated state information at the current time step, while \(h_{t-1}\) represents the updated state from the previous time step. The symbol \(\otimes\) multiplication of the corresponding elements of the matrix. The final calculation result of the BiGRU is a combination of the forward and backward propagation results, with the backward propagation process mirroring the forward propagation. (4) CRF Layer  
Following the BiGRU layer, a Conditional Random Field (CRF) layer is added to predict the label sequence [35]. The CRF effectively models the dependencies between labels, ensuring that the generated label sequence is coherent and reasonable. The goal of this layer is to recognize geographic entities in natural language problems and to annotate the extracted word vectors using the BIOES format. In the annotation stage, the commonly used annotation schemes include BIO and BIOES patterns. The BIOES schema is applicable to the spatiotemporal interrogative data in this study due to its ability to clearly distinguish the boundaries of spatiotemporal interrogative entities from single-word entities. Compared to the simplicity of the BIO schema, BIOES provides clearer marking at the end of the entity and reduces the ambiguity of boundary identification. B denotes the labeling of the start of an entity, I denotes the interior of an entity, E identifies the character located at the end of the entity, S denotes a single word, and O denotes irrelevant information.