## 2 Materials and Methods
### Pretraining Phase Using the Address Corpus Based on XLNet
#### 2.3.2 Objective of Permutation Unknown Character Prediction and Two-Stream Self-Attention Structure  
The objective of permutation language modeling is derived from the XLNet model [30]. Without altering the character order of the original text, the target employs rearrangement to sabotage the index order of text descriptions. This training target not only preserves the high-order and long-range dependencies present in the text context, but also improves on the disadvantages of past autoregressive language modeling's targets that could only exploit unidirectional contexts (forward or backward), enabling a pretrained model to utilize deep, bidirectional contextual information more effectively. Addresses, as special natural languages incorporating geospatial information and hierarchy, need to fully utilize the bidirectional contextual information, so we use a permutation language model objective for pretraining the address corpus. Specifically, we assume that given an address record X of length T, there are a total of T! sequences of permutations. If all permutations are traversed and the parameters of the model are shared, then the model must be able to learn the context of all positions. We take a simplified address record, for example, \"Hangzhou Underwater World\" (\"Hang Zhou Hai Di Shi Jie\" in Chinese pinyin), and predict the third character \"Hai\" in a different order, as shown in Figure 2. In Figure 2b, for instance, the address permutation is disordered as 3\(\rightarrow\)2\(\rightarrow\)4\(\rightarrow\)1\(\rightarrow\)6\(\rightarrow\)5, so when predicting \"Hai\", there is no address context character, and the prediction can only be made based on the previous hidden state. For Figure 2f, the \"Hai\" (3) character learns all five context characters except itself. The objective function of XLNet is to maximize the log-likelihood function of the target subsequence conditional on the nontarget subsequence:  
\[\max_{\theta}\!\!\mathrm{E}_{z\sim Z_{T}}[\log p_{\theta}(X_{z>c}|X_{z\leq c })]=\mathrm{E}_{z\sim Z_{T}}\!\left[\sum_{t=c+1}^{|z|}\log p_{\theta}(x_{z_{t} }|X_{Z<t})\right] \tag{1}\]  
where \(Z_{T}\) denotes the set of all permutations of the index of an address record of length T; \(z\in Z_{T}\) is one of the sequences of indexed permutations, where \(z_{t}\) denotes the t-th element of the sequence of indexed permutations, and \(z_{<t}\) denotes the first t-1 elements of z; \(\mathrm{E}_{z\sim Z_{T}}\) denotes the maximum Expectation; and \(p_{\theta}\) denotes the predicted probability. In addition, XLNet used the partial prediction optimization. It slices a permutation \(z\) into two subsequences, \(z_{\leq c}\) and \(z_{>c}\), where \(c\) is the slice point that slices the two subsequences into a nontarget sequence and a target sequence, respectively. While the above objective of permutation unknown character prediction works well for understanding address semantic by removing ambiguity from the target prediction, it creates the problem that the model does not know the position of the character to be predicted in the original address record. Therefore XLNet [33] introduces a two-stream self-attention structure to let the model know where the character to be predicted is located explicitly, which consists of two sets of hidden representations instead of one. The two streams of representations are updated with a shared set of parameters as follows:\[ \begin{split} g_{z_{t}}^{(m)}\gets Attention(Q=g_{z_{t}}^{(m-1)}, KV=h_{z<t}^{(m-1)};\theta),(query\text{\emph{stream}}:\text{\emph{use}}\text{\emph{z}}\text{ \emph{but}}\text{\emph{cannot}}\text{\emph{see}}\text{\emph{x}}_{z_{t}})\end{split}  \tag{2}\]  
\[ \begin{split} h_{z_{t}}^{(m)}\gets Attention(Q=h_{z_{t}}^{(m-1 )}, KV=h_{z\leq t}^{(m-1)};\theta),(content\text{\emph{stream}}:\text{\emph{use}} \text{\emph{both}}\text{\emph{z}}_{t}\text{\emph{and}}\text{\emph{x}}_{z_{t}}) \end{split}