# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
It is an interesting finding, and we will discuss it further later. ### Ablation Study

Section 3.5 mentions that GeoBERT uses 15% as the masking ratio during the pre-training stage. The following is an ablation study to evaluate the effect of different masking strategies on the passenger flow prediction task. All the models were pre-trained for 40 epochs to save computing resources and time. We tested the masking ratio for 15%, 30%, 50%, and 70%. From Table 15, it can be seen that GeoBERT is highly robust to different masking strategies, and overall, 15% performs the best. ### What Does GeoBERT Actually Learn?--Part 1: Distilling Common Patterns

In order to determine what GeoBERT learns and the difference between the three kinds of POI sequences, we delve into the self-attention mechanism. GeoBERT, following the original BERT model, uses 12 layers with 12 attention heads in each layer. Therefore, there are 144 (12 \(\times\) 12) different attention mechanisms in total in which is difficult to intuit the meaning of its learned weights in such complexity. Fortunately, with the help of attention visualization tools [45], we can explore the attention patterns of various layers and heads and analyse the underlying principles involved. We select a grid with 22 POIs for demonstration purposes. The POI sequences of the three versions are shown in Table 16. The model views of the three different GeoBERT models are illustrated in Figure 11. Each has 12 layers and 12 attention heads in each layer. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**Center Distance**} & \multicolumn{2}{c}{**Shortest Path**} & \multicolumn{2}{c}{**Random Sequence**} \\ \hline
**Mask Ratio** & **MSE** & **MAE** & **MSE** & **MAE** & **MSE** & **MAE** \\ \hline
15\% & 0.1677 & **0.2065** & 0.1697 & 0.2109 & **0.1713** & **0.2090** \\
30\% & **0.1665** & 0.2066 & **0.1652** & 0.2045 & 0.1718 & 0.2114 \\
50\% & 0.1726 & 0.2108 & 0.1715 & 0.2095 & 0.1716 & 0.2111 \\
70\% & 0.1754 &

#### 5.3.1 Pattern 1: Attention to Next Token

Like BERT, most of the attention at a particular position is directed to the next token in the sequence. Figure 12 is an example of GeoBERT-ShortestPath for Layer 1, Head 10. Figure 12a on the left shows the attention for all tokens in a grid, while Figure 12b on the right shows a specific token \"Teahouse\", which is directed to the next token \"Real Estate\". This pattern is considered to be related to the backward RNN, where state updates are made sequentially from right to left. #### 5.3.2 Pattern 2: Attention to Previous Token

In this pattern, much attention is directed to the previous token in the sentence. We still take the shortest Path POI sequence in Table 16 as an example. The attention mechanism in GeoBERT-ShortestPath of Layer 0, Head 2, is illustrated in Figure 13. We can find that most of the attention for \"Store\" is related to \"Real Estate\". Like Pattern 1, this is loosely related to a sequential RNN, in this case, the forward RNN. Figure 12: The pattern of attention to the next token. The attention mechanism for shortest Path POI sequence example in Table 16 is illustrated. Note that the index starts at 0. Most tokens have a heavy attention weight directed to the subsequent tokens. However, this pattern is not absolute since we can see that some tokens are directed to the other tokens. Colors on the top identify the corresponding attention head(s), while the depth of color reflects the attention score: (**a**) attention weights for all tokens in Layer 1, Head 10; (**b**) attention weights for selected token ’Teahouse’. #### 5.3.3 Pattern 3: Long-Distance Dependencies

In this pattern, many attention heads tend to have a long-distance dependency. In particular, attention is paid to identical or related words, including the source word itself. This pattern is also exhibited in the appendix of the original Transforms [28]. Figure 14 listed below is Layer 6 in GeoBERT-ShortestPath. Most of the attention for the token \"Real Estate\" is directed to itself and \"Express\".