# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
This pattern is not so distinct, with attention dispersed over many different words in other heads, which can be seen from the different colors in the right sequence in Figure 14b. Figure 14: Pattern of long-distance dependencies. “Real Estate” is directed to itself and “Express” in (**a**). However, the attention is also dispersed over many different words which can be seen in (**b**). The color in the right sequence represents its corresponding head with yellow for Head 1 and green for Head 2: (**a**) attention weights for selected token “Real Estate” in Layer 6, Head 1; (**b**) attention weights for selected token “Real Estate” in Layer 6, heads 1 (orange) and 2 (green). Figure 13: The pattern of attention to the previous token. In the example of Layer 0, Head 2, most tokens show apparent attention weight directed to the previous tokens. Of course, there are some exceptions, such as the token “Teahouse”, which still has close attention to the next token “Real Estate”: (**a**) attention weights for all tokens in Layer 0, Head 2; (**b**) attention weights for selected token “Store’. ### What Does GeoBERT Actually Learn?--Part2: Deeper Insights

In Section 5.3, we explore the three common patterns learned by GeoBERT from the perspective of self-attentions. The above findings explain part of the rationale behind GeoBERT, but they still require further research. In Part 2, we will drill deeper into GeoBERT's attention mechanism and reveal the answer to two key questions. #### 5.4.1 Question 1: What Is the Difference between the Three POI Sequence Construction Methods? In Section 3.4, we proposed three different POI sequence construction methods and pre-trained three GeoBERT models on them. Previous studies have proven these construction methods (Shortest Path and Center Distance Path) effective in extracting POI location information. To date, no study has compared these methods and analyzed the spatial information in depth in the POI sequences. With the help of Transformer structure's powerful information extraction capabilities, we can further explore the difference between these construction methods and what GeoBERT has learned. In brief, there is a contextual relationship but no exact back-and-forth relationship among adjacent POIs in a grid, and GeoBERT can learn this kind of location information in shallow attention layers. We illustrate the attention mechanism of Layer 1 and Layer 2 for GeoBERT-ShortestPath, GeoBERT-CenterDistance, and GeoBERT-RandomPath in Figures 15-17, respectively. From Layer 1 and Layer 2 of GeoBERT-ShortestPath (Figure 15) and GeoBERT-CenterDistance (Figure 16), we can intuitively find that there are higher attention weights between a token and the two adjacent tokens, presented as pairs of crossed straight lines in the image, while in the GeoBERT-RandomPath (Figure 17), the same phenomenon has not been observed. The above findings indicate two things. The first is that attention Layers 1 and 2 of the GeoBERT can learn the adjacency relationship between POIs. The second is that in terms of how the POIs are organized, both the shortest path and the center distance path contain location information, but random sequences cannot, which is in line with expectations. On the other hand, the pairs of cross lines between adjacent tokens show that POIs have close context relationships but do not have strict back-and-forth ones such as natural language, which has more specific grammar and syntax rules. Figure 11.0 Attention Layer 1 and Layer 2 in GeoBERT-ShortestPath. We can see pairs of cross lines between adjacent tokens, which means that GeoBERT has learned the position information between adjacent tokens. (**a**) Attention Layer 1; (**b**) Attention Layer 2. #### 5.4.2 Question 2: Why Do the Three Models Have Similar Effects on All Five Downstream Tasks? Section 5.1 compared the GeoBERT models pre-trained on three different corpora.