# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **MSE** & **MAE** \\ \hline GeoBERT-CenterDistance & 0.0574 & 0.1578 \\ GeoBERT-ShortestPath & **0.0556** & **0.1559** \\ GeoBERT-RandomPath & 0.0674 & 0.1724 \\ \hline Word2 Vec-CenterDistance & 0.3192 & 0.4177 \\ Word2 Vec-ShortestPath & 0.3190 & 0.4182 \\ Word2 Vec-RandomPath & 0.3227 & 0.4188 \\ \hline GloVe-CenterDistance & 0.4889 & 0.5079 \\ GloVe-ShortestPath & 0.4935 & 0.5101 \\ GloVe-RandomPath & 0.4945 & 0.5098 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Evaluation on store site recommendation (POIs only). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Accuracy** & **F1-Score** \\ \hline GeoBERT-CenterDistance & **0.8359** & **0.7922** \\ GeoBERT-ShortestPath & 0.8256 & 0.7777 \\ GeoBERT-RandomPath & 0.8358 & 0.7908 \\ \hline Word2 Vec-CenterDistance & 0.7846 & 0.7042 \\ Word2 Vec-ShortestPath & 0.8000 & 0.7254 \\ Word2 Vec-RandomPath & 0.7821 & 0.6931 \\ \hline GloVe-CenterDistance & 0.6744 & 0.5171 \\ GloVe-ShortestPath & 0.6923 & 0.5455 \\ GloVe-RandomPath & 0.6799 & 0.5039 \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Evaluation on store site recommendation (with additional features). \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **MSE** & **MAE** \\ \hline GeoBERT-CenterDistance & 0.1491 & 0.1825 \\ GeoBERT-ShortestPath & **0.1446** & **0.1809** \\ GeoBERT-RandomPath & 0.1557 & 0.1901 \\ \hline Word2 Vec-CenterDistance & 0.2563 & 0.2916 \\ Word2 Vec-ShortestPath & 0.2567 & 0.2920 \\ Word2 Vec-RandomPath & 0.2569 & 0.2913 \\ \hline GloVe-CenterDistance & 0.3825 & 0.3703 \\ GloVe-ShortestPath & 0.3772 & 0.3651 \\ GloVe-RandomPath & 0.3865 & 0.3700 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Evaluation on passenger flow prediction. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Accuracy** & **F1-Score** \\ \hline MLP & **0.8564** (+2.45\%) & **0.8163** (+3.04\%) \\ Gating & 0.8538 (+2.14\%) & 0.8119 (+2.49\%) \\ Weighted & 0.8436 (+0.92\%) & 0.8103 (+2.28\%) \\ Concat & 0.8435 (+0.91\%) & 0.8000 (+0.98\%) \\ POIs Only & 0.8359 (+0.00\%) & 0.7922 (+0.00\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Evaluation on store site recommendation (with additional features). only exceeds the best Word2 vec model by 1.5% in Accuracy and 3.82% in F1-score, while in regression tasks, GeoBERT is substantially superior to Word2 vec and GloVe, as shown in Figure 11. The difference illustrates that GeoBERT learned more accurate representation of grids and is more capable of fine-grained tasks. Figure 11: Self-attention mechanism in GeoBERT. GeoBERT consists of 12 layers and 12 heads in each layer. Each row represents a layer (from 0 to 11). Each block represents a head (from 0 to 11). The lines between pairs of tokens in a head show the self-attention weights between them. The darker the color is, the greater the weight between the two tokens. Different layers are represented by different colors, while the same color represents heads in the same layer. These are better viewed in color: **(a)** GeoBERT-ShortestPath; **(b)** GeoBERT-CenterDistance; **(c)** GeoBERT-RandomPath. Figure 10: Results of classification tasks. Word2 vec and GloVe still achieve good results. GeoBERT leads Word2 vec by 4.49% in Accuracy for the store site recommendation task and 1.51% for the working \(\backslash\) living area prediction task. Another interesting finding is about the different training corpora. Among the three training corpora, GeoBERT pre-trained on the shortest path gives better results in general. However, this is not absolute; for example, GeoBERT-CenterDistance gives the best store site recommendation result. The same conclusion can also be drawn for the other two models. Actually, previous studies have neglected to compare different POI sequences. The results of five downstream tasks show that the differences between the three corpora are small, which means the precise context relationship between POIs in a certain area (e.g., a level-7 Geohash grid) does not play a critical role.