# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
* House Price Prediction: As depicted in Table 12, GeoBERT significantly outperformed the other two models, and the performance difference between these three models was substantial. GeoBERT pre-trained on the shortest path obtained the best result. * Store Site Recommendation: The results of store site recommendation with POIs only are shown in Table 13. GeoBERT achieved better performance, while GeoBERT pre-trained on center distance corpus obtained the best result, with 0.8359 in Accuracyand 0.7922 in F1-score. The differences between the three GeoBERT models are not considerable. The results of store site recommendation with additional features are illustrated in Table 14. We can see that with additional features, both Accuracy and F1-score were improved in all cases. Among different combination methods, the MLP method obtained the best performance, increasing Accuracy by 2.45% and F1-score by 3.04%. The result of these two experiments illustrate the following points:

1. GeoBERT obtains good grid embedding from POI data and can be directly used for store site recommendation. 2. GeoBERT is scalable and can be jointly used with additional features. The effect will be improved if more dimensional data is provided. GeoBERT was pre-trained solely on POI data, which can be seen as static urban information. On the other hand, additional features, such as passenger flow each hour, can provide dynamic ubran information. However, in practice, additional features, such as user profiles and passenger flow data, are hard to access and often subject to privacy restrictions, while POIs are not. Therefore, GeoBERT alone can be used to achieve pretty good results which proves that GeoBERT has practical applied value. To sum up, POI data is the most readily available urban data that can contain much geospatial information. However, other features can also add information from different dimensions, especially user consumption behavior and travel behavior. GeoBERT is effective and having additional features would be better for more specific tasks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Accuracy** & **F1-Score** \\ \hline GeoBERT-CenterDistance & 0.7736 & 0.7712 \\ GeoBERT-ShortestPath & 0.7729 & 0.7677 \\ GeoBERT-RandomPath & **0.7739** & **0.7719** \\ \hline Word2 Vec-CenterDistance & 0.7642 & 0.7359 \\ Word2 Vec-ShortestPath & 0.7626 & 0.7337 \\ Word2 Vec-RandomPath & 0.7638 & 0.7344 \\ \hline GloVe-CenterDistance & 0.7398 & 0.7093 \\ GloVe-ShortestPath & 0.7454 & 0.7144 \\ GloVe-RandomPath & 0.7377 & 0.7101 \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Evaluation on working\(\backslash\)living area prediction. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **MSE** & **MAE** \\ \hline GeoBERT-CenterDistance & 0.1932 & 0.1492 \\ GeoBERT-ShortestPath & **0.1790** & **0.1343** \\ GeoBERT-RandomPath & 0.1994 & 0.1383 \\ \hline Word2 Vec-CenterDistance & 0.2503 & 0.2354 \\ Word2 Vec-ShortestPath & 0.2474 & 0.2330 \\ Word2 Vec-RandomPath & 0.2659 & 0.2385 \\ \hline GloVe-CenterDistance & 0.3824 & 0.3150 \\ GloVe-ShortestPath & 0.3958 & 0.3147 \\ GloVe-RandomPath & 0.4082 & 0.3225 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Evaluation on POI number prediction. ## 5 Discussion

### Result on Downstream Tasks

We can find out that GeoBERT outperforms Word2 vec and GloVe on all five downstream tasks, which proves that pre-training on large corpora helps extract the geospatial information of POIs. However, the performance of these three models on different tasks is different. On classification tasks, Word2 vec and GloVe are still competitive, which is illustrated in Figure 10. For example, in the working\ living area prediction task, GeoBERT