# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
In (**b**), there are two “Bus Stations” in a grid, and both attract the most attention. Moreover, the weights for the first “Bus Station” are higher. This difference validates that the sequence order plays a role to some extent: (**a**) Layer 10, Head 7; (**b**) Layer 10, Head 8. ## 6 Conclusions and Future Work

### Conclusions

This paper proposes the first pre-trained geospatial representation learning model called GeoBERT based on POI data and a BERT Model. We collected 17 million POIs and constructed different POI sequences for each level-7 Geohash grid. The sequence is built on three methods: the shortest path, the center distance path, and the random path. After pre-training, GeoBERT was fine-tuned on five downstream tasks. Compared with other NLP models (Word2 vec, GloVe) used in this field, GeoBERT obtained the highest results on all five tasks. Moreover, GeoBERT is highly scalable. Combining the grid embedding learned from GeoBERT with additional features can further improve the performance. Then, we went deep into the attention mechanism in the GeoBERT and analyzed what GeoBERT actually learns. Through detailed visualization analysis, we reached three main conclusions. 1. The shortest path and center distance contain the position information among POIs in a grid, while the random path method does not. 2. GeoBERT learns the position information in the shallow attention layers. In deep attention layers, GeoBERT captures co-occurrence among POIs and identifies the most important POIs, called the anchor POIs in a grid. 3. The sequential relationship between POIs does not play an important role. What matters is the co-occurrence among POIs and the specific anchor POIs learned in deep attention layers. ### Future Work

The above conclusions guide our subsequent work. Since different POI sequences have limited effect on the experimental results, we intend to take POIs in a grid as 2D raster data rather than construct a POI sequence for future work. A grid with POIs can be seen as an image to some extent. The difficulty, however, is how to process the 2D POI data as an image and encode it into vectors. From another point of view, we can further optimize our work in the future by incorporating POI data with satellite images, making it a multi-modal pre-training model. A simple and straight way is to take POI as text and satellite as image. Figure 11.6 Attention mechanism for addition Case 2 in GeoBERT-ShortestPath. The phenomenon is evident, and the above two heads each identify an anchor POI, namely “Supermarket” and “Industry Park”: (**a**) Layer 9, Head 9; (**b**) Layer 10, Head 3. By doing this, we can learn more information about a grid and make the model capable of more tasks. Conceptualization, Y.G. and H.W. ; methodology, Y.G. and Y.X. ; formal analysis, Y.G. and S.W. ; supervision, H.W. and Y.X. ; writing--original draft preparation, Y.G. and S.W. ; writing--review and editing, H.W. and Y.X. ; All authors have read and agreed to the published version of the manuscript. This research was supported by the National Nature Science Foundation of China (Grant No. 62176185), and the Top Discipline Plan of Shanghai Universities-Class I (Grant No. 2022-3-YB-02). POI data can be accessed by AutoNavi API at [[https://developer.amap.com/api/webservice/guide/api/search/](https://developer.amap.com/api/webservice/guide/api/search/)]([https://developer.amap.com/api/webservice/guide/api/search/](https://developer.amap.com/api/webservice/guide/api/search/)) (accessed on 1 June 2022). POI number data and store site recommendation data can be acquired from POI dataset itself. Data used in fine-tuning is provided by ayz.ai at [[http://www.wayz.ai/](http://www.wayz.ai/)]([http://www.wayz.ai/](http://www.wayz.ai/)) (accessed on 1 June 2022). Restrictions apply to the availability of the data (e.g., passenger flow data and working/living data), which is used under license for this study. The authors would like to thank the Wayz.ai for providing data and experiment facilities used in this study.