# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Count** & **Mean** & **Std** & **Min** & **25\%** & **50\%** & **75\%** & **Max** \\ \hline
61,521 & 21.97 & 35.15 & 3 & 4 & 9 & 25 & 883 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Statistics of POI number dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Count** & **Mean** & **Std** & **Min** & **25\%** & **50\%** & **75\%** & **Max** \\ \hline
61,521 & 21.97 & 35.15 & 3 & 4 & 9 & 25 & 883 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Statistics of working/living area dataset. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Count** & **Mean** & **Std** & **Min** & **25\%** & **50\%** & **75\%** & **Max** \\ \hline
34,049 & 28,156 & 62,205 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Statistics of passenger flow dataset. Figure 5: The POI number dataset of Shanghai. Figure 6: The working/living area dataset of Shanghai, where yellow refers to living area and red to working area. Figure 7: The passenger flow dataset of Shanghai, where red refers to high-density areas. #### 3.6.5 Store Site Recommendation

Location is considered a critical factor in the success of a store in the modern retail industry. Choosing an optimal location to open a new store has always been a headache for brick-and-mortar chain enterprises. We design a site selection recommendation task and take a large chain joint-stock bank \(B_{t}\) as an example. To avoid label leakage, all \"bank\" tokens are eliminated from the dataset. The statistics of the data set are shown in Table 7. The dataset construction process is shown in Figure 9 below:

1. First, there are too little data for just one bank brand \(B_{t}\) in one city, so we use other similar large chain joint-stock banks for data enhancement. Specifically, we select nine other large joint-stock bank brands similar to \(B_{t}\). 2. Second, the grids of normally operating banks of selected brands are taken as positive samples. 3. Third, we build the negative samples, which consist of two parts. The first part is the banks of \(B_{t}\) that have already closed in the past. The second part is those non-bank POIs within 500 m of the normally operating banks of \(B_{t}\). Figure 8: The house price dataset of Shanghai, where red refers to higher house prices. Figure 9: The process of building a bank recommendation dataset. We set up two experiments. In the first experiment, only POI data is used to fine-tune GeoBERT. In the second one, we use the grid embedding learned by GeoBERT as the basic features and integrate additional grid features to improve the site recommendation task, which is closer to the actual business situation. We calculate about 131 features, which describe the geospatial characteristics of grids in different perspectives. GeoBERT is pre-trained solely on POI data, which can be seen as static information. On the other hand, additional features, such as passenger flow each hour, can provide dynamic information. Limited by space, only some representative features are listed as follows.The summary of primary notations is illustrated in Table 8.

* Passenger flow: Two kinds of features are used to measure the level of passenger flow in a region. First, we calculate the passenger flow of \(Grid_{i}^{flow_{t}}\) in each hour \(t\) of the day. Moreover, we aggregate the eight grids around the current grid as a block to measure the surrounding environment. A block consists of 9 (3*3) adjacent grids. The calculation principle of block features is the same as the grid features but on a larger area (9 grids). See Equation (1). So, we omit the calculation formula of block features in the following features. \[ \begin{split}& G_{i}^{flow_{t}}=\{N(flow_{t})\}(t=1,2,\dots 2 4)\\ & G_{i}^{block\_flow_{t}}=\sum_{i=1}^{9}G_{i}^{flow_{t}}(t=1,2,\dots 2 4)\end{split} \] (1)
* Diversity: We calculate the number of POIs of different categories in each grid to reflect the diversification and heterogeneity of the environment, as shown in Equation (2).