# GeoBERT: Pre-Training Geospatial Representation Learning on Point-of-Interest
## 3 Materials and Methods
### Build Training Corpus
#### 3.4.1 Shortest Path
\(N_{c}(i)\) is the number of POIs of category \(c\) in \(Grid_{i}\), \(N(i)\) is the total number of POIs, and the \(\Gamma\) is the set of all the POI categories. The diversity of block \(G_{i}^{block}\) is calculated in the same way. \[G_{i}^{div}=\sum_{c\in\Gamma}\frac{N_{c}(i)}{N(i)}\times log\frac{N_{c}(i)}{N (i)}\] (2)
* Competitiveness: Stores of the same category in a region will form a competitive relationship and influence each other. We define the competitiveness feature in Equation (3). \(N_{c}\) represents the total number of stores of the same category of the target stores in the area around the candidate location \(j\), and \(N_{c\_tar}\) is the number of stores of the same category except for the target stores in \(grid_{i}\). \[G_{i}^{compet}=-\frac{N_{c\_tar}(i)}{N_{c}(i)}\] (3)
* Traffic Convenience: It reflects the accessibility to different means of transportation (e.g., bus station, subway station, ferry station, etc. ), as shown in Equation (4). Here, \(tr\) represents a certain type of transportation, and \(N_{tr}(i)\) is the number of all stations of the corresponding transportation type \(tr\) in Gird \(i\). \[G_{i}^{traffic}=\frac{N_{tr\in\Gamma}(i)}{N_{tr}(i)}\] (4) \begin{table}
\begin{tabular}{c c c} \hline \hline
**Positive Samples** & **Negative Samples** & **Total** \\ **(Suitable for Opening a Store)** & **(Unsuitable)** & \\ \hline
701 & 1249 & 1950 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Statistic of store site recommendation dataset. * Residence: It reflects the surrounding residential conditions. Specifically, it is the number of residential buildings of different grades, for example, ordinary residence, high-grade residence, and villa. \[G^{residence}_{i}=\frac{N_{res_{ccT}}(i)}{N_{res}(i)}\] (5) After calculating additional features, we can combine them with GeoBERT. Supposing that the output of the last transformer layer in GeoBERT is \(h\) and additional features are \(x\), we calculate the combined embedding \(Emb\) in the following four methods:
* Concat: We directly concatenate the output of the transformer layer in GeoBERT and all features before the final classifier layer. There is no additional preprocessing. \[Emb=h||x\] (6)
* MLP: We put an MLP (Multilayer Perceptron) layer on additional features first and then concatenate it with the transformer before the final classifier layer. \[Emb=x||MLP(x)\] (7)
* Weighted: We set a learnable weight matrix for each dimension of additional features and then sum it with transformer outputs before the final classifier layer, where \(W\) refers to the weight matrix. \[Emb=h+W\odot x\] (8)
* Gating: We complete a gated summation of transformer outputs and additional features before the final classifier layer, where \(\beta\) is a hyper parameter and \(R\) is an activation function. Detailed information can be found in the paper [43]. \[Emb=h+\alpha k\] \[k=W\odot g+b\] \[g=R(W\odot h+b)\] (9) \[\alpha=min(\frac{\|h\|_{2}}{\|k\|_{2}}*\beta,1)\]

## 4 Experiments and Results

### Baseline and Setup

#### 4.1.1 Baseline

To the best of our knowledge, GeoBERT is the first large pre-trained model in this field. There are two main types of previous studies. The first is to learn POI-type embedding and aggregate them to grid embedding. The second is supervised learning models in which \begin{table}
\begin{tabular}{c l} \hline \hline
**Notations** & **Description** \\ \hline \(G_{i}\) & a level-7 geohash grid \\ \(G^{block}_{i}\) & 9 (3\({}^{\circ}\)3) adjacent grids with \(G_{i}\) in center \\ \(N(i)\) & the number of POI in a grid \(G_{i}\) \\ \(Emb\) & the final embedding after the merge of GeoBERT and addition features \\ \(h\) & the output of the last transformer layer in GeoBERT \\ \(x\) & additional features \\ \(W\) & a weight matrix for additional features \\ \(MLP\) & abbreviation for Multilayer Perceptron \\ \(||\) & concaten operation \\ \(R\) & activation function \\ \(\alpha,\beta,g,k\) & additional parameters only used in Gating method \\ \hline \hline \end{tabular}
\end{table}