# BERT4 Loc: BERT for Location--POI Recommender System
## 3 Methodology
### POI Recommendation Model  
We develop a model to understand the interaction between users and POI. We formalize the problem and introduce our training objectives in this section. The notations used in this table are given in Table 1.  
Let \(U=\left\{u_{1},\ u_{2},\ \ldots,\ u_{|U|}\ \right\}\) be a set of users and \(V=\left\{v_{1},\ v_{2},\ \ldots,\ v_{|V|}\right\}\) be a set of items. The list of interactions of user \(u\ \in\ U\) is denoted by \(S_{u}=\left\{v_{1}^{u},\ v_{2}^{u},\ \ldots,\ v_{n}^{u}\right\}\), where user \(u\) has interacted with item \(v_{t}^{u}\ \in\ V\) at the relative time step \(t\). Each item \(v\in V\) is associated with a set of keywords \(K_{v}=\left\{k_{1},\ k_{2}\ \ldots,\ k_{|K_{v}|}\right\}\) that describe the item \(v\). We refer to set \(K\) as the metadata related to the item. Given the history \(S_{u}\) and the additional metadata \(K_{v_{t}}^{u*}\) for every \(v_{t}^{u}\ \in\ S_{u}\), the POI recommendation task is to predict the next item \(v_{n_{u+1}}^{u}\) in the sequence of a user's interactions. We adapted the deep bidirectional self-attention model BERT [7] for the sequential recommendation task, resulting in our BERT4 Loc model. Our modifications in the original BERT4 REC models are the following. The original BERT4 Rec model has two embeddings: (i) an embedding \(E_{V}\in R^{(|V|\times d)}\) of the POI (item) identifier, and (ii) an additional embedding \(E_{P}\in\ R^{N\times d}\) for the position of the items in the sequence for the Transformer blocks. We include an additional keyword embedding to the Transformer layer and we also consider modification of the loss function. Adding an additional keyword embedding to the Transformer layer: Our first modification to the BERT4 Rec is that we modify the Transformer layer and add an additional embedding \(k_{t}\) of the keywords \(K_{v_{t}}\) of item \(v_{t}\), representing the metadata as additional information. Integrating Bayesian Personalized Ranking (BPR) for computing loss instead of Binary Cross-Entropy (BCE): Our second adaptation in the BERT4 Loc model involves a change in the training process. Instead of the original BCE loss function employed by BERT4 Rec, we integrate the BPR [35] for computing loss. The BCE function uses negative sampling to differentiate between items a user has and has not interacted with. BCE treats the unobserved user-item pairs as negative instances, which may not always reflect the user's actual preferences. BPR is a ranking loss function, which is designed for scenarios involving implicit feedback. We chose BPR over BCE due to the BPR method's effectiveness in handling implicit feedback and optimizing ordinal preferences, making it more suited for \begin{table}
\begin{tabular}{c l} \hline \hline
**Notation** & **Description** \\ \hline \(U\) & Set of users \\ \(V\) & Set of items (POI) \\ \(S_{u}\) & List of interactions of user \(u\) with items \\ \(n_{u}\) & Number of interactions of user \(u\) \\ \(v_{t}^{u}\) & Item at the relative time step \(t\) for user \(u\) \\ \(K_{v}\) & Set of keywords describing item \(v\) \\ \(K\) & Set of side (metadata) information related to the items \\ \(K^{*}\) & Set of all possible keyword combinations \\ \(E_{V}\) & Embedding of the POI (item) identifier \\ \(E_{P}\) & Embedding for the position of items in the sequence \\ \(N\) & Input sequence length \\ \(h_{t}^{0}\) & Sum of item embedding \(e_{t}\) and the positional embedding \(p_{t}\) \\ \(k_{t}\) & Embedding of the keywords \(K_{vt}\) of item \(v_{t}\) \\ \(L\) & Number of Transformer layers \\ \(h_{t}^{L}\) & Last hidden state of the \(L_{th}\) Transformer layer \\ \(BPR\) & Bayesian Personalized Ranking \\ \(X\) & Number of sampled negative items in uniform distribution \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Notations Used. our recommendation task.