# BERT4 Loc: BERT for Location--POI Recommender System
## 4 Experimental Setup
### Evaluation Methodology  
For our experiments, we used a time-based splitting criterion for both datasets. In time-based splitting, the test set indexes are greater than those of the validation set, which are greater than those of the training set. As an example, if a dataset contains at least twenty ratings for each item and user, the available ratings can be divided into train, validation, and test sets. The test set contains the user's most recent six ratings, while the validation set contains the user's previous four ratings. The train set for this dataset is comprised of all previous ratings (at least ten).  
In our work, we sorted the data by the timestamp of user interaction in descending order and split them accordingly. We used the leave-one-out data splitting method where, for each user, we held out the last item of the behavior sequence as the test data, treated the item just before the last as the validation set, and utilized the remaining items for training. We use the following evaluation metrics in this paper.  
Figure 4: Distribution of sentiment polarities on the reviews of both datasets.  
Figure 5: Cities with the most reviews and best ratings.  
Precision is a measure for computing the fraction of relevant items out of all the recommended items. We average the metric for each user \(u\) to obtain the final result. Precision is shown in Equation (1).  
\[\text{Precision@}k=\frac{1}{|U|}\sum\limits_{u\in U}\frac{\left|\hat{R}(u)\cap R( u)\right|}{|\hat{R}(u)|} \tag{1}\]  
where \(R_{u}\) represents the ground-truth set of items that user \(u\) has interacted with and \(\left|\hat{R}(u)\right|\) represents the item count of \(\hat{R}(u)\).  
Recall is a measure for computing the fraction of relevant items out of all relevant items. Recall is defined in Equation (2).  
\[\text{Recall@}k=\frac{1}{|U|}\sum\limits_{u\in U}\frac{\left|\hat{R}(u)\cap R( u)\right|}{|\hat{R}(u)|} \tag{2}\]  
Hit Ratio (\(HR\)) is a way of calculating how many 'hits' are in a \(k\)-sized list of ranked items. If there is at least one item that falls in the ground-truth set, it is called a hit. HR is defined in Equation (3).  
\[HR@k=\frac{1}{|U|}\sum\limits_{u\in U}\delta\left(\hat{R}(u)\cap R(u)\ \
eq\varnothing\right) \tag{3}\]  
where \(\delta(.)\) is an indicator function; \(\delta(.)=1\) if it is a hit and 0 if otherwise. \(\varnothing\) denotes the empty set.  
Normalized Discounted Cumulative Gain (NDCG) is a measure of ranking quality, where positions are discounted logarithmically. It accounts for the position of the hit by assigning higher scores to hits at top ranks. It is defined as in Equation (4).  
\[NDCG@k=\frac{1}{|U|}\sum\limits_{u\in U}\left(\frac{1}{\sum\limits_{i=1}^{\min \left|\left|\left|R(u)\right|\cdot K\right|}\frac{1}{\log_{2}(i+1)}\ \sum\limits_{i=1}^{K}\delta(i\in R(u))\ \frac{1}{\log_{2}(i+1)}\right)\right. \tag{4}\]  
where \(\delta(.)\) is an indicator function.  
We consider the top-k values, where k is 10, 20, and 50. We compared the performance of various recommender systems in top-k metrics, including Precision, Recall, Fl-score, Hit Rate (HR), and Normalized Discounted Cumulative Gain (NDCG), with their respective Mean and Standard Deviation (Mean \(\pm\) SD) values. The results were reported for both datasets. Regarding the calculation of Mean values, we took the average of the performance metric values across all users or items in the test set. For example, to calculate the Mean Precision, we first calculated the Precision value for each user or item in the test set, and then took the average of these values. The Standard Deviation (SD) was calculated to measure the variability of the performance metric values across all users or items in the test set.