Experiments and Results	In this section, we first describe two different $k$-core subsets of the HotelRec dataset that we used to evaluate multiple baselines on two tasks: rating prediction and recommendation performance. We then detail the models we employed, and discuss their results.

Experiments and Results ::: Datasets	We used the aforementioned dataset HotelRec, containing approximately 50 million hotel reviews. The characteristics of this dataset are described in Section SECREF12 and Section SECREF18 Following the literature , , we focused our evaluation on two $k$-core subsets of HotelRec, with at least $k$ reviews for each user or item. In this paper, we employed the most common values for $k$: 5 and 20. We randomly divided each of the datasets into $80/10/10$ for training, validation, and testing subsets. From each review, we kept the corresponding "userID", "itemID", rating (from 1 to 5 stars), written text, and date. We preprocessed the text by lowering and tokenizing it. Statistics of both subsets are shown in Table TABREF2.

Unlike commonly used recommendation datasets, the hotel domain suffers from higher data sparsity and therefore, traditional collaborative-filtering approaches cannot be applied , , . Furthermore, rating a hotel is different than traditional products, because the whole experience lasts longer, and there are more facets to review . In contrast, we propose in this work HotelRec, a novel large-scale hotel recommendation dataset based on hotel reviews from TripAdvisor, and containing approximately 50 million reviews. A sample review is shown in Figure FIGREF1. To the best of our knowledge, HotelRec is the largest publicly available hotel review dataset (at least 60 times larger than previous datasets). Furthermore, we analyze various aspects of the HotelRec dataset and benchmark the performance of different models on two tasks: rating prediction and recommendation performance. Although reasonable performance is achieved by a state-of-the-art method, there is still room for improvement. We believe that HotelRec will offer opportunities to apply and develop new large recommender systems, and push furthermore the recommendation for hotels, which differs from traditional datasets.

HotelRec	Everyday a large number of people write hotel reviews on on-line platforms (e.g., Booking, TripAdvisor) to share their opinions toward multiple aspects, such as their Overall experience, the Service, or the Location. Among the most popular platforms, we selected TripAdvisor: according to their third quarterly report of November 2019, on the U.S. Securities and Exchange Commission website, TripAdvisor is the world's largest online travel site with approximately $1.4$ million hotels. Consequently, we created our dataset HotelRec based on TripAdvisor hotel reviews. The statistics of the HotelRec dataset, the 5-core, and 20-core versions are shown in Table TABREF2; each contains at least $k$ reviews for each user or item. In this section, we first discuss about the data collection process (Section SECREF8), followed by general descriptive statistics (Section SECREF12). Finally, Section SECREF18 analyzes the overall rating and sub-ratings.

Experiments and Results ::: Rating Prediction	We show in Table TABREF35 the performance in terms of the mean square error (MSE) and the root mean square error (RMSE). Surprisingly, we observe that the neural network TransNet and its extension perform poorly in comparison to the matrix factorization model HFT and the simple Mean baselines. Although TransNet learns a user and item profile based on the most recent reviews, it cannot capture efficiently the interaction from these profiles. Moreover, the additional collaborative-filtering component in TransNet-Ext seems to worsen the performance, which is consistent with the results of musat2013recommendation; in the hotel domain, the set users who have rated the same hotels is sparser than usual recommendation datasets. Interestingly, the Mean model obtains the best performance on the 20-core subset, while HFT achieves the best performance on the 5-core subset. We hypothesize that HFT and TransNet(-Ext) models perform better on the 5-core than 20-core subset, because of the number of data. More specifically, HFT employs Latent Dirichlet Allocation  to approximate topic and word distributions. Thus, the probabilities are more accurate with a text corpus approximately ten times larger.

HotelRec: a Novel Very Large-Scale Hotel Recommendation Dataset	Today, recommender systems are an inevitable part of everyone's daily digital routine and are present on most internet platforms. State-of-the-art deep learning-based models require a large number of data to achieve their best performance. Many datasets fulfilling this criterion have been proposed for multiple domains, such as Amazon products, restaurants, or beers. However, works and datasets in the hotel domain are limited: the largest hotel review dataset is below the million samples. Additionally, the hotel domain suffers from a higher data sparsity than traditional recommendation datasets and therefore, traditional collaborative-filtering approaches cannot be applied to such data. In this paper, we propose HotelRec, a very large-scale hotel recommendation dataset, based on TripAdvisor, containing 50 million reviews. To the best of our knowledge, HotelRec is the largest publicly available dataset in the hotel domain (50M versus 0.9M) and additionally, the largest recommendation dataset in a single domain and with textual reviews (50M versus 22M). We release HotelRec for further research: this https URL.

To the best of our knowledge, only a few number of datasets for hotel reviews have been created: 35k , 68k , 140k , 142k , 235k , 435k , and 870k . However, the number of users, items, and interactions is limited compared to traditional recommendation datasets. In contrast, the HotelRec dataset has at least two orders of magnitude more examples. Statistics of HotelRec is available in Table TABREF2.

Conclusion	In this work, we introduce HotelRec, a novel large-scale dataset of hotel reviews based on TripAdvisor, and containing approximately 50 million reviews. Each review includes the user profile, the hotel URL, the overall rating, the summary, the user-written text, the date, and multiple sub-ratings of aspects when provided. To the best of our knowledge, HotelRec is the largest publicly available dataset in the hotel domain ($50M$ versus $0.9M$) and additionally, the largest recommendation dataset in a single domain and with textual reviews ($50M$ versus $22M$). We further analyze the HotelRec dataset and provide benchmark results for two tasks: rating prediction and recommendation performance. We apply multiple common baselines, from non-personalized methods to competitive models, and show that reasonable performance could be obtained, but still far from results achieved in other domains in the literature. In future work, we could easily increase the dataset with other languages and use it for multilingual recommendation. We release HotelRec for further research: https://github.com/Diego999/HotelRec.

HotelRec ::: Overall and Sub-Ratings	When writing a review, the Overall rating is mandatory: it represents the evaluation of the whole user experience towards a hotel. It is consequently available for all reviews in HotelRec. However, sub-ratings only assess one or more particular aspects (up to eight), such as Service, Cleanliness, or Location. Additionally, they are optional: the user can choose how many and what aspects to evaluate. Among all the reviews, $35\,836\,414$ ($71.30\%$) have one or several sub-ratings, with a maximum of eight aspects. The distribution of the number of assessed fine-grained aspects is shown in Table TABREF19, where All represents the coverage over the whole set of reviews, and With Sub-Ratings over the set of reviews having sub-ratings (i.e., approximately 35 million). Interestingly, most of the sub-ratings are evaluated in a group of three or six aspects. We hypothesize that this phenomenon came from a limitation of TripAdvisor on the user interface, where the set of aspects to evaluate was predefined. We analyze in Table TABREF20 the distribution of the reviews with fine-grained and Overall ratings.

Experiments and Results ::: Evaluation Metrics and Baselines	We evaluated different models on the HotelRec subsets, 5-core and 20-core, on two tasks: rating prediction and recommendation performance. We have separated the evaluation because most models are only tailored for one of the tasks but not both. Therefore, we applied different models for each task and evaluated them separately. For the rating prediction task, following the literature, we reported the results in terms of Mean Square Error (MSE) and Root Mean Square Error (RMSE). We assessed the recommendation performance of a ranked list by Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) , as in he2017neural. We truncated the ranked list at 5, 10 and 20. The HR measures whether a new item is on the top-$k$ list and NDCG measures the position of the hit by assigning higher scores to hits at top ranks. As in he2017neural, we computed both metrics for each test user and reported the average score. Regarding the models, we employed the following baselines: Mean: A simple model that predicts a rating by the mean ratings of the desired item.

Experiments and Results ::: Recommendation Performance	The results of the baselines are available in Table TABREF36. HR@$k$ and NDCG@$k$ correspond to the Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG), evaluated on the top-$k$ computed ranked items for a particular test user, and then averaged over all test users. First, we can see that NeuMF significantly outperforms all other baselines on both $k$-core subsets. The other methods GMF and MLP - both used within NeuMF - also show quite strong performance and comparable performance. However, NeuFM achieves higher results by fusing GMF and MNLP within the same model. Second, if we compare ItemKNN and UserKNN, we observe that on both subsets, the user collaborative filtering approach underperform compared to its item-based variant, that matches the founding in the rating prediction task of the previous section, and the work of musat2013recommendation,musat2015personalizing. Additionally, PureSVD achieves comparable results with UserKNN. Finally, the two non-personalized baselines RAND and POP obtain unsurprisingly low results, indicating the necessity of modeling user's preferences to a personalized recommendation.

Introduction	The increasing flood of information on the web creates a need for selecting content according to the end user's preferences. Today, recommender systems are deployed on most internet platforms and play an important role in everybody's daily digital routine, including e-commerce websites, social networks, music streaming, or hotel booking. Recommender systems have been investigated over more than thirty years . Over the years, many models and datasets in different domains and various sizes have been developed: movies , Amazon products , , or music . With the tremendous success of large deep learning-based recommender systems, in better capturing user-item interactions, the recommendation quality has been significantly improved . However, the increase in recommendation performance with deep learning-based models comes at the cost of large datasets. Most recent state-of-the-art models, such as , , or  necessitate large datasets (i.e., millions) to achieve high performance. In the hotel domain, only a few works have studied hotel recommendation, such as  or . Additionally, to the best of our knowledge, the largest publicly available hotel review dataset contains $870k$ samples .

Related Work	Recommendation is an old problem that has been studied from a wide range of areas, such as Amazon products , beers , restaurants, images , music , and movies . The size of the datasets generally varies from hundreds of thousands to tens of millions of user-item interactions; an interaction always contains a rating and could have additional attributes, such as a user-written text, sub-ratings, the date, or whether the review was helpful. At the time of writing, and to the best of our knowledge, the largest available recommendation corpus on a specific domain and with textual reviews, is based on Amazon Books and proposed by he2016ups. It contains a total of 22 million book reviews. In comparison, HotelRec has $2.3$ times more reviews and is based on hotels. Consequently, HotelRec is the largest domain-specific public recommendation dataset with textual reviews and on a single domain. We highlight with textual reviews, because some other datasets (e.g., Netflix Prize ) contain more interactions, that only includes the rating and the date.

Finally, the average user has $2.24$ reviews, and the median is $1.00$. Relating to the items, there are $365\,056$ hotels, which is roughly 60 times smaller than the number of users. This ratio is also consistent with other datasets , . Figure FIGREF13 displays the distribution of reviews per hotel. The distribution also has a shape of a Power-law distribution, but its center is closer to $3\,000$ than the 100 of the user distribution. However, in comparison, only $0.26\%$ hotels have less than five reviews and thus, the average reviews per hotel and the median are higher: $137.69$ and $41.00$. Finally, we analyze the distribution of words per review, to understand how much people write about hotels. The distribution of words per review is shown in Figure FIGREF13. The average review length is $125.57$ words, which is consistent with other studies .

HotelRec ::: Descriptive Statistics	HotelRec includes $50\,264\,531$ hotel reviews from TripAdvisor in a period of nineteen years (from February 1, 2001 to May 14, 2019). The distribution of reviews over the years is available in Figure FIGREF13. There is a significant activity increase of users from 2001 to 2010. After this period, the number of reviews per year grows slowly and oscillates between one to ten million. In total, there are $21\,891\,294$ users. The distribution of reviews per user is shown in Figure FIGREF13. Similarly to other recommender datasets , , the distribution resembles a Power-law distribution: many users write one or a few reviews. In HotelRec, $67.55\%$ users have written only one review, and $90.73\%$ with less than five reviews. Additionally, in the 5-core subset, less than $15\%$ of $2\,012\,162$ users had a peer with whom they have co-rated three or more hotels.

It is a common baseline to benchmark the recommendation performance; ItemKNN/UserKNN : Two standard item-based (respectively user-based) collaborative filtering methods, using $k$ nearest neighbors; PureSVD : A similarity based approach that constructs a similarity matrix through the SVD decomposition of the rating matrix; GMF : A generalization of the matrix factorization method that applies a linear kernel to model the latent feature interactions; MLP : Similar than GMF, but it models the interaction of latent features with a neural network instead of a linear kernel; NeuMF : A model combining GMF and MLP to better model the complex user-item interactions. Due to the large size of the HotelRec dataset, especially in the 5-core setting (around 20 million reviews), running an extensive hyper-parameter tuning for each neural model would require a high time and resource budget. Therefore, for the neural model, we used the default parameters from the original implementation and a random search of three trials. For all other models (i.e., HFT, ItemKNN, UserKNN, PureSVD), we ran a standard grid search over the parameter sets.

Unsurprisingly, the Overall rating is always available as it is mandatory. In terms of aspects, there is a group of six that are majorly predominant (following the observation in Table TABREF19), and two that are rarely rated: Check-In and Business Service. Surprisingly, these two aspects are not sharing similar rating averages and percentiles than the others. We explain this difference due to the small number of reviews rating them (approximately $2\%$). Furthermore, most ratings across aspects are positive: the 25th percentile is 4, with an average of $4.23$ and a median of 5. Finally, in Figure FIGREF21, we computed the Pearson correlation of ratings between all pairs of aspects, including fine-grained and Overall ones. Interesting, all aspect-pairs have a correlation between $0.46$ and $0.83$. We observe that Service, Value, and Rooms correlate the most with the Overall ratings. Unsurprisingly, the aspect pair Service-Check In and Rooms-Cleanliness have a correlation of $0.80$, because people often evaluate them together in a similar fashion.

For each review, we collected: the URL of the user's profile and hotel, the date, the overall rating, the summary (i.e., the title of the review), the written text, and the multiple sub-ratings when provided. These sub-ratings correspond to a fine-grained evaluation of a specific aspect, such as Service, Cleanliness, or Location. The full list of fine-grained aspects is available in Figure FIGREF1, and their correlation in Section SECREF18 We naively parallelized the crawling on approximately 100 cores for two months. After removing duplicated reviews, as in mcauley2013hidden, we finally collected $50\,264\,531$ hotel reviews.

Interestingly, Location is the aspect that correlates the least with the others, followed by Business Service, and Check-In.

It is a good baseline in recommendation ; HFT : A latent-factor approach combined with a topic model that aims to find topics in the review text that correlate with latent factors of the users and the items; TransNet(-Ext): The model is based on zheng2017joint, which learns a user and item profile based on former reviews using convolutional neural networks, and predicts the ratings using matrix factorization methods afterward. They added a regularizer network to improve performance. TransNet-Ext is an extension of TransNet by using a collaborative-filtering component in addition to user and item reviews history. For the recommendation performance task, we used the following models : RAND: A simple model recommending random items; POP : Another non-personalized recommender method, where items are recommended based on their popularity (i.e., the number of interactions with users).

HotelRec ::: Data Collection	We first crawled all areas listed on TripAdvisor's SiteIndex. Each area link leads to another page containing different information, such as a list of accommodations, or restaurants; we gathered all links corresponding to hotels. Our robot then opened each of the hotel links and filtered out hotels without any review. In total, in July 2019, there were $365\,056$ out of $2\,502\,140$ hotels with at least one review. Although the pagination of reviews for each hotel is accessible via a URL, the automatic scraping is discouraged: loading a page takes approximately one second, some pop-ups might appear randomly, and the robot will be eventually blocked because of its speed. We circumvented all these methods by mimicking a human behavior with the program Selenium, that we have linked with Python. However, each action (i.e., disabling the calendar, going to the next page of reviews) had to be separated by a time gap of one second. Moreover, each hotel employed a review pagination system displaying only five reviews at the same time, which majorly slowed down the crawling. An example review is shown in Figure FIGREF1.

Question: Did they experiment on this dataset?
Answer: Yes