RecGPT: Generative Pre-training for Text-based Recommendation (2024)

Abstract: We present the first domain-adapted and fullytrained large language model, RecGPT-7B, and its instruction-following variant, RecGPT-7BInstruct, for text-based recommendation. Experimental results on rating prediction and sequential recommendation tasks show that our model, RecGPT-7B-Instruct, outperforms previous strong baselines. We are releasing our RecGPT models as well as their pretraining and fine-tuning datasets to facilitate future research and downstream applications in text-based recommendation. Public “huggingface” links to our RecGPT models and datasets are available at: https://github. com/VinAIResearch/RecGPT.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–313 August 11-16, 2024 ©2024 Association for Computational Linguistics
We present the first domain-adapted and fullytrained large language model, RecGPT-7B, and its instruction-following variant, RecGPT-7BInstruct, for text-based recommendation. Experimental results on rating prediction and sequential recommendation tasks show that our model, RecGPT-7B-Instruct, outperforms previous strong baselines. We are releasing our RecGPT models as well as their pretraining and fine-tuning datasets to facilitate future research and downstream applications in text-based recommendation. Public “huggingface” links to our RecGPT models and datasets are available at: https://github. com/VinAIResearch/RecGPT.

heading: 1 Introduction
text: Recommendation systems assist in comprehending user preferences and offering suitable content suggestions for users (Ansari et al., 2000; Sarwar et al., 2000; Pazzani and Billsus, 2007). Currently, recommendation systems have found wide applications across various domains, such as e-commerce (Schafer et al., 2001; Kang and McAuley, 2018), news (Wang et al., 2018), and movies (Sun et al., 2019). The evolution of recommendation systems has witnessed a shift from fundamental methods to more sophisticated and modern approaches. Conventional methods mine interaction matrices to exploit user-item relationships (Koren et al., 2009; Konstan et al., 1997; He et al., 2017), and subsequently, they incorporate deep learning techniques such as CNN and RNN to extract item features and capture user preferences (Wang et al., 2018; Hidasi et al., 2016). However, this task-specific setting suffers from data sparsity, a lack of flexibility to capture fluctuations in user preferences over time, and challenges in scaling to a large number of users and extensive datasets. Later works, inspired by attention mechanisms and the Transformers architecture (Vaswani et al., 2017a), model user histories
as sequences of items and then encode information in dense vectors (Kang and McAuley, 2018; Sun et al., 2019; Zhou et al., 2020).
With the advancement of large language models (LLMs), recent works leverage the capacity of LLMs in understanding user preferences (Geng et al., 2023; Rajput et al., 2023). The model P5 (Geng et al., 2022), which represents users and items by IDs, endeavors to aggregate recommendation tasks under a unified conditional generation model based on T5 (Raffel et al., 2020). In addition, Liu et al. (2023) evaluate the potential usage of ChatGPT in different recommendation tasks. More recently, Ji et al. (2024) fine-tune LLaMA (Touvron et al., 2023) with LoRA (Hu et al., 2022) for sequential recommendation. Recommendation tasks frequently exhibit shared characteristics such as user sets, item sets, and interactions, thus suggesting the possibility of training a unified model for multiple tasks, as opposed to employing distinct models for each task. Adopting a single model approach, as done in P5, not only encourages model generalization but also fosters collaborative learning across tasks. However, representing users and items by IDs, as in P5, may not fully align with the textual understanding capability of LLMs. It might be more effective to represent items by their textual descriptions and users by their text-based interaction history with items.
In this paper, (I) we introduce the first domainadapted and fully-trained LLM series named RecGPT for text-based recommendation, which comprises the base pre-trained model RecGPT-7B and its instruction-following variant, RecGPT-7BInstruct. In this context, we pre-train RecGPT-7B using a relatively large recommendation-specific corpus of 20.5B tokens, while RecGPT-7B-Instruct is the model output by further fine-tuning RecGPT7B on a dataset of 100K+ instructional prompts and their responses. (II) We conduct experiments for rating prediction and sequential recommendation
302
tasks, demonstrating that our RecGPT-7B-Instruct outperforms strong baselines, including P5. (III) We publicly release our models along with the pretraining and fine-tuning datasets. We hope that this release can foster future research and applications in text-based recommendation.

heading: 2 Our model RecGPT
text: This section describes the data and outlines the architecture and optimization setup used for RecGPT.

heading: 2.1 Pre-training and Fine-tuning data
text: We collect a rich and comprehensive set of datasets from various domains, including: Amazon Product (McAuley et al., 2015), Anime,1 BookCrossing,2 Food (Majumder et al., 2019), Goodreads (Wan and McAuley, 2018), HotelRec (Antognini and Faltings, 2020), MovieLens (Harper and Konstan, 2015), Netflix (Bennett and Lanning, 2007), Steam,3 WikiRec (AlGhamdi et al., 2021), and Yelp.4 Specifically, we select datasets that contain item titles, a key factor for item representation. Each item is associated with metadata comprising attributes such as title and brand, along with user interactions such as rating and review. We
1 https://www.kaggle.com/datasets/CooperUnion/
anime-recommendations-database 2 https://www.kaggle.com/datasets/ruchi798/bookcrossing-dataset
3 https://www.kaggle.com/datasets/tamber/steam-video-games 4 https://www.yelp.com/dataset
perform a cleaning pre-process on the collected datasets by discarding: (i) items without titles, (ii) users with fewer than 5 interactions, and (iii) all background and demographic user information. Ultimately, we have 10,156,309 users, 10,309,169 items, and 258,100,698 interactions in total. Detailed statistics of each cleaned dataset are shown in Table 4 in Appendix A.
Then we randomly split each cleaned dataset into pre-training/fine-tuning subsets with a 99.5/0.5 ratio at the “user” level (i.e., users in the finetuning subset do not appear in the pre-training subset, and vice versa).5 Regarding pre-training, users are represented solely through their interaction history with items. Each user’s interaction history, referred to as a text document, is formatted as a chronologically-ordered list of text-based data points i1, i2, ..., in, where ik is represented by the corresponding k-th item’s metadata and interactions. For example, in the pre-training sample in Table 1, i1 is “Title: Rock-a-Stack; Brand: Fisher-Price; Review: My son loves to empty this
stacker and play with and teeth on the rings; Rating: 5.0/5.0”. Totally, we create a pre-training corpus of 10M+ documents with 20.5B tokens.
When it comes to fine-tuning for instruction following, given the nature of our datasets, we create prompt-response pairs for two popular tasks in the recommendation system domain: rating prediction and sequential recommendation. For each user with the history i1, i2, ..., in, the last item in is considered as the next item to be predicted in sequential recommendation, given the history context i1, i2, ..., in−1. Meanwhile, the rating of the (n − 1)-th item in−1 is used as the label for rating prediction, given the remaining history context i1, i2, ..., in−1 without the rating of the (n − 1)- th item. Depending on task requirements, unused features within each data point ik of the user history are discarded, streamlining the prompts and their responses for enhanced task relevance and efficiency. Altogether, we create a fine-tuning dataset of 100K+ instructional prompt and response pairs.
Examples of a pre-training document and prompt-response pairs are shown in Table 1. Details on the data formats used in pre-training and fine-tuning are presented in Appendix B.
5There are 4 datasets where we do not apply the 99.5/0.5 ratio. Refer to Section 3.1 for more details.

heading: 2.2 RecGPT-7B
text: RecGPT-7B is a Transformer decoder-based model (Brown et al., 2020; Vaswani et al., 2017b) that incorporates (Triton) flash attention (Dao et al., 2022) and ALiBi (Press et al., 2022) for context length extrapolation. Additionally, we use a “max_seq_len” of 2048, “d_model” of 4096, “n_heads” of 32, “n_layers” of 32, and GPT-NeoX’s tokenizer with a vocabulary of 50K tokens, resulting in a model size of about 7B parameters. Utilizing the Mosaicml “llm-foundry” library,6 we initialize the parameter weights of RecGPT-7B with those from the pretrained MPT-7B (Team, 2023) and continually pretrain on our pre-training corpus of 20.5B tokens. For optimization, we employ the LION optimizer (Chen et al., 2023) and sharded data parallelism with FSDP, set a global batch size of 128 (i.e., 128 * 2048 = 260K tokens per batch) across 8 A100 GPUs (40GB each), and use a peak learning rate of 2.5e-5. The training runs for 2 epochs, using mixed precision training with bfloat16, and takes about 18 days. This is equivalent to 20.5B * 2 / 260K = 157K training steps (here, the learning rate is warmed up for the first 2K training steps).
The total number of GPU hours used for pretraining is 18 * 8 * 24 = 3456. With the GPU power consumption at 400W, the pre-training process uses 3456 * 400 = 1,382,400 Wh, equivalent to the carbon emission of about 0.585 tCO2eq.

heading: 2.3 RecGPT-7B-Instruct
text: We then fine-tune the base pre-trained RecGPT7B for instruction following regarding rating prediction and sequential recommendation, using the dataset consisting of 100K+ instructional prompts and their responses from Section 2.1. We employ LION, set a global batch size of 128 across 8 A100 GPUs (40GB each), use a peak learning rate of 1.0e-5, and run for 2 epochs. The resulting finetuned model is named RecGPT-7B-Instruct.
Fine-tuning RecGPT-7B-Instruct takes 4 hours using a node of 8 A100 GPUs (40GB each), totaling 32 GPU hours. This is equivalent to the carbon emission of about 0.0054 tCO2eq.

heading: 3 Experiments
text: We conduct experiments to compare our RecGPT7B-Instruct with strong baselines for rating prediction and sequential recommendation tasks.
6https://github.com/mosaicml/llm-foundry: A robust library that supports both pre-training and fine-tuning.

heading: 3.1 Experimental setup
text: Evaluation datasets: We carry out experiments on 4 benchmark datasets across different domains, including “Amazon Beauty”, “Amazon Sports and Outdoors” and “Amazon Toys and Games” (McAuley et al., 2015), as well as Yelp. Following previous works (Geng et al., 2022; Ji et al., 2024), for those three Amazon datasets, we employ the 5-core version 2014,7 while for Yelp, we consider transactions from Jan 1, 2019, to Dec 31, 2019.
Data leakage issue: We further discover a data leakage issue that has not been pointed out before. As the four experimental benchmark datasets used in the evaluation are not pre-defined with a trainingvalidation-test split, previous works apply different splitting strategies for each evaluation task (Geng et al., 2022). Let’s consider the Amazon Beauty dataset, which is utilized in training P5 (Geng et al., 2022), as an example (similar findings apply to other datasets). The dataset comprises users, items, and interactions between them. An interaction example may be: user X purchasing item Y and providing a review and rating of 4.0/5.0. The original dataset is presented as interaction records without a predefined training-validation-test split. P5 employs different data splitting strategies for different tasks. For the rating prediction task, P5 randomly divides the data into training, validation, and test sets with an 80-10-10 ratio, respectively. For the sequential recommendation task, P5 aggregates data by user to construct users’ histories, comprising their interactions. Then, P5 utilizes a leave-oneout manner, where the last item in the history is reserved for testing, the second-last item for validation, and the remaining items for training. Consequently, there are interactions in the training set for the rating prediction task, which also belong to the test set for the sequential recommendation task, and vice versa (i.e., there are interactions in the training set in the sequential recommendation task, which also belong to the test set in the rating prediction task). Merging the training sets from both tasks for multitask training, as performed in P5, without filtering out duplicate data results in data leakage.
For a consistent test set, we still reuse their splits but remove interactions from the training set if they appear in the test set. This ensures that the test data is not leaked into the training data. Note that
7https://cseweb.ucsd.edu/~jmcauley/datasets/ amazon/links.html
for these 4 experimental benchmarks, we report our final scores on the test split, while the training split is only used for pre-training RecGPT-7B to mimic real-world scenarios (i.e., we do not use the training/validation split for supervised fine-tuning of instruction following).
Evaluation metrics: For rating prediction, we employ Root Mean Square Error (RMSE) and Mean Absolute Error (MAE), while for sequential recommendation, we use top-k Hit Ratio (HR@k) and top-k Normalized Discounted Cumulative Gain (NDCG@k). Smaller values of RMSE and MAE, and higher values of HR and NDCG, indicate better performance.
Inference: We utilize vLLM (Kwon et al., 2023) as an inference engine. For rating prediction, for a given input prompt, we apply the sampling decoding strategy with “temperature” of 1.0, “top_p” of 0.9 and “top_k” set at 50, and then extract the predicted value from the generated response output. For sequential recommendation, following previous works (Geng et al., 2022; Ji et al., 2024), for a given input prompt, we use the beam search decoding strategy with a beam size of 10 to generate 10 response outputs and use their beam search scores for ranking. In addition, due to the hallucinatory nature of LLMs, the generated outputs might differ slightly from the ground truth labels. Therefore, we implement a semantic similarity matching approach with a text embedding model and a matching module, built on top of Sentence Transformers (Reimers and Gurevych, 2019) and FAISS (Johnson et al., 2021) respectively. This approach utilizes dot product-based similarity over dense vector representations to associate each generated output with the most similar item in the item set.

heading: 3.2 Main results
text: Rating prediction: Table 2 lists rating prediction results for our RecGPT-7B-Instruct and the
previous strong baselines on the four experimental datasets. We find that, in general, pre-trained LLMbased approaches, specifically P5 (Geng et al., 2022), ChatGPT (GPT-3.5-turbo), and RecGPT-7BInstruct, outperform conventional rating prediction methods MF (Koren et al., 2009) and MLP (Cheng et al., 2016). Although ChatGPT is not specifically designed for this task, it demonstrates promising performance scores that surpass those of P5 on the “Beauty” dataset. We find that RecGPT-7B-Instruct achieves the best results across all datasets in terms of both evaluation metrics RMSE and MAE, yielding new state-of-the-art performance scores.
Sequential recommendation: Table 3 presents the obtained results with cutoff thresholds of 5 and 10 for HR and NDCG for different models on the sequential recommendation task. Not surprisingly, ChatGPT, which faces a limitation in terms of indomain data, attains lower scores than other baselines on the “Beauty” dataset. This highlights the crucial role of in-domain training data in sequential recommendation for models to comprehend the item set. GenRec (Ji et al., 2024), fine-tuned
with LoRa (Hu et al., 2022) on the entire training split, does not perform competitively on the “Toys and Games” dataset, compared to the fully finetuned model RecGPT-7B-Instruct. Additionally, our RecGPT-7B-Instruct achieves competitive results with P5 and OpenP5 (Xu et al., 2023) on the “Beauty” dataset. Moreover, RecGPT-7B-Instruct notably outperforms P5 on both the “Sports and Outdoors” and “Toys and Games” datasets.
Ablation analysis: To examine how pre-training contributes to the improvement in the performance scores of RecGPT-7B-Instruct, we also conduct supervised fine-tuning (SFT) for instruction following on the base pre-trained MPT-7B. The finetuning process for MPT-7B is carried out in the same manner as for our RecGPT-7B-Instruct, as detailed in Section 2.3. Tables 2 and 3 also present the results of MPT-7B with SFT. We find that RecGPT7B-Instruct performs substantially better than MPT7B with SFT, highlighting the significant contribution of continual pre-training RecGPT-7B for domain adaptation in the context of recommendation.
In Table 2, rating prediction most likely relies on the review text to predict the score, which might be viewed as a sentiment classification task with more fine-grained labels. This task is thus not as difficult (compared to the sequential recommendation task), given tens of thousands of examples for rating prediction fine-tuning. Also, the base LLM model MPT-7B is pre-trained on a 1T-token corpus that likely contains many reviews from the web. So the substantial improvement of RecGPT7B-Instruct over the baseline “MPT-7B with SFT” for the rating prediction task is not as large as for the sequential recommendation task.

heading: 4 Conclusion
text: We have introduced the first domain-adapted and fully-trained LLMs for text-based recommendation, which include the base pre-trained RecGPT-7B and its instruction-following variant, RecGPT-7BInstruct. We demonstrate the usefulness of RecGPT by showing that RecGPT-7B-Instruct outperforms strong baselines in both rating prediction and sequential recommendation tasks. Through the public release of RecGPT models and the pre-training and supervised fine-tuning datasets, we hope that they can foster future research and applications in text-based recommendation.
Limitations
The knowledge of the LLM about the tasks and the item set is solely based on training data and the intrinsic memory of the base model. Models might not be aware of items that are not covered in the training data. If this incident occurs, models could generate irrelevant information and suffer from hallucinations. This limitation also applies to all LLM-based methods. Furthermore, in this work, we only evaluate two popular tasks; we will conduct experiments for other recommendation tasks in future work.

heading: Acknowledgement
text: We extend our thanks to Khoa D. Doan (khoa.dd@vinuni.edu.vn) for initial discussions.

heading: A Datasets
text: The statistics of our cleaned datasets are presented in Table 4. Note that some datasets have two versions associated with different publication times (e.g., Amazon and Yelp). To maintain consistent test data with previous works (Geng et al., 2022; Xu et al., 2023; Liu et al., 2023), we retain the older versions (2014 for Amazon and 2020 for Yelp) for testing purposes and use the newer versions (2018 for Amazon and 2021 for Yelp) to enrich our pretraining data. We filter out overlapped users along with their interactions in the newer dataset to prevent duplication and data leakage.
Note that if a user has a long interaction history with many items (i.e., the number of tokens exceeds the max_seq_length of 2048), we pre-split the history into smaller chunks with a similar number of items, ensuring that the number of tokens in each chunk is smaller than 2048. Each chunk is then considered a separate user’s interaction history.

heading: B Data format used in training and inference
text: We present the prompt templates used in our work. Note that in both pre-training and fine-tuning phases, if a user has a long interaction history with many items (i.e., the number of tokens exceeds the max_seq_length of 2048), we pre-split the history into smaller chunks with a similar number of items, ensuring that the number of tokens in each chunk is smaller than 2048. Each chunk is then considered a separate user’s interaction history.
B.1 Data format used in pre-training phase
Amazon
Given the interaction history of a user with products as follows: Title: {title}; Brand: {brand}; Review: {review}; Rating: {rating}/5.0 ... Title: {title}; Brand: {brand}; Review: {review}; Rating: {rating}/5.0
Amazon Books
Given the interaction history of a user with books as follows: Title: {title}; Brand: {brand}; Review: {review}; Rating: {rating}/5.0 ... Title: {title}; Brand: {brand}; Review: {review}; Rating: {rating}/5.0
Anime
Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres}; Rating: {rating}/10.0 ... Title: {title}; Genres: {genres}; Rating: {rating}/10.0
BookCrossing
Given the interaction history of a user with books as follows: Title: {title}; Author: {author}; Rating: {rating}/10.0 ... Title: {title}; Author: {author}; Rating: {rating}/10.0
Food
Given the interaction history of a user with food recipes as follows: Title: {title}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; Review: {review_text}; Rating: {rating}/5.0
Goodreads
Given the interaction history of a user with books as follows: Title: {title}; Author: {author}; Genres: {genres}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; Author: {author}; Genres: {genres}; Review: {review_text}; Rating: {rating}/5.0
HotelRec
Given the interaction history of a user with hotels as follows: Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0
MovieLens
Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres}; Rating: {rating}/5.0 ... Title: {title}; Genres: {genres}; Rating: {rating}/5.0
Netflix Given the interaction history of a user with movies/shows as follows: Title: {title}; Rating: {rating}/5.0 ... Title: {title}; Rating: {rating}/5.0
Steam Given the interaction history of a user with video games as follows: Title: {title} ... ...Title: {title}
WikiRec Given the interaction history of a user with Wikipedia articles as follows: Title: {title}; Description: {description} ... Title: {title}; Description: {description}
Yelp Given the interaction history of a user with businesses as follows: Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0
B.2 Data format used in fine-tuning and inference
B.2.1 Rating prediction task Amazon ### Instruction: Predict rating for the last item. Given the interaction history of a user with products as follows: Title: {title}; Brand: {brand}; Review: {review}; Rating: {rating}/5.0 ... Title: {title}; Brand: {brand}; Review: {review}; Rating: ### Response: {rating}/5.0
Amazon Books ### Instruction: Predict rating for the last item. Given the interaction history of a user with books as follows:
Title: {title}; Author: {author}; Review: {review}; Rating: {rating}/5.0 ... Title: {title}; Author: {author}; Review: {review}; Rating: ### Response: {rating}/5.0
Anime
### Instruction: Predict rating for the last item. Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres}; Rating: {rating}/10.0 ... Title: {title}; Genres: {genres}; Rating: ### Response: {rating}/10.0
BookCrossing
### Instruction: Predict rating for the last item. Given the interaction history of a user with books as follows: Title: {title}; Author: {author}; Rating: {rating}/10.0 ... Title: {title}; Author: {author}; Rating: ### Response: {rating}/10.0
Food
### Instruction: Predict rating for the last item. Given the interaction history of a user with food recipes as follows: Title: {title}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; Review: {review_text}; Rating: ### Response: {rating}/5.0
Goodreads
### Instruction: Predict rating for the last item. Given the interaction history of a user with books as follows: Title: {title}; Author: {author}; Genres: {genres}; Review: {review_text}; Rating: {rating}/5.0 ...
Title: {title}; Author: {author}; Genres: {genres}; Review: {review_text}; Rating: ### Response: {rating}/5.0
HotelRec
### Instruction: Predict rating for the last item. Given the interaction history of a user with hotels as follows: Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; City: {city}; Review: {review_text}; Rating: ### Response: {rating}/5.0
MovieLens
### Instruction: Predict rating for the last item. Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres}; Rating: {rating}/5.0 .. Title: {title}; Genres: {genres}; Rating: ### Response: {rating}/5.0
Netflix
### Instruction: Predict rating for the last item. Given the interaction history of a user with movies/shows as follows: Title: {title}; Rating: {rating}/5.0 ... Title: {title}; Rating: ### Response: {rating}/5.0
Yelp
### Instruction: Predict rating for the last item. Given the interaction history of a user with businesses as follows: Title: {title}; City: {city}; Review: {review_text}; Rating: {rating}/5.0 ... Title: {title}; City: {city}; Review: {review_text}; Rating: ### Response: {rating}/5.0
B.2.2 Sequential recommendation task Amazon
### Instruction: Predict the next item. Given the interaction history of a user with products as follows: Title: {title}; Brand: {brand} ... Title: {title}; Brand: {brand} ### Response: Title: {title}; Brand: {brand}
Amazon Books
### Instruction: Predict the next item. Given the interaction history of a user with books as follows: Title: {title}; Author: {brand}; ... Title: {title}; Author: {brand}; ### Response: Title: {title}; Author: {brand};
Anime
### Instruction: Predict the next item. Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres} ... Title: {title}; Genres: {genres} ### Response: Title: {title}; Genres: {genres}
BookCrossing
### Instruction: Predict the next item. Given the interaction history of a user with books as follows: Title: {title}; Author: {author} ... Title: {title}; Author: {author} ### Response: Title: {title}; Author: {author}
Food
### Instruction: Predict the next item. Given the interaction history of a user with food recipes as follows: Title: {title} ... Title: {title} ### Response:
Title: {title}
Goodreads
### Instruction: Predict the next item. Given the interaction history of a user with books as follows: Title: {title}; Author: {author}; Genres: {genres} ... Title: {title}; Author: {author}; Genres: {genres} ### Response: Title: {title}; Author: {author}
HotelRec
### Instruction: Predict the next item. Given the interaction history of a user with hotels as follows: Title: {title}; City: {city} ... Title: {title}; City: {city} ### Response: Title: {title}; City: {city}
MovieLens
### Instruction: Predict the next item. Given the interaction history of a user with movies/shows as follows: Title: {title}; Genres: {genres} .. Title: {title}; Genres: {genres} ### Response: Title: {title}
Netflix
### Instruction: Predict the next item. Given the interaction history of a user with movies/shows as follows: Title: {title} ... Title: {title} ### Response: Title: {title}
Steam
### Instruction: Predict the next item. Given the interaction history of a user with video games as follows: Title: {title} ...
Title: {title} ### Response: Title: {title}
WikiRec ### Instruction: Predict the next item. Given the interaction history of a user with Wikipedia articles as follows: Title: {title}; Description: {description} ... Title: {title}; Description: {description} ### Response: Title: {title}; Description: {description}
Yelp ### Instruction: Predict the next item. Given the interaction history of a user with businesses as follows: Title: {title}; City: {city} ... Title: {title}; City: {city} ### Response: Title: {title}; City: {city}

heading: Future Work
text: Through the public release of RecGPT models and the pre-training and supervised fine-tuning datasets, we hope that they can foster future research and applications in text-based recommendation. Limitations
The knowledge of the LLM about the tasks and the item set is solely based on training data and the intrinsic memory of the base model. Models might not be aware of items that are not covered in the training data. If this incident occurs, models could generate irrelevant information and suffer from hallucinations. This limitation also applies to all LLM-based methods. Furthermore, in this work, we only evaluate two popular tasks; we will conduct experiments for other recommendation tasks in future work