The Art of Asking: Prompting Large Language Models for Serendipity Recommendations

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-26 04:13:15

Abstract: Serendipity means an unexpected but valuable discovery. Its elusive nature makes it susceptible to modeling. In this paper, we address the challenge of modeling serendipity in recommender systems using Large Language Models (LLMs), a recent breakthrough in AI technologies. We leveraged LLMs’ prompting mechanisms to convert a problem of serendipity recommendations into a problem of formulating a prompt to elicit serendipity recommendations. The formulated prompt is called SerenPrompt. We designed three types of SerenPrompt: discrete with natural words, continuous with trainable tokens, and hybrid that combines the previous two types. In the meanwhile, for each type of SerenPrompt, we also designed two styles: direct and indirect, to investigate whether it is feasible to directly ask an LLM a question on whether an item is a serendipity, or we should breakdown the question into several sub-questions. Extensive experiments have demonstrated the effectiveness of SerenPrompt in generating serendipity recommendations, compared to the state-of the-art models. The combination of the hybrid type and the indirect style achieves the best performance, with relatively low sacrifice to computational efficiency. The results demonstrate that natural words and virtual tokens, as building blocks of LLM prompts, each have their own advantages. The better performance of the indirect style speaks to the effectiveness of decomposing the direct question on serendipity.

Main paper: 
 introduction: 
 Serendipity means an unexpected but valuable discovery. As early as 1997, Gup [6] expressed his concern about the "end of serendipity" in the digital world, recalling with the fondness his childhood experiences coming across interesting tidbits of information while flipping encyclopedia pages. Gup's concerns are echoed by others in more recent studies (e.g., [18, 21, 23] ). The sense that the online environment is increasingly determined promotes a widespread feeling that serendipity is threatened.Even with today's deep learning models, modeling serendipity is difficult due to the elusive nature of serendipity. The element of unexpectedness in serendipity means surprise and accident, which are susceptible to modeling and prediction. The recent rise of Large Language Models (LLMs), especially ChatGPT, has brought global excitement about what AI could do for humans. LLMs begin taking on surprising emergent abilities when they reach a certain size. In line with Anderson's well-known suggestion that "more is different" [1] , LLMs appear to go through a form of phase transition, bringing about new capacities for which they were not explicitly trained. In this paper, we investigated whether LLMs have such emergent capacities that are helpful for serendipity recommendations, a long-standing research challenge in recommender systems research community.Since last year, initial efforts have been made to explore the potential of LLMs as a promising technique for the next generation recommender systems, due to the fact that recommender systems could be regarded as question answering (QA) systems: given a question of a user's previous preferences, the system generates an answer about this user's future preferences. In addition, recommender systems typically contain a large amount of text information, such as user reviews, item descriptions, which aligns with the data format of an LLM. Specifically, we will leverage LLMs' prompting mechanism, the new paradigm compared to the pre-training and fine-tuning paradigm for a language model, to convert the problems of serendipity recommendations into problems of formulating prompts, to elicit serendipity recommendations. The resulting prompts are called SerenPrompt. We experimented with discrete prompts with manually selected natural words, continuous prompts with trainable tokens during a lightweight model training, and hybrid prompts that combine natural words and trainable tokens.Equally importantly, serendipity is a difficult concept to study due to its elusive and subjective nature. Most studies on serendipity have decomposed the concept into a few more tangible subconcepts, such as diversity, novelty, coverage, unexpectedness, surprise, interestingness, value, and relevance. Very few studies have studied serendipity as a whole concept. Since LLMs are believed to be an "encyclopedia", containing comprehensive knowledge of human society through extensive pre-training, we would like to investigate whether it is feasible to directly ask an LLM a question on whether an item is a serendipity, or we should decompose the question into several sub-questions on the sub-concepts of serendipity.The major contribution of this paper is three-fold:• the design of SerenPrompt with three types and two styles, as the first work exploring how to prompt LLMs for serendipity recommendations • the decomposition of the task of recommending serendipity into two sub-tasks: recommending the unexpected and recommending the relevant • a computational approach to "calculate" the ground truth of unexpectedness for the core pre-training task, avoiding a tedious human labeling process 
related work: 
 This project draws on several research lines. We will review these areas in the following subsections.The word "serendipity" is used to describe the process of making unexpected discoveries by accident. In the early 2000s, serendipity was first introduced to the context of recommender systems to broaden users' selections and increase their satisfaction [7] . While perceived as valuable, serendipity is also seen as elusive, unpredictable, and hard to control to be used [3] . Although there is some disagreement as to the precise nature of serendipity, all accounts agree that the following two aspects are central: an unexpected chance and a relevant discovery. These two aspects have informed us on the design of SerenPrompt.It is worth distinguishing between serendipity, diversity, and novelty because they share some common characteristics. We believe diversity increases the chance of serendipity, but not every diversified piece is serendipitous: only those unexpected and relevant items are serendipity. As to novelty, it means being new and unknown, not necessarily unexpecting or surprising. In contrast, serendipity suggests how strongly an item violates an expectation. Therefore, it is worth clarifying that, different from many other studies (e.g., [2, 10, 27] ), we do not use diversity and novelty as the sub-concepts of serendipity. In this paper, we interpret and operationalize the concept of serendipity using two sub-concepts: unexpectedness and relevance.Since 2018, a few information retrieval researchers have attempted to build deep learning models for serendipity recommendations. Examples are SerRec [20] , HAES [13] , DESR [27] , NSR [27] , PURS [12] , and SNPR [28] . These conventional deep learning efforts collectively demonstrate the potential of neural networks in representing users' serendipity needs. However, these studies' serendipity definitions varied in order to leverage various existing recommendation datasets and avoid collecting the direct ground truth on serendipity, making both the models and the results not sufficiently systematic or generalizable. In addition, the conventional deep learning models' limited sequence representation capacity and limited natural language understanding capability make their performance as a recommendation model not ideal, especially for those complex and multi-step recommendation tasks including serendipity recommendations. Therefore, in this paper, we investigated the potential of LLMs as recommendation models for this long-standing challenging task of serendipity recommendations.Technically, there are three main ways to leverage LLMs for recommendation tasks: pre-training, fine-tuning, and prompting. Practically, both pre-training and fine-tuning an LLM need heavy computational resources, usually not immediately available in academia.Prompting therefore becomes the popular access to LLMs, to adapt a recommendation task into a question answering task or a language generation task, instead of the other way around during the pre-training and fine-tuning process.Prompting is the new paradigm for adapting LLMs to specific downstream tasks. A prompt refers to a text template that serves as the input of LLMs. Prompting enables LLMs to unify different downstream tasks into language generation tasks [5] . Generally, the types of prompts can be categorized as discrete and continuous [16] . Discrete prompts consist of natural words, relying heavily on human experiences to craft. Although discrete prompts have succeeded in many tasks, handcrafted prompts may be with costs and not globally optimal. Continuous prompts introduce learnable prompt tokens to automatically search for the best prompt templates. In the following paragraph, we will briefly review the recent efforts of using the prompting techniques (both discrete and continuous) for recommendation tasks.A straightforward prompting approach is discrete prompting. For instance, Liu et al. [15] employ ChatGPT and propose separate task descriptions with a few demonstrations (examples) tailored to different types of recommendation tasks, such as top-K recommendations, rating predictions, and explanation generation. In contrast to discrete prompts, continuous prompting employs learnable tokens (vectors or text embeddings) as a prompt. For instance, Wu et al. [26] apply contrastive learning to capture user representations and apply them into prompt tokens. In addition to directly using pre-calculated embeddings, continuous prompts can also be learned using the current task-specific loss function. For example, Zhang et al. [29] adopt randomly initialized continuous prompts and optimize them with respect to a recommendation loss function. Compared to discrete prompts, continuous prompts are more flexible for tuning on a continuous space but at the cost of explainability [8] .Those efforts mentioned above demonstrate the promising potential of prompting LLMs for recommendation tasks. All of those efforts are for the conventional accuracy-oriented recommendations. In this paper, we would like to investigate the feasibility of LLMs for serendipity-oriented recommendations, a more complex and challenging task. We believe LLMs have a huge potential for this task because of their stronger sequence representation capacity and language understanding ability. To convert the serendipity recommendation problem to a prompt to an LLM, we need to include the user information 𝐻 𝑢 and the item information 𝑖 in the prompt. Commonly, a well-designed prompt for an LLM contains three parts: a task description, a few demonstrations (examples), and an input question. The task description defines a task and introduces the related concepts. The demonstrations provide some task and solution examples for the LLM to better understand the task. The input question directly asks the question. In our case, given a task description 𝐼 , a demonstration set 𝐷 = {(𝑑 1 , 𝑑 2 , ..., 𝑑 𝑘 }, and the input question 𝑄, which includes the current user 𝑢's history 𝐻 𝑢 and item 𝑖's information, the prediction ŷ𝑢,𝑖 generated from LLMs can be formulated as follows:EQUATIONWe are interested to know what kind of prompt templates benefit the performance of serendipity recommendations. To this end, we designed three types of templates, as shown in Figure 1: 1) discrete with natural words, 2) continuous with vector tokens to search for the best prompt, and 3) hybrid that combines the discrete and continuous templates. 
prompting llms via discrete templates: 
 As the most common type of prompts, discrete templates formulate the prompts using natural language. We designed two styles of discrete templates. One style is direct: directly asking whether an item is a serendipity. The second style is indirect, asking whether an item is unexpected and relevant, and then inferring whether it is a serendipity.Direct. This is the direct way to ask an LLM whether a candidate item is a serendipity to a user. As in Table 1 for this Style, the task description 𝐼 contains the definition of serendipity and the task requirement. In the demonstration set 𝐷, we expect the LLM learns from the representative examples to better understand the task, similar to the idea of providing some additional task-specific training instances in a supervised machine learning approach. We provide both serendipity (positive) and non-serendipity (negative) examples. For the input question 𝑄, we include the user information 𝐻 𝑢 and the candidate item information 𝑖. Specifically, 𝐻 𝑢 is a series of item names that have been interacted by the user 𝑢. 𝑖 is just the candidate item name. In addition to 𝐻 𝑢 and 𝑖, 𝑄 also limits the answer format to be binary: "Yes" or "No" on whether the candidate item is a serendipity to this user, in order to prevent the LLM from being verbose or digressing.Indirect. This style prompts an LLM to breakdown the task of serendipity recommendations into two sub-tasks: judging whether an item is unexpected and then whether the item is relevant. Therefore, as in Table 1 for this Style, the task description 𝐼 , in addition to defining serendipity, further provides the definitions of being unexpected and being relevant in the recommendation context. Accordingly, the demonstration set 𝐷 contains examples of serendipity items satisfying both conditions: being unexpected and being relevant. The input question 𝑄 is similar to that of Discrete Style 1, but with an extra requirement to consider those two conditions when answering the question. Through this way, the serendipity recommendation task is converted into two question answering tasks.The two styles above are to exploit both the direct and the indirect knowledge contained in an encyclopedia-like LLM through a series of predefined natural language prompts and the pre-specified possible answer words (yes or no). This is the core philosophy of the prompt learning paradigm, i.e., predicting an answer word from the LLM's vocabulary, as if the task-specific prompts had been inserted into the large corpus for training the LLM.On the other hand, these manually designed templates, though with well-crafted statements, are obviously not exhaustive for all possible designs. Therefore we will use some virtual tokens to search for a few more template designs using the the continuous prompting approach. Different from the discrete prompts, continuous prompts use learnable tokens (vectors) in the task description 𝐼 and the input question 𝑄. The two sets of learnable tokens can be generated by inputting the original natural words of 𝐼 and 𝑄 through neural network layer(s) added in front of the LLM. The token generation process could be described as follows:EQUATIONwhere 𝐼 and 𝑄 are the natural words of the task description and the input question. 𝑓 (•) is the added neural network layer(s), to map the natural words into two sets of tokens of[𝑃 𝐼 1 ]...[𝑃 𝐼 𝑛 1 ] and [𝑃 𝑄 1 ]...[𝑃 𝑄 𝑛 2 ]respectively. 𝑛 1 and 𝑛 2 are their numbers of tokens. Compared with the discrete prompts, the demonstration set 𝐷 is not included since 𝑓 (•) can be trained by training instances, which essentially play the role of providing additional demonstrations to the LLM. Therefore the continuous prompts are all tokens, as in Table 2 . Previous research shows that Transformers [25] are effective in mapping or encoding text into vectors. Therefore, in this paper, we selected the Transformers as 𝑓 (•).This way, the continuous prompts provide more freedom by adding learnable tokens in search of the best prompts, although it may introduce some uncertainties and cost some explainability. Similar to the discrete prompts, we designed two styles: direct and indirect, corresponding to the direct question on serendipity and the indirect questions on unexpectedness and relevance.Direct. This Style trains 𝑓 (•) to learn the tokens for the direct question on serendipity. The input of 𝑓 (•), 𝐼 , and 𝑄, will be the same as Discrete Style 1 as in Table 1 . During the learning process, the LLM is frozen, and only the parameters of 𝑓 (•) are updated through the loss values between the model answers and the ground truths. We adopted the commonly used cross-entropy loss function to train 𝑓 (•).EQUATIONwhere Θ is the set of the learnable parameters of 𝑓 (•). S denotes the set of training instances for serendipity. |S| denotes the size of training instances. ŷ𝑢,𝑖 is the predicted probability on "Yes" or "No". 𝑦 𝑢,𝑖 is the ground truth value.Style 2: Indirect. In this Style, we did not directly learn 𝑓 (•) for the prompt tokens for the question of serendipity. Instead, we first pre-trained 𝑓 (•) with two pre-training tasks, and then fine-tuned it for the question on serendipity. The pre-train and fine-tune process is expected to learn better token representations of LLM prompts for serendipity recommendations. The two pretraining tasks are: learning a prompt representation to ask whether a candidate item is unexpected to a user; and learning a prompt representation to ask whether a candidate item is relevant to a user. Specifically, the 𝐼 and 𝑄 for the first pre-training task are:𝐼 : "Unexpectedness means something that surprises somebody because the person is not expecting it. In recommendation tasks, being unexpected means the items are unlikely to be recommended to a user given this user's preferences. Given a user's history, please answer "Yes" or "No" on whether the candidate item is unexpected to the user. "𝑄: "Given a user with a history 𝐻 𝑢 , could you answer whether the candidate item 𝑖 is unexpected to this user 𝑢?"Similar to Continuous Style 1, during the pre-training process for this first pre-training task, the LLM is frozen, and only the parameters of 𝑓 (•) are updated through the loss values between the model answers and the ground truths on unexpectedness. We adopted the commonly used cross-entropy loss function similar to Continuous Style 1. The 𝐼 and 𝑄 for the second pre-training task are similar to the first pre-training task. The only change is from about unexpectedness to about relevance.To pre-train 𝑓 (•) with the two different pre-training tasks, a commonly used approach is sequentially training it with the two tasks. It may cause 𝑓 (•) to "remember" only the second task and "forget" the first task [11, 24] . Therefore, we constructed a mixed two-task dataset to pre-train 𝑓 (•) simultaneously. The mixed dataset sampled the training instances for the two pre-training tasks respectively and then mixed them in one dataset. That way, the pre-training process is able to strike a balance between the two tasks. Considering that two pre-training tasks may not contribute equally to the final vector representation, a sampling ratio 𝑒 𝑙 was experimented to control the proportion of the two tasks in the mixed two-task dataset.After being pre-trained, 𝑓 (•) is expected to obtain some "prior knowledge" on serendipity. We further fine-tuned 𝑓 (•) using the direct training instances on serendipity. The process is the same with the training task in Continuous Style 1.Although continuous templates are more flexible compared to discrete ones, the quality of the generated tokens highly relies on the selection of 𝑓 (•) and the training instances. More importantly, these tokens are randomly initialized, which may introduce some uncertainties and noises [29] . Therefore we further propose a hybrid prompt, which is a mixture of natural words and virtual tokens. We expect this type of prompt is able to provide stable input to the LLMs while controlling the level of uncertainties and noises.In a hybrid prompt with both natural words and virtual tokens, we need to decide on what part(s) of the prompt should use natural words and what part(s) should use virtual tokens. One principle we used is that if a part is information-rich, and natural words for it may not be sufficiently expressive, we will use virtual tokens. If a part needs to provide precise knowledge to the LLM without introducing any uncertainty, we will go with natural words. Therefore, we selected the 𝐻 𝑢 and 𝑖 elements in the input question 𝑄 to be the virtual tokens, because they (a series of names) may not contain the rich information needed by the LLM to recommend serendipity. In contrast, the task description 𝐼 and the remaining part of 𝑄 used natural words, since they reduce uncertainty and provide useful background knowledge. For the virtual tokens parts, we learned a mapping function 𝑚(•) to project the natural words of 𝐻 𝑢 and 𝑖 to two sets of virtual tokens[𝑃 𝐻 𝑢 1 ]...[𝑃 𝐻 𝑢 𝑛 3 ] and [𝑃 𝑖 1 ]...[𝑃 𝑖 𝑛 4 ]: 𝑚(𝐻 𝑢 , 𝑖) → [𝑃 𝐻 𝑢 1 ]...[𝑃 𝐻 𝑢 𝑛 3 ]; [𝑃 𝑖 1 ]...[𝑃 𝑖 𝑛 4 ](5) Similar to the continuous prompts, we do not have the demonstration set 𝐷 in the prompts since the training instances for 𝑚(•) play the role of providing additional demonstrations. Similar to the discrete and continuous prompts, we also designed two styles for the hybrid prompt templates.Direct. This style trains 𝑚(•) to generate token representations for a direct question of serendipity. As in Table 3 for this Style, the task description 𝐼 is the same with Discrete Style 1. The input question 𝑄 is also the same with Discrete Style 1 except that we replaced 𝐻 𝑢 and 𝑖 with virtual tokens.To learn the optimal virtual tokens, we froze the LLM's parameters and only optimized 𝑚(•) by calculating the loss values between the model answers and the ground truths on serendipity. We used the cross-entropy loss function.Style as in Table 3 , we do not directly learn the tokens for the direct question of serendipity. Instead, we first pre-trained the tokens with those two pre-training tasks, and then fine-tuned them for the question of serendipity. 
experiments 4.1 construction of ground truths: 
 In this paper, for continuous and hybrid prompts, we need the ground truth data to train, pre-train, or fine-tune the Transformer encoders added to the LLMs. For discrete prompts, we need the ground truth data to provide various positive and negative demonstrations. Specifically, we need three types of ground truth: serendipity, unexpectedness, and relevance.For serendipity, we used SerenLens [4] , an existing large-scale ground truth dataset on serendipity books, as the base dataset. We further converted this base into an instruction format that is compatible with the LLM prompt and output formats, in order to serve as the training instances for the Transformer encoders as well as demonstrations in discrete prompts. In total, we obtained 5,114 training instances.For unexpectedness, we did not have any existing base dataset to convert. Therefore, we propose a computational approach to "calculate" the ground truth of unexpectedness, avoiding the tedious human labeling process. In Psychology, unexpectedness is defined as violation of expectation [19] . We propose a computational operationalization of this definition. Specifically, we first calculated the conditional likelihood of seeing an item given a user's history of interacted items. We then used a low level of such conditional likelihood as a high level of unexpectedness. Therefore, the level of unexpectedness of an item 𝑖 to a user with a history 𝐻 𝑢 is calculated as:EQUATIONwhere the negative sign is to indicate the opposite relationship between the cognitional likelihood and the level of unexpectedness. The logarithm function is to smooth the larger values. Using the Law of Total Probability, the conditional probability in Equation 6could be rewritten as:𝑢𝑛𝑒𝑥𝑝 (𝑢,𝑖 ) = -𝑙𝑜𝑔 𝑝 (𝑖 |𝐻 𝑢 ) = -𝑙𝑜𝑔 ∑︁ 𝑖 𝑢 ∈𝐻 𝑢 𝑝 (𝑖 |𝑖 𝑢 )𝑝 (𝑖 𝑢 |𝐻 𝑢 ) (7)where 𝑖 𝑢 is a user's historically interacted item in 𝐻 𝑢 , 𝑝 (𝑖 𝑢 |𝐻 𝑢 ) is the occurring probability of 𝑖 𝑢 in 𝐻 𝑢 , and 𝑝 (𝑖 |𝑖 𝑢 ) could be calculated as:𝑝 (𝑖 |𝑖 𝑢 ) = 𝑛 (𝑖, 𝑖 𝑢 ) 𝑖 ∈𝐼 𝑛 (𝑖, 𝑖 𝑢 ) (8)where the numerator 𝑛(𝑖, 𝑖 𝑢 ) is the co-occurrence count for an item 𝑖 and 𝑖 𝑢 in all users' histories. The denominator is the sum of such co-occurrence count over each item 𝑖 in the item set 𝐼 . All of the components on the right side of this Equation 7 could be calculated from a dataset. After calculating all the items' 𝑢𝑛𝑒𝑥𝑝 (𝑢,𝑖 ) values for the user 𝑢 with the history 𝐻 𝑢 , we selected the items with the top values as the user's unexpected items (positive cases) and the items with the bottom values as the expected items (negative cases). We applied this approach to the Amazon Review Data [17] to calculate the level of unexpectedness between a book and a user. In total, we obtained 46,920 user-book pairs (the positive and negative pairs combined) and used them as the base to reformat according to the LLM prompt and output requirements to serve as the training instances or demonstrations.For relevance, we used the Amazon Review Data [17] again and followed the common practice in the recommendation research community: the observed interaction between a user and a book establishes a relevance label for this user-book pair. Other userbook pairs with no observed interactions are deemed as irrelevance. We obtained more than 100 million cases (user-book pairs). We sampled them according to the sampling ratio ( 𝑒 𝑙 ) (mentioned in Section 3.2.2) with respect to the unexpectedness dataset. We then reformatted the sampled cases into training instances or demonstrations.We chose two open-source LLMs for the implementation of Seren-Prompt: Flan-T5 (the 11B version) and Llama 2 (the 13B version). They are two different representative open-source LLMs with strong performance on various tasks. Flan-T5 (the 11B version) is an enhanced version of T5 that has been fine-tuned in a series of tasks. It has comparable performance in many language tasks to much larger models, such as PaLM (the 62B version). Structurally, it has both Transformer encoders and decoders. On the other hand, Llama 2 (the 13B version) is an enhanced version of Llama 1 developed by Meta, with stronger performance. Structurally, it only has Transformer decoders. These two models are the two representative LLM structures.Since serendipity is relatively rare compared to non-serendipity in the ground truth datasets, we adopted a recall-based metric, Hit Ratio (HR). HR seren @k measures the proportion of times the serendipity item is retrieved in the top-k position (1 for yes and 0 otherwise). In order to take the rank information into consideration and assign higher weights on higher ranks, I propose another metric called Serendipity-Based Normalized Discounted Cumulative Gain (NDCG seren ) based on the well-known metric Normalized Discounted Cumulative Gain (NDCG). NDCG seren @k is calculated as:EQUATIONFor both HR seren @k and NDCG seren @k, a higher value indicates a better performance.To evaluate the performance of the series of SerenPrompt, we selected the following two groups of representative baseline recommendation models. The first group consists of one randomnessbased method and four well-known deep learning recommendation models for serendipity: RAND, DESR [14] , PURS [12] , SNPR [28] , and SerenEnhance [4] .The second group is two well-known deep learning recommendation models for relevance tasks: SASRec [9] and BERT4Rec [22] . Both of the two groups of models are state-of-the-art deep learning models published in top venues in recent years.For the continuous and hybrid prompts, 80% of the data in SerenLens was used for training 𝑓 (•) or 𝑚(•) and the rest 20% was for testing. Since discrete prompts do not have a training process, we directly used the testing set for evaluations. For all discrete prompts, we use 10 demonstrations with 5 positive ones and 5 negative ones. For Continuous Style 2 and Hybrid Style 2 prompts, we pre-trained the virtual tokens on the mixed two-task dataset sampled from the UnexpectedBooks and the RelevantBooks datasets. For all prompts, for each user in the test set, we held one serendipity item as the testing positive sample, and then paired it with 99 non-serendipity items that were randomly sampled from the dataset as the negative samples. We compared the LLM prompted by SerenPrompt with the baseline models using the metrics (HR seren @k and NDCG seren @k). For all models, we adopted 5-fold cross-validation to evaluate the performance. We trained our models using the Adam optimizer. We set the learning rate 0.001, the hidden dimension 128, the dropout rate 0.2, and the regularizer decay 0.001 for all the models. Other model-specific hyper-parameters either followed their original studies or were adjusted for the training performance in this study. We reported the results using the optimal hyper-parameter settings. In addition, for fair comparisons, we set the head number of the multihead attention 2 for the models involving Transformers: SASRec, BERT4Rec, SNPR, SerenEnhance, and LLMs with SerenPrompt. 
hyperparameter analysis: 
 The core hyperparameters of SerenPrompt are the numbers of the virtual tokens (i.e., 𝑛 1 , 𝑛 2 , 𝑛 3 , and 𝑛 4 ) and the sampling ratio ( 𝑒 𝑙 ) between the unexpectedness and relevance tasks in the mixed dataset in the pre-training stage for both the continuous and the hybrid prompts. We investigated the effects of changing these hyperparameters on the recommendation performance. Numbers of the virtual tokens. Following the study of [29] , we adopted a coarse hyperparameter training strategy, which makes 𝑛 1 = 𝑛 2 and 𝑛 3 = 𝑛 4 . We chose 𝑛 1 , 𝑛 2 , 𝑛 3 , and 𝑛 4 from the set of values {1,2,3,4,5} and explored the optimal settings with the best HR seren @10. As shown in Figure 2a , for the continuous prompts, as 𝑛 1 or 𝑛 2 increases, the performance of SerenPrompt increases first and decreases then. When 𝑛 1 = 𝑛 2 = 2, HR seren @10 reaches the highest values for both Flan-T5 and Llama 2. The results indicate that both too few and too many tokens will result in ineffective prompts. Too few tokens may lack sufficient task information, while too many tokens may suffer from noisy and ambiguous information. For the hybrid prompts, as shown in Figure 2b , when 𝑛 3 = 𝑛 4 = 1, HR seren @10 reaches the highest value for both Flan-T5 and Llama 2. That means only one token is sufficient to represent the user or the candidate item. We also observe that as the number of virtual tokens increases, the performance of the LLMs with SerenPrompt keeps decreasing. Compared with the continuous prompts, the hybrid ones require fewer virtual tokens. Therefore, in the following subsections, we will only report the results of SerenPrompt with 𝑛 1 = 𝑛 2 = 2 and 𝑛 3 = 𝑛 4 = 1 where applicable. Sampling ratio of the two-task dataset. Using the optimal numbers of virtual tokens, we further investigated the effects of different 𝑒 𝑙 . As illustrated in Figure 3 , Flan-T5 and Llama 2 have similar HR seren @1 HR seren @5 HR seren @10 NDCG seren @5 NDCG seren @10 The results indicate that a two-task dataset with an equal amount of training instances for the unexpectedness and relevance tasks is most effective to provide balanced "prior knowledge" on serendipity. In the following subsections, we will only report the results using 𝑒 𝑙 = 50 50 where applicable.From Table 4 , we know that the hybrid template with the pretraining and fine-tuning process (Hybrid Style 2: Indirect) achieves the best performance among all the baseline models and the other types of SerenPrompt. In general, the LLMs prompted by the continuous templates and hybrid templates obtain a better performance than discrete templates. It suggests that compared to natural words, virtual tokens are more powerful in expressing a prompt for serendipity recommendations. In addition, no matter which type of SerenPrompt, Style 2's performance is better than Style 1. The results prove the effectiveness of the decomposition of serendipity. Breaking down the direct question on serendipity into two sub-questions on unexpectedness and relevance is more helpful for the LLM to recommend serendipity.It is interesting to note that prompting LLMs via discrete templates, which do not involve any model training, obtain a performance close to the other serendipity-oriented deep recommendation models (e.g., DESR, PURS, and SNPR). The results demonstrate the power of LLMs. They only need a task description and a few demonstrations to be on a par with the state-of-the-art baseline models. They are able to get around of the need of massive ground truth data. The potential offers many avenues for not only serendipity recommendations, but all kinds of recommendations in general.Since the LLMs contain billions of parameters, the efficiency of prompting LLMs via SerenPrompt is another critical evaluation for serendipity recommendations. To compare the efficiency among different prompt templates, we calculated the average inference time of the LLM on 1 instance. We tested all the templates on the testing set of SerenLens dataset with 8 NVIDIA Tesla V100S GPUs. We kept all the hyperparameter settings the same for each template. The results are shown in Table 5 . We observe that the continuous templates for both Flan-T5 and Llama 2 have the least inference time while the discrete templates have the most. The results indicate that templates with more virtual tokens and fewer natural words improve the LLM's efficiency. In addition, the hybrid templates obtain the best recommendation performance on HR seren @10 with relatively lower sacrifice to inference time, achieving a good compromise between the two conflicting goals: performance and efficiency.To have an intuitive understanding of the model results, we selected an example user to showcase the recommendation results using different types and styles of prompts. As shown in Figure 4 , there is a user interested in the books with topics of women, urban life, and romance according to her or his history. We also find that this user had a serendipity experience on finding a book titled The Awakening: A Vampire Huntress Legend as in his or her written review:"I haven't picked up a good vampire novel in quite a while...I really didn't have high expectations for this novel or any others in this genre. What a surprise to discover that a Philadelphia Sistah has written a bona fide, nail-biting vampire novel that is equal if not better than Anne Rice, et al."This book is a fantasy book with the romance and horror elements in it. It is not the usual type of this user. As shown in Figure 4 , only the hybrid prompts with both styles and the Continuous Style 2 prompts were able to recommend this book in their top-5 recommendation list (as highlighted in the red box). The discrete prompts with both styles tend to recommend books more closely following this user's history, such as romance and health books. The continuous and the hybrid prompts are bolder and more deviating from the user's usual type. 
conclusions: 
 In this paper, we investigated the potential of prompting LLMs to obtain serendipity recommendations. We designed three types of prompts: discrete, continuous, and hybrid, to investigate the expressiveness or effectiveness of natural words and virtual tokens used in prompts for serendipity recommendations. Meanwhile, for each type, we also designed two styles: direct and indirect, to investigate whether it is feasible to ask a direct question on serendipity or it is better to breakdown the direct question into two sub-questions. Extensive experiments have shown that the combination of the hybrid type and the indirect style achieves the best performance with relatively low sacrifice to computational efficiency, and outperforms all of the state-of-the-art baseline models. 
 
 Appendix:  


Decision: The paper compares different ways to obtain serendipitous recommendations, via different forms of prompting, and via vector similarity. 

  Relevance: 

* The paper is very relevant for conference topics. 



Soundness: 

* The paper is a good idea. A bit straight-forward, but yes, this is how I would do it too. 

* The empirical evaluation is thorough. It only uses one data set SerenLens on book recommendation, but I assume that serendipity datasets are hard to come by. 

  

Innovation: 

* I am not sure what to take from this paper other than, "yup, it works". But maybe this is exactly what we want in a paper. 


Several reviewers pointed out smaller issues that can be easily addressed. 

Overall, I think the paper is worthwhile accepting.

Reviews: review: main_review: main_review: Strengths: 

1. The paper introduces an approach to leveraging Large Language Models for generating serendipity recommendations, a relatively new method in recommender systems. 

2. The introduction of SerenPrompt, with variations in discrete, continuous, and hybrid forms, adds significant depth and robustness to the study. 

3. Extensive experiments were conducted that not only demonstrate the effectiveness of SerenPrompts but also compare them against state-of-the-art models, achieving excellent performance. 

4. The paper provides clear definitions and decomposes the concept of serendipity. 


Weaknesses: 

1. The heavy use of technical jargon could be reduced to enhance readability. 

2. While the hybrid approach shows promising results, the computational costs and practicality of deploying such LLM-based systems in real-world scenarios should be thoroughly discussed. 

3. The focus is primarily on serendipity recommendations without discussing how these findings might generalize to other types of recommendation systems or domains. 

4. There is a lack of discussion on potential ethical issues and biases that might arise from serendipity recommendations, especially given the subjective nature of what might be considered serendipitous.

Three questions for the authors: 

First, If you change the large language model that you use, will the experimental results change completely? 

Second, all prompt methods so far are somewhat randomized since changing just a few words may give different generated results. How are the authors going to address this issue in this paper? 

Third, it looks like that these tailored prompt methods will inevitably produce better results. Is it because of the power of LLM itself? Can the authors explain why?
main_review: main_review: * The experimental setting is limited to the domain of the book. Is this because large language models possess richer knowledge in the domain of the book? The generalization ability of the proposed method needs further validation. Would it also hold advantages in e-commerce scenarios?
* Would this method yield the same conclusions on smaller-scale models?
* The novelty of this paper requires further explanation. I'm curious about the necessity of incorporating prompt learning and large models, considering potential efficiency sacrifices.
* Would user-item interaction histories pose challenges with long-text inputs? If so, how would these be addressed?
* In comparison with the primary baseline model in terms of efficiency, I'd like to know whether the efficiency trade-off is acceptable alongside the improvements.
main_review: main_review: **Strengths:**
- SerenPrompt design in the paper is an innovative way to handle serendipity recommendations using LLMs. It effectively combines the advantages of discrete, continuous, and hybrid prompts, enabling precise tuning for better recommendations.
- The authors made a notable contribution by integrating the concept of serendipity directly into the prompting process, which is a novel approach within the field of LLM-driven recommendation systems. This method might encourage more personalized and diverse user experiences.
- The paper’s experimental setup is rigorous, using standard metrics for evaluation, which helps in validating the effectiveness of the proposed methods against other state-of-the-art models.

**Weaknesses:**
- This paper tests SerenPrompt on two large language models which contain over 10 billion parameters. If possible, I'm curious if the authors could test this on a slightly smaller model (<10B), like llama3-8b? Do smaller models have that kind of power? Insights into how model scalability affects performance would be valuable, especially for practical implementations.
- Some research on prompt tuning need to be cited and featured in related work [1-3]. There has been some work on hybrid prompt tuning, and the innovation of this paper is to introduce the approach to the subfield of serendipity recommendation, which needs to be spelled out in more detail. 
- Some comparative case studies would be helpful, illustrating the reasons for the failure of the baseline model and the proposed methodology for comparison would be good

[1] Gu, Yuxian, et al. "PPT: Pre-trained Prompt Tuning for Few-shot Learning." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[2] Yang, Kexin, et al. "Tailor: A soft-prompt-based approach to attribute-based controlled text generation." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

[3] Zhang, Rui, et al. "Knowledge-Augmented Frame Semantic Parsing with Hybrid Prompt-Tuning." ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
main_review: main_review: The overall writing is clear and easy to understand. The methods of discrete and continuous prompts originate from the NLP community but are adapted to the recommendation field. They propose a novel method to calculate the ground truth of unexpectedness w/o human efforts. They can achieve state-of-the-art results in serendipity recommendations with hybrid prompting LLMs, but the efficiency of their model may be doubted. Here are my problems:
- Do you think using LoRA or Adapter to fine-tune LLMs efficiently can achieve better results than continuous prompts? Again, we still need the FLOPs of the proposed methods.
- Since the pre-training stage of the Transformer Encoder requires a lot of computation. I'd like to know if the paper can prove this encoder can generalize to more evaluation tasks, not only SerenLens.
- Do you think you can augment the item information with more description, not just the candidate item name, to achieve better performance even with the discrete prompts? It may induce longer input, but if it works, we may not need to train continuous prompts?
- Do you think using the GPT-4 level model to perform this task is feasible only with discrete prompts? If not, why?