Generative Explore-Exploit: Training-free Optimization of Generative Recommender Systems using LLM Optimizers (2024)

Abstract: Recommender systems are widely used to suggest engaging content, and Large Language Models (LLMs) have given rise to generative recommenders. Such systems can directly generate items, including for open-set tasks like question suggestion. While the world knowledge of LLMs enable good recommendations, improving the generated content through user feedback is challenging as continuously finetuning LLMs is prohibitively expensive. We present a training-free approach for optimizing generative recommenders by connecting user feedback loops to LLM-based optimizers. We propose a generative explore-exploit method that can not only exploit generated items with known high engagement, but also actively explore and discover hidden population preferences to improve recommendation quality. We evaluate our approach on question generation in two domains (e-commerce and general knowledge), and model user feedback with Click Through Rate (CTR). Experiments show our LLM-based explore-exploit approach can iteratively improve recommendations, and consistently increase CTR. Ablation analysis shows that generative exploration is key to learning user preferences, avoiding the pitfalls of greedy exploit-only approaches. A human evaluation strongly supports our quantitative findings.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5396–5420 August 11-16, 2024 ©2024 Association for Computational Linguistics
Recommender systems are widely used to suggest engaging content, and Large Language Models (LLMs) have given rise to generative recommenders. Such systems can directly generate items, including for open-set tasks like question suggestion. While the world knowledge of LLMs enable good recommendations, improving the generated content through user feedback is challenging as continuously finetuning LLMs is prohibitively expensive. We present a training-free approach for optimizing generative recommenders by connecting user feedback loops to LLM-based optimizers. We propose a generative explore-exploit method that can not only exploit generated items with known high engagement, but also actively explore and discover hidden population preferences to improve recommendation quality. We evaluate our approach on question generation in two domains (e-commerce and general knowledge), and model user feedback with Click Through Rate (CTR). Experiments show our LLM-based explore-exploit approach can iteratively improve recommendations, and consistently increase CTR. Ablation analysis shows that generative exploration is key to learning user preferences, avoiding the pitfalls of greedy exploit-only approaches. A human evaluation strongly supports our quantitative findings.

heading: 1 Introduction
text: Recommender systems are widely used for various applications, including suggesting items in catalogs (music, books, videos) (Zhang et al., 2023a), e-commerce (Haramaty et al., 2023), and search results (Najork, 2023; Metzler et al., 2021). A core part of a recommender system is improving the relevance of recommended items based on the user’s preferences. Such preferences can be either explicit (e.g. user provided preferences about a brand in
* Work done during an internship at Amazon.
e-commerce queries), or implicit based on engagement or Click Through Rate (CTR) of provided recommendations. Furthermore, the more users interact with a recommender system, the more the recommended items are optimized to match such explicit or implicit user preferences.
Traditional recommender systems use optimization approaches such as (deep) collaborative filtering (Molaei et al., 2021), matrix factorization (Sarwar et al., 2002), and reinforcement learning (Afsar et al., 2023). A common thread is that they are all applied to a static set of items. However, the increasing capabilities of Large Language Models (LLMs) has led to the development of generative recommender systems (Zheng et al., 2023; Li et al., 2023a). Generative recommenders can be used to directly generate items or text content for recommendation. The immense world knowledge of LLMs makes them excellent at certain generative recommendation tasks such as question or query suggestion (Vedula et al., 2024). Unlike traditional recommender systems, they do not choose from a fixed item set, and the range of valid items that could be generated is vast.
5396
We investigate how LLM-based generative recommenders can be iteratively optimized based on implicit user feedback, e.g. by following CTR signals (cf. Figure 1). Such recommenders needs to support very large numbers of input contexts (e.g. items or queries). However, fine-tuning them to improve recommendations is prohibitively costly given their size. Our approach is training free: improving item recommendations does not need finetuning; instead we propose novel methods to synthesize context-specific implicit engagement signals (e.g. CTR) as part of the LLM input. We also propose a generative explore-exploit mechanism to generate and evaluate new candidate items. Our work is applicable to numerous applications such as search (Najork, 2023; Metzler et al., 2021), question answering (Huber et al., 2022; Qin et al., 2023), and query/question suggestion features in information seeking systems (Mitra et al., 2021).
To demonstrate our approach, we focus on the specific task of Question Generation (QG) (Pothirattanachaikul et al., 2020). This involves suggesting engaging (e.g. high CTR) questions to help users explore a topic. Experiments show that by leveraging both generative exploration and exploitation, our approach can adapt its recommendations to match the preferences of a population whose preferences are not directly observable. We show that a feedback loop based on CTR can successfully guide the LLM to explore new questions types and topics, and exploit previously generated ones. Finally, we show that our approach converges faster to questions that meet user needs, compared to baselines without CTR signals.
Our work makes the following contributions:
• To the best of our knowledge, we are the first to propose a training-free generative recommender approach, which optimizes its recommendations using a feedback loop based on implicit user feedback such as CTR.
• We create an offline experimental framework to simulate user preferences and their click behaviors to enable efficient development.
• A detailed evaluation on e-commerce and general knowledge domains demonstrates the effectiveness of our approach. We show that generative exploration and the use of prior CTR performance data are key elements of improving LLM-generated recommendations.

heading: 2 Related Work
text: LLMs for Recommendation. Recent research has been exploring generative recommender systems (Zheng et al., 2023; Wang et al., 2023a; Li et al., 2023a,b), by integrating LLMs into different stages (Lin et al., 2023) such as feature augmentation (Xi et al., 2023; Wu et al., 2023), representation enhancement (Li et al., 2023c; Rajput et al., 2023), item scoring (Zhang et al., 2023c; Tang et al., 2023), and user interaction (Dong et al., 2023; Yang et al., 2021). Most of these studies follow a retrieval-based paradigm, either scoring and recommending existing candidate items or generating a ranked item list grounded to an existing item pool. One shortcoming of such works is that they may fail to satisfy the needs of diverse or unseen users. Furthermore, they do not easily adapt to evolving domains or applications such as growing product catalogs or modified article contents. A solution to these issues is to generate the items for recommendation (Zheng et al., 2023; Wang et al., 2023a). We follow a similar approach, but differ in important aspects such as we do not require any training to generate content (i.e. questions) that are relevant to latent user needs.
Prior work has employed LLMs within recommender systems for CTR prediction, aiming to estimate the probability of users engaging with items (Fu et al., 2023; Li et al., 2023d). Similar to us, Liu et al. (2023) designed specific zero-shot and fewshot prompts to assess how LLMs predict recommendation ratings. To our knowledge, we are first to use LLMs to simultaneously generate recommendations and optimize CTR in the same framework. We do this by building upon the findings of Yang et al. (2023), by leveraging LLMs ability to understand and make use of the connection between textual input (questions in this work) and its performance on a target task. Our approach learns to generate questions that can achieve higher CTR.
Question Generation. QG is important in various NLP applications (Mulla and Gharpure, 2023) such as reading comprehension (Ghanem et al., 2022), conversational recommendation (Kostric et al., 2021). We specifically focus question suggestions, i.e., questions that users might want to ask a system. A notable use case is People Also Ask (PAA) questions in web search (Pothirattanachaikul et al., 2020), which suggests users potential next questions they can ask to further explore a topic.
Unlike PAA where questions are typically static, we generate novel and more engaging suggestions based on engagement signals from users interactions (clicks). This helps create questions aligned with the interests of the user population.
User Click Simulation. Several recommender systems studies have developed ranking models to simulate user click behavior in the web search domain (Dupret and Piwowarski, 2008; Zhang et al., 2023b). Recently, Wang et al. (2023b) prompted LLMs with personas and rules to simulate user behavior within conversational recommender systems. Inspired by their work, we design a user simulator that is representative of real-world user cohorts with varying personas and needs, and use LLMs to evaluate the question relevance to these personas.

heading: 3 Problem Definition
text: Given a population of users U = {u1, . . . , un}, a user u may search for multiple topics t1, . . . , tk (e.g. “Biology”, “Smartphones”). The task is to generate a fixed set of questions to be suggested for a topic t, referred as item pool (IP), such that the questions are likely to be of interest to as many users as possible in U that have searched for t. Actual user interests about t are hidden and can only be observed through interactions (e.g. clicks) with the generated items.
Using a sufficiently large LLM with reasonable instruction following capabilities, we want to generate IP for t, which is a list of N questions that maximize Click Through Rate (CTR) from all users u ∈ U that have interests in t.

heading: 4 Generative Recommender Approach
text: Our training-free generative recommender approach is outlined in Figure 2. At a high level, our proposed approach iteratively improves LLMgenerated outputs as follows. An LLM is used to generate the initial candidate items for a given task, and the click-through rates (CTR) of these items are then measured to gauge user engagement and preference. Based on these observed CTRs, a prompt is constructed to encapsulate this performance data. This prompt serves as a basis for applying a dual approach of generative exploration and exploitation using an LLM optimizer via in-context learning. The optimizer refines the generated items by bal-
ancing the exploration of new possibilities and the exploitation of known high-performing elements, to enhance the quality and engagement of the generated content. Our method is suitable for tasks where many valid generations are possible, like summarization and question generation.
Our approach is iterative, with user interaction data collected in each round i in order to refine the IPi on a given topic t. More specifically, in each iteration: (i) the n worst items are dropped from IP, (ii) n new items are generated and added to IP, and (iii) the performance of items in the updated IP is observed via an interaction feedback loop. Items in the i-th iteration are indicated by IPi.
Item Pool (IP) Initialization. In the first iteration, only the target topic of interest t is known. Hence, we initialize the item pool IP0, by simply generating relevant questions for t (prompts are shown Figure 9). These questions are recommended to users, and their interactions (clicks) are used to refine IPi in subsequent iterations to improve CTR.
Iterative Refinement using CTR. We iteratively refine IP based on the CTR signal; in each iteration, IP is updated by dropping low CTR items and adding new questions that are optimized to increase the overall CTR of IP. To improve the CTR we rely on the ability of LLMs to act as an optimizer for a target task by following instructions, without requiring any fine-tuning (Yang et al., 2023). We iteratively update the input instructions to the LLM by including previously generated items and their observed population CTRs. This allows LLMs to optimize their output based on the instructions, with the CTR values providing the LLM with both positive and negative instances of good as well as bad (low CTR) question. With sufficient iterations, the LLMs are able to converge to an IP that maximizes CTR. To do this, we propose two approaches: (i) FULL-CTR and (ii) EXPLORE-EXPLOIT.
FULL-CTR: Here, we provide the LLM all previously generated questions1 along with their observed CTR scores As part of the prompt, the LLM is instructed to optimize its generated questions such that the questions in IPi+1 will obtain high CTR. Similar to Yang et al. (2023), Figure 10 provides the prompt used to iteratively refine IP.
1We experimented with providing only the latest state of IP, however, this caused LLM to re-generate previously generated and dropped questions.
EXPLORE-EXPLOIT: The main shortcoming of FULL-CTR is that it can only exploit observed user preferences, and biases generation towards these. This greedy approach can prevent the optimizer from exploring different generations that may fail, thus limiting overall quality. We overcome this with our EXPLORE-EXPLOIT strategy, where in every iteration we drop the worst questions, and generate two sets of questions as follows. First, in an explore phase, a set of n questions is generated by providing only IPi, without any CTR values. Second, similar to FULL-CTR, a set of n questions is generated using an exploit prompt, designed to generate questions that are on the same topic as the best performing question in IPi (See Figure 10 for the prompt). To avoid saturating IPi+1 with only questions on one topic, EXPLORE-EXPLOIT instructs the LLM to explore new topics for which to generate new questions that increase the diversity of the entire IP set.

heading: 5 User Click Simulator
text: Developing and evaluating our approach requires user engagement data, which is expensive to collect, and involves complex privacy concerns. To enable fast offline experimentation, we simulate feedback by modeling user preferences and their click behavior on suggested questions using LLMs.
We represent users with different interests and goals via specific prompts (see Table 4 for details). These user personas p have specific interests for different topics. They do not represent a single user, but rather a population of users with similar char-
acteristics. We split the simulator into two steps: (i) relevance scoring, and (ii) action simulation.

heading: 5.1 Relevance Scoring
text: For a question qi generated for a topic t, we evaluate its relevance to a persona pj by prompting an LLM to score qi on a scale r ∈ {1, . . . , 10}:
ri,j = QS(qi, pj , t)
The prompt used in QS is given in Figure 8. When using the LLM to compute r, during decoding we set the temperature to 1 to induce variation in the behavior of a persona, thus, mimicking how different users falling under the same persona would behave. The score ri,j is computed once and independently from other questions in IP. Independent scoring reduces any potential bias stemming from other questions in IP.

heading: 5.2 Action Simulation
text: We obtain CTR values by simulating S user interactions based on the pre-computed rij values. In each interaction, we uniformly sample a persona (pj) and uniformly sample a set of K questions from IP. pj then takes one of K + 1 actions: (i) clicking one of the K questions, or (ii) not clicking anything. For qi ∈ K and pj we model the probability of qi being clicked as a temperature T softmax:
P (CLICK|pj , qi) = erij/T
eRS/T + ∑
qk∈K erkj/T
(1)
where RS represents a fixed “rejection score”, which is the logit for clicking on none of the options. Figure 3 shows the resulting CTRs for varying relevance scores and values of RS. Low RS values (e.g. RS < 10) allows for unrealistic high CTR (e.g., +90%) and high RS values (e.g. RS > 12) suppress the CTR heavily, not leaving much room for improvement. We set RS = 11 in our experiments, which yields potentially realistic CTR values while allowing for room to improve by increasing the relevance of suggested questions.

heading: 6 Experimental Setup
text: Here, we discuss the experimental setup, namely the domains on which we assess our proposed approach and competitors. Furthermore, we explain in detail the personas used for experimentation, which aim at mimicking real user cohorts. Finally, we will define the evaluation metrics used.

heading: 6.1 Data and Domains
text: Table 1 provides an overview of the details for the two domains we experiment with. The table lists sample topics of interest, alongside the personas we experiment with. The complete list of topics for both domains is shown in Appendix A.
E-Commerce: Suggested questions in online shopping have high utility as they allow users to explore the product space and make informed purchase decisions. The inputs are 50 random product categories from the Amazon Review Dataset (Ni et al., 2019), while the personas represent shoppers.
General Knowledge: The inputs are Wikipedia article titles, while the personas represent users interested in a particular aspect of the article. We randomly sample 50 featured Wikipedia articles ranging across different categories (see Appendix B for details).

heading: 6.2 User Personas
text: E-Commerce Personas: The personas are defined in terms of shopping preferences, a common way to describe customer behavior (Carmel et al., 2020; Haramaty et al., 2023). Inline with preferences proposed in literature, we use the following: Price, Quality, and Brand Reputation, Features and Functionality, and Ethical Considerations. We experiment with user populations that consist of only one persona type as well as populations that contain multiple personas. To extract CTR for populations that include multiple personas, we first compute question relevance, r (cf §5.1) for a persona and question pair, then consequentially for each user click simulation, we randomly select one of the personas.
General Knowledge Personas: Defining personas for this domain is more challenging. This is mainly due to Wikipedia being very diverse, and user preferences can vary greatly depending on the Wikipedia article or category.
Since personas, theoretically can be generated independently per article and category, hence, being highly sparse and potentially representing an unrealistic scenario to experiment with, we simplify the process and consider personas based on their focus as shown in Table 1 Such personas are general enough to be applicable to most Wikipedia articles, allowing us to gain representative insights.

heading: 6.3 Approach Setup
text: We set the size of IP to 5 questions, and the number of generated (and dropped) questions at every iteration to n = 1. We allow our approach to refine IP for 15 iterations (I = 15). The number of click simulations in each iteration is set to S = 5000; a persona is shown K = 3 questions at each simulation. We empirically set the softmax temperature to T = 1.5 and the rejection score to RS = 11 (cf. §5). Additional details about the simulator setup are provided in Appendix C.
LLM: For all experiments we use GPT-4 (OpenAI, 2023), considered the most capable LLM (at the time of writing).2 For simplicity, the same LLM is used for question generation and the user simulator. All prompts are provided in Appendix D.

heading: 6.4 Approach Configurations
text: To evaluate the effectiveness of our proposed approach and its components, we compare against ablations that assess the impact of each component.
RANDOM-CTR: Instead of using the CTR signal, it uses random CTR values (between [0%− 15%]) for dropping the worst question and writing a new one. This ablation tests the impact of CTR signals.
NO-DROP: expands the IP up to N iterations for which it is executed, without dropping any questions. The CTR value of the NO-DROP at iteration i reflects the performance of directly generating 20 questions (initially |IP |=5, and maximum number of iterations is 15). This ablation highlights the usefulness of the iterative nature of our approach, where questions with lowest CTR are dropped.
PARTIAL-CTR: uses CTR signal to drop the worst performing questions from IP at every iteration, however, in the input instruction prompt we do not provide the CTR that the previously generated questions obtained. In this way we can test the impact of CTR signal in the instruction prompt and its outcome in terms of obtained CTR by IP.

heading: 6.5 Evaluation Metrics
text: We define metrics for two evaluations: (i) item relevance scoring, (ii) recommendation performance.
2The gpt-4-1106-preview model was used.
Item Relevance Scoring: To measure AGREEMENT, we compute the agreement between human annotators when judging which question in a pair is more relevant for a persona. We also compute LLM ACC., which measures the accuracy or alignment of LLMs with the human judgment.3 Namely, we measure if the LLM assigns a higher score to the question from the pair that was judged by annotators as being more relevant.
Recommendation Performance: To assess the overall performance we compute the following metrics: (i) CTR values across iterations, (ii) average CTR score across N iterations, and (iii) human annotation, by comparing questions in IP0 vs. IPI .

heading: 7 Results
text: We present the experimental results for the two components of our approach: (i) item relevance scoring, and (ii) recommendation performance.

heading: 7.1 Relevance Scoring Results
text: Table 2 shows the human evaluation results for the question relevance scoring (cf. §5.1). This assesses if LLMs can reliably be used to judge question relevance. Due to the more complex nature of the e-commerce domain, we only evaluate relevance scoring on this domain. We sample three different personas from three different product categories, and randomly pair questions across different iterations. This results in 188 pairs for evaluation. Each pair is judged by two expert annotators.4
3Here we consider only cases where the human annotators agree on the more relevant question for a persona.
4We recruited expert internal annotators who were trained
In 70.2% of cases human annotators agree on which question for a given pair is more relevant. In the subset of question pairs with human agreement, the LLM ACC. score is 77.3%. This means that LLMs correctly assign higher relevance scores to questions that are also judged by annotators as being more relevant for a persona. Note here that we only show LLM ACC. for the cases where there is human agreement, since we assume that there is no ambiguity and thus can objectively assess LLM’s alignment with human annotators. Across different product categories, AGREEMENT relatively consistent and varies between 67.9% and 73.6%, while LLM ACC. changes more drastically between 58.3% (TV Antennas) and 90.6% (Cookware Sets).
In sum, these results demonstrate the ability of LLMs in accurately predicting question relevance for personas. This allows us to reliably simulate CTR signals using our proposed user simulator.

heading: 7.2 Recommendation Quality Results
text: In the following we present the evaluation of recommendation quality for our approach and ablations.

heading: 7.2.1 E-commerce Question Suggestion
text: Figure 4 shows the averaged results obtained across different personas (with varying preference counts) for all approaches.
Exploration is key for improving generated recommendations. Overall, EXPLORE-EXPLOIT achieves a statistically significantly higher CTR when compared against all other competing approaches.5 This shows EXPLORE-EXPLOIT utilizes LLMs to both explore new questions, which ensures that they are relevant for multiple personas, and at the same time, whenever CTR improvement is observed through our user simulations, it can pivot and exploit such question shapes and aspects to generate questions that are likely to be relevant for the underlying population base. The results clearly demonstrate the importance of not only exploiting identified high-CTR items, but also actively exploring to discover the hidden preferences of the target population, which the EXPLOREEXPLOIT is method is able to do.
on the provided evaluation protocol on how to determine when a question is better and more suited for a persona.
5p-value < .001, as measured by the Z-test for proportions.
Question Relevance scores improve iteratively. EXPLORE-EXPLOIT demonstrates a nearly constant increase in scores. If we compare IP0 against IP15, we see an improvement of more than ∆ = +2 points. We also observe that FULL-CTR performs better than PARTIAL-CTR, however the gap is relatively small. This indicates that the LLM is able to make use of the CTR signal provided via in-context learning (e.g. question and corresponding CTR score), without having an explicit strategy to explore new questions and exploit the best performing ones, however, its effectiveness is limited. We also more explicitly investigated the LLM’s optimization capability through the CTR signal using a synthetic setup, where we scored questions based on their length, allowing us to have a clear and deterministic signal. This further validated our conclusions. For a more detailed analysis of the results for this setup we refer the reader to Appendix E.
Using observed CTR is critical for Exploitation. The large gap between PARTIAL-CTR and the NODROP and RANDOM-CTR baselines shows the importance of using the CTR signal to drop the worst performing questions from the question pool. We also note that RANDOM-CTR and NO-DROP baselines perform equally poor and fail to improve the question relevance scores across iterations. This is expected since they do not make use of the CTR signal to generate more engaging questions or filter out the worst performing ones.
CTR consistently improves in all settings. Since CTR is simulated by relying on question relevance score, here too, the best performing approach is EXPLORE-EXPLOIT. From IP0 to IP15 we notice an improvement of more than ∆ CTR = +11% percentage points for populations with single personas, and more than ∆ CTR = +7% for populations with 3 personas. A similar trend, as for question relevance, is observed between FULL-CTR and PARTIAL-CTR, where the gap in their CTR scores is at most 3%. When we compare EXPLOREEXPLOIT against FULL-CTR and PARTIAL-CTR, we observe that the differences in the question relevance scores are further amplified for the CTR where EXPLORE-EXPLOIT obtains significantly higher scores. Moreover, CTR results from NODROP and RANDOM-CTR highlight the importance of the feedback loop, allowing LLMs to iteratively refine the questions in IP, thus, increasing their relevance and thereby their CTR.
Example Recommendations: To show how EXPLORE-EXPLOIT leverages CTR signals, we provide examples from the first and last iterations of the model in Appendix G. We observe that our EXPLORE-EXPLOIT approach can effectively discover the hidden preferences of the user population. For example, if the simulated users for the query “spray bottles” have a hidden preference for Ethical Considerations, our approach converges to generate questions such as “Are there ecofriendly biodegradable options for spray bottles?” without any direct knowledge of the user preferences. Similarly, if the user population includes a preference for Quality, optimizing questions for the query “cookware sets” with our approach results in highly relevant questions such as “Are copper cookware sets prone to tarnishing over time?”.

heading: 7.2.2 General Knowledge Domain
text: Figure 5 shows the results for the general knowledge domain (see Figure 13 for the results for all personas). Here too, as in the e-commerce domain, approaches that make use of CTR are able to iteratively improve question relevance and CTR, while the RANDOM-CTR and NO-DROP baselines fail to improve. These results demonstrate that our approach is applicable across domains and tasks.

heading: 7.3 Persona-level Results
text: Evaluations for all personas are listed in Appendix F. Results show that our approach can discover the preferences of a diverse set of personas.

heading: 7.4 Human Evaluation of Recommendations
text: For the best performing EXPLORE-EXPLOIT approach, we carried out a human evaluation to understand the preferences of human annotators for the item pool, IP. Annotators, without knowing the source iteration of the questions, performed a pairwise preference annotation of IP0 against IP15 for the e-commerce domain.
Out of 25 pairwise comparisons, in 88% (22 cases), annotators judged IP15 as their preferred question set for a given persona and topic. This result further validates the improvements we see in terms of question relevance scores as well as CTR.

heading: 8 Considerations for Online Deployment
text: In this work we used a simulator to facilitate offline development. This approach allowed us to develop and test various algorithms without needing to directly involve real users or incur the associated costs and risks. While having the simulator as a stand-in for real user feedback served to significantly expedite the development process, transitioning to real-world deployment presents a set of new challenges and requires modifications.
The core requirement is to replace the simulatorgenerated feedback with actual user engagement. This means the recommendations generated by the system must be served to real users, and their interactions (such as CTR and dwell time) need to be collected and stored for each input context. As real engagement is noisy and sparse, implementing efficient data pipelines is key. Caching might be
needed to store and serve the item pool for each context in order to minimize latency. Furthermore, the optimization iterations may need to run at fixed intervals, e.g. after every N user engagements or based on a predetermined time schedule. This periodic and continuous optimization ensures that the model evolves in response to the latest user data.
Finally, moving from the cohort-based personalization explored here (where recommendations rely on generalized user segments) to user-level personalization will require further changes. Tailoring recommendations to individuals will require collecting more granular data.

heading: 9 Conclusion
text: We proposed a novel method to improve LLMbased generative recommender systems by iteratively refining recommendations based on implicit feedback loops from CTR signals. We additionally defined a user simulator to effectively simulate user interactions with such recommended items.
Our novel Generative EXPLORE-EXPLOIT approach does not require any fine-tuning, and only relies on an LLM optimizer using in-context learning by synthesizing observed CTR performance data and incorporating them into the prompt. Experiments with our approach show that while leveraging historical CTR data is crucial to exploit known engagement patterns, the inclusion of a generative explore phase is equally important for discovering user preferences. Evaluations on the task of question generation across two domains (e-commerce and world knowledge) show that our proposed generative recommender approach is able to generate questions that are highly relevant to its user population in just a few iterations, which in turn results in higher engagement as measured through CTR.
The generative EXPLORE-EXPLOIT method is is particularly suitable for tasks where there are many potentially valid suggestions that can be generated. While we studied question generation in this paper, it can applied to a range of tasks such as summarization (Fetahu et al., 2023a), personalized headline generations (Cai et al., 2023), and followup question suggestion (Fetahu et al., 2023b). By avoiding the use of reward models or fine-tuning, our approach can effectively scale to scenarios with billions of items, while also being able to support user-level personalization.
Limitations
There are some limitations in the current version of this work that we would like to highlight.
User-level Personalization. In this work we did not model individual users, but instead modeled cohorts of users. Our current approach can be adapted to address this gap by extending the feedback loop to be user specific. The advantages of our trainingfree approach are even more relevant in such a setting, allowing us to efficiently scale personalized recommendations to millions of users. We leave this line of inquiry for future work.
Budget Constraints. Due to limitations on our budget we could not run all possible experiments. For example, our work presents only the question generation task, while, in principle, the proposed framework can be applied to different recommendation tasks. Similarly, we couldn’t host open source LLMs of quality close enough to ChatGPT (i.e., with size >70B): in preliminary experiments we noticed that smaller models are not great in following our instructions and to reason over the numerical CTR signal.
Offline Experimental Framework. As we don’t have a real system to rely on, it is impossible to run real user studies. For this reason, we report only experiments with simulated users/persona. We tried our best in assessing the quality of the simulator and we believe the results are acceptable.
Baselines. We did not adapt existing optimization approaches, e.g., multi-armed bandits, to our task. While this is in principle possible by generating a very large pool of questions, this will remain a static set; in our approach the questions pool is dynamic so we believe a comparison would not be fully fair. We did not consider the possibility of a hybrid approach, which we leave as future work.

heading: Acknowledgements
text: We would like to thank Eugene Agichtein, Oleg Rokhlenko, and Saar Kuzi for their feedback.

heading: A Details of Dataset Topics
text: Table 3 lists the 50 product categories and the 50 Wikipedia article titles used as input topics in this work.

heading: B Wikipedia Data
text: For the Wikipedia domain, we obtain a diverse set of 72 Wikipedia article titles (These generated titles were verified to be actual Wikipedia pages) by prompting GPT-4 to write 6 diverse and interesting Wikipedia article titles for each of the 12 Wikipedia categories6 and then randomly sample 50 articles.

heading: C User Simulator Details
text: C.1 Effect of Softmax Temperature
We wanted to ensure that the user simulator converted the LLM generated relevance scores into meaningfully different simulated user behavior. However, with a temperature setting of 1 in Equation 1, the action distribution is always extremely peaked, leading to only extremely relevant questions receiving meaningful click probabilities. Figure 6 shows the effect of T K = 3, RS = 11, for question scores ranging from 1 to 10, and makes it clear that a higher temperature is to produce diverse action distributions. Based on this, we set the temperature for our experiments to T = 1.5, leading to more sensitivity to changes in question relevance score, even when the scores are lower.
6https://en.wikipedia.org/wiki/Wikipedia:Contents/Categories, general reference category is excluded
C.2 Click Simulation Analysis
In real user studies, one only obtains a noisy estimate of user interest through CTR, which is why we draw samples from an action distribution rather than directly reporting the “true” CTR of the user population. Here, we investigate the effect of the number of simulated user interactions (S) in each iteration on the reliability of the resulting CTR values by varying S between 100 and 50,000. Figure 7 shows the variance in the resulting CTR for the tested S values. As the number of simulations increase, amount of variation in the obtained CTR values goes down and approaches to the theoretical values. In this study, we use 5,000 simulations which is both realistic and adds a manageable noise to the CTR calculation process.

heading: D Prompts
text: There are two sets of prompts required for our experiments: those which define the simulated users, and those which are used to guide an LLM to optimize questions for the simulated population. Table 4 and Figure 8 show the prompts used for the simulator. The persona descriptions from the former are substituted into the prompts from the latter in order to obtain the relevance scores for questions.
For generating and refining the question pool, we use two prompts, again specified per-domain. Figure 9 shows the prompts used to initialize the question pool for each domain, and Figure 10 shows the prompts used for the CTR and EXPLORE-EXPLOIT methods to improve on the question pool based on the measured CTRs.
E Verification of LLM Optimization using Length-based Scoring
Instead of scoring the questions with LLMs based on their relevance to the personas as in §5.1, we also evaluate how different methods perform when each question is scored based on its length, measured in number of words and number of characters. Word and character based scoring of a question Q is calculated as following:
QS-W(q) = max(min((|q|w − 4)× 9
11 + 1, 10), 1)
QS-C(q) = max(min((|q|c − 20)× 9
55 + 1, 10), 1)
where |q|w and |q|c are the number of words and characters in q, respectively. These formulas map the word and character counts of a question to a score between 1 and 10 with a maximum length of 15 words and 75 characters and a minimum length of 4 words and 20 characters. This deterministic scoring ensures a clear and observable trend in the CTR values of the questions and allows us to test whether and LLM can learn this trend and optimize for it. This setup requires the LLMs to learn to ignore the semantics and focus on the length of the questions. Although one can use any type of input for this setup, for simplicity, we opt for using the same inputs (i.e., product categories) that we use in our main experiments for the shopping domain (Section 7.2.1).
Next, we investigate RANDOM-CTR, PARTIAL-CTR and CTR methods on the artificial setup with scoring based on question length. Figure 11 displays the average question scores and CTRs for the three methods across 40 iterations for word count and character count based scoring. We increased the number of iterations to 40 for this experiment to make sure that all methods converged. The trend is similar and consistent for both scoring methods. In early iterations the average question length goes up for all methods including the RANDOM-CTR baseline, which is suprising since the RANDOM-CTR baseline does not use the CTR information to drop shorter questions at each iteration. We argue that this increase in the average question length is due to the fact that the prompt that is used to generate the initial 5 questions is different from the prompt that is used to generate the new questions at every iteration and, although it is not stated in the prompt, the iterative prompt, for some unclear reason, leads to longer questions. The improvement for the PARTIAL-CTR is expected since it uses the CTR data to drop shorter questions at every iteration
while keeping the longer questions and the performance increases whenever a longer question is generated. We also observe from the CTR method that providing the CTR data to the LLM leads to a faster and greater increase in the average question length throughout the iterations. This indicates that the GPT-4 implicitly recognizes that the customers preferred longer questions over the shorter ones and it is more likely to generate longer questions at the next iterations than the PARTIAL-CTR method.

heading: F Persona-level Results
text: F.1 E-Commerce
Figure 12 shows the average question scores and CTRs of different methods in the e-commerce domain for populations with single personas. Consistent with the findings discussed in Section 7.2, EXPLOREEXPLOIT demonstrates the most significant relative improvement in both question scores (between +1 and +3) and CTRs (between +2.5% and +18%). For three out of the five personas (Quality, Features and Functionality and Ethical Considerations ), FULL-CTR method performs comparable to the EXPLOREEXPLOIT method, indicating that even without an explicit explore-exploit instruction, LLM is able to find a good balance between exploring new topics and exploiting the best performing questions. Note that the variance in achieved peak question scores and CTRs can be substantial across different personas. For all methods, achieved question scores and CTRs are substantially lower for the personas with Price and Brand Reputation preferences compared to the other three personas. This is most likely because Price and Brand Reputation are more specific preferences compared to the other three, hence often they are not deeply explored in the question generation phase. Overall, these results indicate that our proposed methods enhance the question generation process, consequently improving CTRs, for a diverse set of personas.
F.2 General Knowledge
Figure 13 illustrates the average question scores and CTRs in the general knowledge domain, focusing on five personas with a single preference. Similar to the observations in §F.1, EXPLORE-EXPLOIT demonstrates the most notable improvement in question scores (between +1 and +4) and CTRs (between +3 and +25) when comparing the last iteration to the initial one. Similar to the personas in the ECommerce domain, we observe significant variations in the ultimate question scores and CTRs across different personas. For instance, by iteration 10, the “event-focused” persona could attain approximately a 30% CTR, whereas the maximum CTR achieved for the “location-focused” persona after all iterations was only 6%. This is likely because for many of the Wikipedia articles (e.g., Tabata Training) location related questions are not typical and hence LLM does deeply not explore generating questions that are relevant to this persona. These results demonstrate that our proposed methods effectiveness could generalize to different domains.
F.3 Average CTR Values
Table 5 presents the average and last CTR values after 15 iterations for all of the tested methods and populations in this study. In terms of average CTRs, EXPLORE-EXPLOIT outperforms all other methods in 12 out of 13 cases, with FULL-CTR winning once. For “LOCATION-FOCUSED” persona, FULLCTR achieves 0.2% higher CTR than EXPLORE-EXPLOIT. Considering the CTRs at the last iteration, EXPLORE-EXPLOIT outperforms in 11 out of 13 cases, with FULL-CTR winning twice. Notably, with the “Event-Focused” persona, EXPLORE-EXPLOIT surpasses FULL-CTR by approximately 8.8% at the last iteration. Overall, the persona-level results are consistent with our observations in §F.1 and §F.2.
0 2 4 6 8 10 12 14 2.00
2.25
2.50
2.75
3.00
3.25
3.50
P ri
ce
Q S
NO-DROP RANDOM-CTR
PARTIAL-CTR FULL-CTR
EXLPLORE-EXPLOIT
0 2 4 6 8 10 12 14
1
2
3
4
5
C T
R (%
)
NO-DROP RANDOM-CTR
PARTIAL-CTR FULL-CTR
EXLPLORE-EXPLOIT
0 2 4 6 8 10 12 14
3.5
4.0
4.5
5.0
5.5
6.0
6.5
Q ua
lit y
Q S
0 2 4 6 8 10 12 14 5
10
15
20
C T
R (%
)
0 2 4 6 8 10 12 14
2.2
2.4
2.6
2.8
3.0
3.2
B ra
nd R
ep ut
at io
n
Q S
0 2 4 6 8 10 12 14
1.0
1.5
2.0
2.5
3.0
3.5
C T
R (%
)
0 2 4 6 8 10 12 14
4.5
5.0
5.5
6.0
6.5
7.0
F ea
tu re
s an
d F
un ct
io na
lit y
Q S
0 2 4 6 8 10 12 14
15
20
25
30
C T
R (%
)
0 2 4 6 8 10 12 14
Iteration
2
3
4
5
E th
ic al
C on
si de
ra ti
on s
Q S
0 2 4 6 8 10 12 14
Iteration
5
10
15
20
C T
R (%
)
Figure 12: Average question scores and CTR in the e-commerce domain for populations with single personas.

heading: G Generated Questions
text: In Tables 6 and 7, we show the questions that are from the 1st iteration and last iteration for the EXPLOREEXPLOIT method for E-commerce and Wikipedia domains, respectively. We could clearly see that after refinement, the generated questions are much more relevant to the Persona and Topic. For example, when Persona is “Quality” and Topic is “Cookware Sets”, irrelevant questions like “How many pieces are typically included in a cookware set?” are replaced with more relevant questions at the last iteration.

heading: Future Work
text: We leave this line of inquiry for future work. Budget Constraints. Due to limitations on our budget we could not run all possible experiments. For example, our work presents only the question generation task, while, in principle, the proposed framework can be applied to different recommendation tasks. Similarly, we couldn’t host open source LLMs of quality close enough to ChatGPT (i. e. , with size >70B): in preliminary experiments we noticed that smaller models are not great in following our instructions and to reason over the numerical CTR signal. Offline Experimental Framework. As we don’t have a real system to rely on, it is impossible to run real user studies. For this reason, we report only experiments with simulated users/persona. We tried our best in assessing the quality of the simulator and we believe the results are acceptable. Baselines. We did not adapt existing optimization approaches, e. g. , multi-armed bandits, to our task. While this is in principle possible by generating a very large pool of questions, this will remain a static set; in our approach the questions pool is dynamic so we believe a comparison would not be fully fair. We did not consider the possibility of a hybrid approach, which we leave as future work