CTRL-Rec: Controlling Recommender Systems With Natural Language

## Abstract

Abstract When users are dissatisfied with recommendations from a recommender system, they often lack fine-grained controls for changing them. Large language models (LLMs) offer a solution by allowing users to guide their recommendations through natural language requests (e.g., “I want to see respectful posts with a different perspective than mine”). We propose a method, CTRL-Rec , that allows for natural language control of traditional recommender systems in real-time with computational efficiency. Specifically, at training time, we use an LLM to simulate whether users would approve of items based on their language requests, and we train embedding models that approximate such simulated judgments. We then integrate these user-request-based predictions into the standard weighting of signals that traditional recommender systems optimize. At deployment time, we require only a single LLM embedding computation per user request, allowing for real-time control of recommendations. In experiments with the MovieLens dataset, our method consistently allows for fine-grained control across a diversity of requests. In a study with 19 Letterboxd users, we find that CTRL-Rec significantly enhances users’ sense of control and satisfaction with their recommendations, without reducing engagement.

## 1 Introduction

In this work, we propose CTRL-Rec, a method that integrates natural language controls into traditional recommender systems. CTRL-Rec allows for balancing between users’ explicitly stated preferences and their engagement signals, and importantly, is capable of directly influencing the retrieval stage of modern recommender systems, rather than being confined to post-retrieval ranking.

Motivation. Natural language could serve as an intuitive and flexible interface for controlling recommender systems (Friedman et al., 2023; Malki et al., 2025). Imagine that a user could simply say “I want content that helps me learn about other perspectives,” (or any of the other requests in [Figure 2](https://arxiv.org/html/2510.12742v1#S1.F2)) and have their recommendations update immediately in real-time. These controls could act as an important counter-balance to the engagement-focused(^1^11Modern recommender systems are primarily based on optimizing user engagement, e.g., likes, clicks, replies, etc. Platforms do also incorporate non-engagement signals like user controls and surveys to some extent (Cunningham et al., 2025), however, current integrations are typically limited by the sparsity of existing non-engagement signals.) nature of modern recommender systems, giving individuals greater agency to align the recommender system with reflective or aspirational preferences that aren’t captured by their engagement history (Ekstrand & Willemsen, 2016; Kleinberg et al., 2024a; Morewedge et al., 2023; Lazar et al., 2024; Rezk et al., 2024; Lukoff et al., 2021, 2023; Tan et al., 2025; Zhang et al., 2022; Feng et al., 2024). For instance, a user might want to see thoughtful long-form history videos, even if haven’t typically engaged with such content. More intentional curation using stated preferences could also allow users to counter divisive but undesired content amplified by engagement-optimizing algorithms (Milli et al., 2025; Rathje et al., 2024).

Large language models (LLMs) offer promising capabilities for fulfilling such requests (Malki et al., 2025; Kolluri et al., 2026; Jahanbakhsh et al., 2025). However, two key practical challenges remain in fully integrating natural language with conventional recommender systems. First, it is essential to be able to balance user engagement with stated preferences, since users will articulate only a small subset of their preferences, requiring the system to infer others from their behavior (Malki et al., 2025). Second, while LLMs could be used to directly re-rank a small set of items already present in a user’s feed (Kolluri et al., 2026; Jia et al., 2024; Piccardi et al., 2024; Jahanbakhsh et al., 2025), if the initial pool of items is too limited, then re-ranking alone may not sufficiently reflect the user’s expressed preferences. Therefore, it is important to enable natural language controls that can influence the retrieval of items from the outset—a task that is computationally challenging to directly apply LLMs to, given that the retrieval stage often involves billions of items.

CTRL-Rec. Our work tackles both challenges. First, to balance engagement with stated preferences, we overcome the “type mismatch” between conversational and traditional recommender systems. While traditional recommenders focus on predicting user-item interactions—such as whether a user will like a specific post—natural language requests are often broad and free-form, potentially applying to many items at once (e.g., “I want to see posts that are funny and witty but not mean”). To bridge this gap, we use LLMs to simulate users’ judgments of particular items based upon their natural language requests. These user-request-based predictions can then be integrated into the standard weighting of signals that traditional recommender systems optimize, allowing for a balance of stated preferences and engagement.

Second, building upon ideas from dense retrieval (Izacard & Grave, 2021; Karpukhin et al., 2020; Khattab & Zaharia, 2020), we enable real-time control by distilling LLM-generated judgments into a dual-encoder (two-tower) architecture for computational efficiency. This approach enables natural language controls to directly influence which items are retrieved, rather than being limited to the final ranking stage. While naively computing the simulated user-item judgments would require $m$ LLM queries for each new user request (where $m$ could be in the billions or trillions), CTRL-Rec instead requires only a single LLM embedding computation per user request.

Figure: Figure 1: Our approach differs from standard engagement recommenders by allowing to optimize for both revealed preferences (through engagement signals) and stated preferences (through natural language requests). This allows users to explicitly control their recommendations while maintaining the benefits of engagement-based recommendation.
Refer to caption: x1.png

In summary, our contributions are:

- 1.
A scalable method for integrating language-based controls into traditional recommender systems, allowing for real-time control of recommender systems ([Figure 1](https://arxiv.org/html/2510.12742v1#S1.F1)). Only one LLM embedding computation is required per user request.
- 2.
Empirical validation on MovieLens using both genre-specific and subjective, open-ended controls. We find that our approach effectively steers recommendations according to user requests while maintaining engagement quality.
- 3.
Human study with Letterboxd users. In a study with 19 Letterboxd users, we find that CTRL-Rec was positively received by users and significantly enhanced users’ sense of control and satisfaction with recommendations compared to traditional controls.

Figure: Figure 2: CTRL-Rec has the potential to be able to handle many kinds of user requests, unlocking many novel forms of user control.
Refer to caption: x2.png

## 2 Related Work

User Control of Recommender Systems. Traditional interfaces for allowing users to better align recommender systems with their stated preferences (Milli et al., 2021; Kleinberg et al., 2024b; Agarwal et al., 2024; Yang et al., 2019) include controls such as “See less often” buttons or domain-specific controls such as genre toggles. However, research has shown that these controls are rarely used (Cunningham et al., 2025), are often ineffective when they are used (Mozilla, 2023), and that users still frequently feel a lack of agency or control over their recommender systems (Lukoff et al., 2021, 2023). Recent work by Kolluri et al. (2026) and Jahanbakhsh et al. (2025) aims to broaden user control by introducing value-based toggles, using LLMs to rerank social media content according to selected values (e.g., “cheerful,” “knowledge”, “tradition”). In contrast, our approach enables users to specify their own free-form values and applies these preferences directly to the retrieval process, rather than only to post-retrieval ranking. Relatedly, Malki et al. (2025) leverage LLMs to create custom Bluesky feeds. However, for future work, they emphasize the importance of real-time, in-context control: “Rather than treating feedbuilding as a separate setup activity, future systems should support lightweight, iterative changes directly within the feed interface.” CTRL-Rec’s efficiency supports this kind of immediate, interactive control.

LLMs for Recommender Systems and Retrieval. There has been an increasing amount of research focused on leveraging large language models for recommendation (Wu et al., 2024; Wang et al., 2024; Lin et al., 2024; Huang et al., 2024) and information retrieval more broadly (Karpukhin et al., 2020; Khattab & Zaharia, 2020). Our work sits somewhere between recommendation and retrieval, as we provide a way for users to provide natural language descriptions of their preferences which are not necessarily one-time searches (e.g. “never show me angry content about U.S. politics”).

Prior research has directly used LLMs to score the relevance of items to users based on natural language preference strings (Sanner et al., 2023). However, due to the high computational cost associated with LLMs, they are limited to scoring or re-ranking only a small candidate set of items (Jia et al., 2024; Kolluri et al., 2026). More recent work in generative retrieval trains models to directly generate the identifier of the next item the user will interact with (Rajput et al., 2023; Li et al., 2025), however, this approach also suffers from high computational cost and scalability issues to large corpora (Pradeep et al., 2023). In contrast, our approach enables efficient real-time control by distilling synthetic LLM judgments. The general idea of generating synthetic labels via an LLM and then distilling has been applied before in dense retrieval (Izacard & Grave, 2021; Bonifacio et al., 2022; Huang & Chen, 2024),
although with different types of judgments and for different purposes than our goal here (e.g. scoring “relevance” of answers to user questions). Note that, while in this work we distill the LLM judgments into a dual-encoder architecture (Karpukhin et al., 2020; Bromley et al., 1993), our general framework is agnostic to the exact distillation method used and could leverage other methods from dense retrieval.

The goal of our work is to give users explicit control over standard recommendation interfaces through natural language requests. In this way, our work contrasts from other research on providing recommendations limited to a chat interface (Gao et al., 2023; Friedman et al., 2023; He et al., 2023) that do not affect traditional recommender systems. It also contrasts from work that uses natural language features or profiles for recommendation (Mysore et al., 2023b; Kim et al., 2024; Paischer et al., 2024), but where these profiles are extracted or learned, rather than provided by the user as a means of control. Extracted natural language profiles are also common in more recent works on scrutable recommender systems (Radlinski et al., 2022; Penaloza et al., 2025; Ramos et al., 2024; Gao et al., 2025; Mysore et al., 2023a), which also allow for user control via editing of said profiles. However, CTRL-Rec differs both in methodology for connecting natural language to recommendations, and extends the paradigm to accommodate immediate requests (which are non-persistent), enabling more dynamic control.

Figure: Figure 3: Overview of our method. We train a distilled model to approximate LLM judgements of whether a user would like a specific item based on their natural language request. This obviates the need for LLM calls (apart from a single request embedding operation) at test time.
Refer to caption: x3.png

## 3 CTRL-Rec: Control Through Language for Recommendations

Our system integrates natural language control into traditional recommender systems by introducing a novel preference prediction component that estimates how well items align with user-specified requests.

Traditional Recommender Scoring.
Modern recommender systems typically optimize a weighted linear combination of different user-item signals (Milli et al., 2023; Cunningham et al., 2025; Smith, 2021; Twitter, 2023).
Generally, for a given user $u$, the score of an item $i$ is computed as

$$ $\text{score}_{\text{base}}(u,i)=\sum_{k}w_{k}f_{k}(u,i)$ (1) $$

where each function $f_{k}$ returns the value of the $k$-th user-item signal and $w_{k}$ is the weight on that signal. The different signals $f_{k}(u,i)$ are typically probabilities or scores for how likely the user $u$ is to interact with item $i$ in different ways (e.g., liking, commenting). Some signals may also be independent of the user $u$, for example, a signal for the likelihood that the item $i$ violates content moderation standards.

Request-aware Recommender Scoring in CTRL-Rec.
Our key insight is that we can naturally incorporate natural language controls into this framework by explicitly incorporating the user’s natural language request $r$ as an independent signal. Our updated scoring function is defined as:

$$ $\text{score}_{\text{CTRL-Rec}}(u,i,{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}r})=\underbrace{\text{score}_{\text{base}}(u,i)}_{\text{revealed preference + other signals}}\,\,+\,\,w_{\text{control}}\underbrace{{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}v(u,i,r)}}_{\text{stated preference}}$ (2) $$

Here, the function $v(u,i,r)$ represents how satisfied user $u$ would be with item $i$ given their request $r$, and the parameter $w_{\text{control}}$ modulates the trade-off between engagement (the user’s revealed preferences) and the user’s preferences stated in natural language. The nature of the user request $r$ can be very flexible. It could be a simple immediate request or search, or represent more complex, long-term preferences that users would like the recommender system to always follow (e.g. “Never recommend me war movies”) or only follow under certain conditions (e.g. “I do not want to see political content after 8pm”).(^2^22For the latter case, the user state $u$ must contain sufficient information.)

Platforms typically select the weights on different signals through A/B tests (Cunningham et al., 2025; Twitter, 2023; Milli et al., 2023). By tuning the $w_{\text{control}}$ weight, the platform (or potentially even the user)(^3^33Platforms could easily expose an interface that allows the user to select how to trade-off between their language request (stated preferences) and engagement (revealed preferences).) can find an effective balance between the user’s revealed and stated preferences. Also, while [Equation 2](https://arxiv.org/html/2510.12742v1#S3.E2) uses a linear interpolation between engagement and stated preferences to align with industry practice, other promising combination strategies are possible (Milli et al., 2021), such as threshold-based filtering or multiplicative scoring. Empirically, we found that directly interpolating $\text{score}_{\text{base}}$ and $v(u,i,r)$ performed worse than first ranking all movies by each metric separately and then interpolating between those ranks as above – so we follow this approach in our experiments. We suspect that this was due to two functions having very differently tailed distributions within [0, 1], which ranks are more robust to.

### 3.1 Value Prediction

The key challenge for our method is estimating $v(u,i,r)$—how well an item aligns with a user’s request—and doing so efficiently. A naive approach, which we call direct LLM scoring, is to ask an LLM directly to rate how well each candidate item matches the user’s request.
However, the direct LLM scoring is computationally infeasible for at-scale deployments: it requires performing $m$ LLM queries per user request, where $m$ is the number of items in the candidate pool.(^4^44One might wonder if it is possible to simply do one LLM query per user request but ask the LLM to generate ratings for all $m$ items. When $m$ is large, this request requires a long context, particularly if the items are also represented by richer natural language descriptions as in [Section 4](https://arxiv.org/html/2510.12742v1#S4). The long context makes this approach either infeasible or slow.)

To address this, we can distill the LLM’s judgments into a more efficient scoring function—what we call the distilled approach—that reduces the computation needed at deployment time from $m$ LLM queries per user request to just one LLM embedding computation. Specifically, we fine-tune LLM embedding models $f$ and $g$ to approximate LLM judgments: $v(u,i,r)\approx f(u,r)^{T}g(i)$. The item embeddings $g(i)$ can be computed offline, meaning that at deployment time, only one LLM embedding computation per user request (plus a matrix multiplication) is needed. Moreover, as we show in [Section C.1](https://arxiv.org/html/2510.12742v1#A3.SS1) this approach can be further optimized by batching simultaneous user requests together. While the idea of using LLM embeddings for engagement prediction has been explored before, we are not aware of prior work on distilling LLMs for this specific item-value prediction task which is conditioned on user’s natural language requests.

## 4 Experiments

To evaluate the benefits of CTRL-Rec—which enables both (a) real-time control at the retrieval stage and (b) balancing engagement with stated preferences—we require access to a large and up-to-date inventory of items, along with user engagement data. Although CTRL-Rec’s advantages are likely to be most evident in high-volume, rapidly changing content domains such as social media, news recommendation, or e-commerce, these domains typically lack publicly available, timely large-scale item inventories with sufficient engagement data. Access to a recent inventory is particularly important, as we aim to test our method in user studies, and users would expect recommendations to be relevant and current.(^5^55For example, while the MIND dataset (Wu et al., 2020) is a publicly available news dataset with impression logs, it does not contain up-to-date news data suitable for a user study.)

In light of this, we test CTRL-Rec in the movie recommendation domain, which has standard large-scale datasets of items and user interactions, including recent movies that remain relevant to users today. We evaluate CTRL-Rec’s performance through two types of complementary experiments(^6^66Meta served in only an advisory capacity. The research participant study and all experiments were led and implemented by UC Berkeley, UW, and UT Austin.): (a) a reachability experiment (Dean et al., 2020; Curmei et al., 2021), and (b) a human study involving users of Letterboxd, a movie ratings platform. In the reachability experiment, we randomly select two users from the MovieLens dataset—a source user and a target user—and assess how easily the source user’s recommendations can be adjusted to resemble those of the target user. This large-scale experiment evaluates CTRL-Rec’s potential steering to *any* observed preferences in the dataset. By contrast, our human study is smaller in scale but focuses on whether CTRL-Rec help real users steer their recommendations with the *specific* preferences they have in mind. When referring to engagement-based recommendations, across all of our experiments we use the SAR (Sequential Association Rules) algorithm, as implemented by Microsoft’s recommender module (Graham et al., 2019).

[Figure 4](https://arxiv.org/html/2510.12742v1#S4.F4) shows qualitative examples returned by CTRL-Rec for three different queries. Despite the nuanced and open-ended nature of some of these requests, CTRL-Rec effectively produces relevant results. During deployment, each query requires only one LLM embedding computation, allowing for real-time control of recommendations.

### 4.1 Training Details

CTRL-Rec was trained on 220,000 simulated user preferences generated from the MovieLens 32M dataset, which contains $\sim$32m ratings by $\sim$330k unique users across $\sim$87k movies. For each movie in the dataset, we prompted Claude 3.5 to generate a summary as described in [Appendix B](https://arxiv.org/html/2510.12742v1#A2), which is used as input to the CTRL-Rec as a richer representation of each movie than just the title. In other domains, one could use available metadata one has for the item. Our training data generation process consisted of two main stages: (1) request generation, and (2) preference scoring.

Request Generation. We first generated a diverse set of natural language requests by sampling users from the MovieLens dataset and instructing gemini-2.5-flash-preview-05-20 to generate requests across 10 different categories designed to span the range of ways users might express movie preferences. These categories include one-time situational requests, long-term persistent preferences, aspirational requests for personal growth, similarity-based comparisons, and various types of filtering requests ranging from simple to complex logical operations. This approach aims at broad coverage of request types that users might realistically make, though in practice a deployed system could be trained on the actual distribution of user requests.

Preference Scoring. For each simulated preference, we randomly sample movies from the dataset and used Llama 3.1 70B to score how well each movie matched the specific preference request. These LLM-generated scores serve as the target preferences for the distillation step described in our method (see [Appendix C](https://arxiv.org/html/2510.12742v1#A3) for details on extracting scores from LLMs). This process was repeated across many user-preference-movie combinations to generate our final training dataset of 220,000 preference examples. See [Section D.2](https://arxiv.org/html/2510.12742v1#A4.SS2) for detailed descriptions of the 10 request categories and the prompts used to generate each type.

Figure: Figure 4: Qualitative examples of CTRL-Rec recommendations for three diverse natural language requests ($w_{\text{control}}=0.995$ & no engagement history).

### 4.2 Reachability Experiment

In our reachability experiment (Dean et al., 2020; Curmei et al., 2021), we randomly select two users from the MovieLens dataset—a source user and a target user—and evaluate how easily the source user’s recommendations can be adjusted to resemble the engagement-based recommendations of the target user using both CTRL-Rec and traditional filters (specifically, for genre and decade). We chose these particular filters to align with the existing controls available on Letterboxd (see [Figure 8](https://arxiv.org/html/2510.12742v1#A4.F8) for a screenshot), a popular movie ratings platform whose users we recruit for our experiments in [Section 4.3](https://arxiv.org/html/2510.12742v1#S4.SS3).

Our experimental procedure simulates a user who has a specific target recommendation feed in mind and systematically explores different combinations of filters and LLM requests to achieve it. We compare two approaches: (1) filters only, and (2) filters combined with CTRL-Rec’s natural language control.

Feed Quality Evaluation. For each experiment trial, we generate the top-10 recommendations for both the source and target users using our engagement-based recommender system. We evaluate recommendation quality using two metrics: (1) cosine similarity distance (primary), and (2) percentage overlap (secondary). The cosine similarity between recommendation feeds is computed as:

$$ $\text{sim}(F_{1},F_{2})=\frac{\bar{\mathbf{e}}_{F_{1}}\cdot\bar{\mathbf{e}}_{F_{2}}}{\|\bar{\mathbf{e}}_{F_{1}}\|_{2}\|\bar{\mathbf{e}}_{F_{2}}\|_{2}}$ (3) $$

where $\bar{\mathbf{e}}_{F}=\frac{1}{|F|}\sum_{i\in F}\mathbf{e}_{i}$ is the average embedding for feed $F$, and $\mathbf{e}_{i}$ represents the item embedding for item $i$. This is our primary optimization target as it measures semantic similarity between recommendation sets. However, since embedding distance is approximate and relies on the quality of the underlying embeddings, we also use percentage overlap as a secondary metric, which provides a straightforward ground-truth measure that is easy to interpret – but is much more noisy, as in many cases the percentage overlap is 0.

Filter-Only Approach. We evaluate various filter combinations (genre and decade) applied to the source user’s recommendations. To optimistically estimate users’ capacity to use filters effectively, we use a greedy approach based on the target feed’s genre and decade distributions. Since movies can belong to many genres and decades, considering all possible filter combinations would be computationally expensive, and most combinations would yield empty feeds with no results. Our greedy approach focuses on filter combinations most likely to yield meaningful results while still aiming to overestimate what users might realistically achieve through manual exploration.
We select the combination that produces recommendations closest to the target feed in embedding space. See [Section D.1](https://arxiv.org/html/2510.12742v1#A4.SS1) for details on our greedy approach.

Filter + LLM Approach. For the CTRL-Rec approach, we simulate an iterative user who refines their natural language requests based on feedback using Gemini Flash 2.5 as the LLM agent. We select starting filter combinations by taking the best-performing filter combination from the filter-only approach and considering all of its subsets as potential starting points. The intuition is that while the best filter combination may be optimal when filters are the only available lever, it might filter out important movies that an LLM could better target through natural language. By allowing the LLM to start from subsets (including no filters at all), we test whether natural language control can compensate for reduced filtering.

For each starting filter combination, an LLM agent: (1) observes current recommendations, (2) compares them to the target feed, (3) generates a natural language request to steer recommendations closer to the target, (4) applies the request via CTRL-Rec, and (5) repeats for up to 3 iterations. The agent refines its strategy based on previous results, simulating how users might iteratively adjust queries. For each starting filter combination, we select the natural language request (across all iterations) that maximizes cosine similarity to the target feed.

Figure: (a) Percentage of distance to target feeds achieved by different approaches.
Refer to caption: figures/sim/perc_distance2.png

Results. As seen in [Figure 5](https://arxiv.org/html/2510.12742v1#S4.F5), we see that both filters and CTRL-Rec are very effective at bridging the gap between source and target feeds, with a large majority of the gap being closed for source-target pairs which are more distant. However, across all distances, CTRL-Rec consistently outperforms the filters. Metrics are plotted averaged across bins (with standard errors), where bins were formed by taking quintiles of the source-target distances. Moreover, [Figure 5](https://arxiv.org/html/2510.12742v1#S4.F5) also shows that distance reduction in embedding space (our primary measure) is accompanied by an increase in overlap with the target feed (our secondary measure), showing that the embedding distance metric we optimized also correlates with other desirable measures.

### 4.3 Human Study with Letterboxd Users

Our reachability experiment provided insight into the theoretical controllability advantages of CTRL-Rec, but we also wanted to see whether and how these advantages surface in real-world use. Thus, we conducted a human study with $n=19$ Letterboxd users to evaluate the effectiveness of CTRL-Rec in a movie recommendation task leveraging users’ own Letterboxd data. We recruited participants through social media posts, university Slack channels, and random sampling of personal connections that met the study criteria (18+ years old, have 10+ movie ratings on Letterboxd). Our study was approved by our institution’s IRB, and our participants were paid a $20 USD gift card for participating in the 30-minute study.

Figure: Figure 6: Human study results comparing CTRL-Rec with traditional filters to filters-only baseline. Participants found CTRL-Rec’s recommendations more engaging, were more satisfied with them, and found them easier to control. Significance levels are indicated using $q$-values (false discovery rate adjusted $p$-values): ^∗ indicates $q<0.05$; ^∗∗ indicates $q<0.01$.
Refer to caption: x4.png

Setup. Our study had a within-subjects design with two counterbalanced conditions: (a) CTRL-Rec(^7^77We used a $w_{\text{control}}=0.995$ for the trade-off between engagement and stated preferences ([Equation 2](https://arxiv.org/html/2510.12742v1#S3.E2)).) + genre and decade filters, (b) genre and decade filters only. We generated an initial set of recommendations with an engagement-based recommender feed based on their Letterboxd ratings using the recommender in [Section 4.2](https://arxiv.org/html/2510.12742v1#S4.SS2). Participants were then given 8 minutes per condition to use the interface to find their ideal movie recommendations. For each recommended movie, we also provided participants “Interested” or “Watched” buttons that they could use to ‘like’ the movie or indicate that they had already seen it. The “Interested” (like) button was used to measure user engagement, while the “Watched” button was used to track the rate of redundant recommendations—movies that reflect historical engagement but are unlikely to predict future engagement because the user has already seen them. After each condition, participants answered Likert scale survey questions (on a 7-point scale) to rate their experience with the system. Upon completing both conditions, participants were asked to assess the overall value of the natural language controls and took part in a brief, semi-structured exit interview about their experiences. Further details and study materials can be found in [Appendix E](https://arxiv.org/html/2510.12742v1#A5).

Quantitative Results. We compared participants’ Likert scale ratings across CTRL-Rec and our baseline for a variety of measures, including controllability, satisfaction with recommendations, and engagement. All measures can be found in Table [1](https://arxiv.org/html/2510.12742v1#A5.T1). To assess statistical significance, we used Wilcoxon signed-rank tests. To account for multiple comparisons, we applied the Benjamini-Krieger-Yekutieli (BKY) two-stage method (Benjamini et al., 2006), which controls the false discovery rate (FDR). For each measure, we report both the raw $p$-value and the corresponding $q$-value in Table [1](https://arxiv.org/html/2510.12742v1#A5.T1). The $q$-value represents the minimum FDR at which a result is considered significant; it is an adjusted $p$-value that accounts for the expected proportion of false positives among all significant findings. All results that are significant at the conventional $\alpha=0.05$ level are also significant at an FDR threshold of $q\leq 0.039$.

Let $M_{C}$ and $M_{B}$ denote the mean [Likert rating or engagement ratio] for CTRL-Rec and the baseline, respectively. CTRL-Rec was rated as significantly more controllable than the baseline ($q=0.039;M_{C}=5.4>M_{B}=4.4$). Users could also express their preferences significantly more easily in CTRL-Rec ($q=0.001;M_{C}=6.2>M_{B}=3.2$). Additionally, participants reported greater satisfaction with their recommendations in CTRL-Rec ($q=0.038;M_{C}=5.7>M_{B}=4.9$). Despite the need to articulate their preferences in natural language when using CTRL-Rec, participants did not report a significant increase in effort compared to the baseline ($q=0.268;M_{C}=3.6>M_{B}=3.4$). Additionally, we found that CTRL-Rec recommended significantly fewer movies users have already watched ($q=0.027;M_{C}=0.05<M_{B}=0.15$), but with no significant difference in user engagement, i.e., the percent of movie recommendations that they ‘liked’ ($q=0.308;M_{C}=0.16>M_{B}=0.14$). Overall, participants agreed that the natural language controls in CTRL-Rec were valuable ($M=6.1\pm 0.9$). Once natural language controls were available, participants relied less on the provided filters, and the perceived value of filters was significantly lower in CTRL-Rec compared to the baseline ($q=0.006;M_{C}=4.2<M_{B}=6.1$).

Qualitative Results. We qualitatively analyzed participants’ quotes from our studies to identify specific factors that made CTRL-Rec more controllable for users. Many participants appreciated that CTRL-Rec allowed them to simultaneously find movies based on abstract “vibes” while also supporting highly specific queries. For example, P5 shared that “I understand the vibe of what I want, but I don’t know that I would put it into a particular genre—that’s kind of a really broad catch-all.” P3 thought that questions like “what’s a good film about food to help explain this anthropological problem” can’t be easily answered without natural language controls. The flexibility of CTRL-Rec allowed participants to easily tune the level of specificity: P10 initially wrote a detailed query but then realized “I don’t want to just be given exactly what I want” and saw value in “deliberately underspecifying” her preferences to the system.

CTRL-Rec also allowed participants to anchor their recommendations on other movies they’ve enjoyed. P9 included “Movies like Dune or Game of Thrones or Lord of the Rings” in his query. Similarly, P16 described his overall strategy as: “I was thinking of 5-star movies that I like. And I was trying to find something similar.” This resulted in recommendations that were both high-quality and pleasantly unfamiliar. Even P20’s simple query of “Find older movies I would like” yielded recommendations they thought were “pretty good, I’d never heard of [them] but seems like some things I would like.” P12 described CTRL-Rec allowed him to reach “the outer regions of [common recommendations] where there are weird things I haven’t seen but interest me,” whereas the baseline system recommended mostly “stuff I’ve heard of, or films that I’d actually watched and forgotten.” P23 echoed this, sharing that CTRL-Rec’s recommendations are hard to come by with conventional search: “I think the recommendations are really nice. I’m surprised that I don’t know about these, so it’s exciting. I probably wouldn’t have come across them on Google.” This is also reflected in the quantitative results from [Figure 6](https://arxiv.org/html/2510.12742v1#S4.F6): CTRL-Rec generally showed users movies that they were less familiar with, relative to the standard engagement recommender system combined with filters (as seen by “Watched Ratio”).

Finally, an interesting use case of CTRL-Rec for participants was finding recommendations based on group preferences. P20 queried CTRL-Rec for movies him and his partner could enjoy together, while P23, who organizes regular movie nights with friends, shared that CTRL-Rec would be valuable for finding movies that “a bunch of friends who are also doing a PhD and who are into sci-fi can really enjoy in a group setting.”

## 5 Discussion

Our results demonstrate that CTRL-Rec effectively integrates natural language requests into traditional recommender systems, granting users fine-grained real-time control over their recommendations. The success of our distilled approach suggests that natural language control can be made computationally efficient enough for practical deployment, requiring only a single embedding computation per user request. Findings from our human study with Letterboxd users highlight the real-world value of our method in bolstering system controllability and user satisfaction of recommended items without compromising user engagement.

Limitations and Future Work. Our evaluation is limited by the scarcity of publicly available domains that offer both large-scale, up-to-date item inventories and corresponding engagement data. As a result, we focus on the movie recommendation domain, utilizing the MovieLens dataset as our item inventory. While standard LLMs can generate movie recommendations via direct queries, this approach is significantly more computationally expensive. Our chosen domain thus serves as a proof-of-concept for more complex and dynamic settings—such as rapidly evolving social media feeds or large-scale e-commerce catalogs—where the efficiency gains of our method would be especially valuable.

The MovieLens dataset presents its own limitations. It primarily consists of well-known movies that LLMs likely encountered during training, so we do not directly test generalization to truly novel items—a common challenge for retrieval-based systems. Additionally, for our user study, we recruited participants from the Letterboxd platform, primarily due to its ability to export movie ratings that can be used to provide personalized recommendations each user. However, this may introduce bias, as Letterboxd users are typically film enthusiasts and may not reflect the broader population.

Our current evaluation is limited to text-based items, though our framework would naturally extend to other modalities (e.g., images or videos). Finally, we do not address more complex feed-level requests (e.g., “I want half of my recommendations to be of type X, half of type Y”), which remain an interesting avenue for future work.

Conclusion. We have presented CTRL-Rec, a framework for incorporating natural language user controls into traditional recommender systems. Our key insight is that LLMs can be used to simulate how well items align with user requests, and these simulated judgments can be distilled and decomposed into efficient dot products between user-request and item embeddings. This decomposition allows us to reduce the computational cost by orders of magnitude, requiring only one LLM embedding computation per user request rather than per item. Our experiments demonstrate that this approach allows for effective steering of recommendations according to user requests while maintaining high engagement metrics. The method is computationally efficient and practical for real-world deployment. We believe this work represents a step toward recommender systems that better balance engagement optimization with explicit user preferences and control.

## Acknowledgments

This work has taken place in part in the Rewarding Lab at UT Austin. During this project, the Rewarding Lab has been supported by NSF (IIS-2402650), ONR (N00014-22-1-2204), ARO (W911NF-25-1-0254), Emerson, EA Ventures, UT Austin’s Good Systems grand challenge, and Open Philanthropy. We thank Luke Thorburn and Manzil Zaheer for their feedback. We thank the Berkeley Existential Risk Initiative (BERI) for funding our user study.

## References

- Agarwal et al. (2024)
Arpit Agarwal, Nicolas Usunier, Alessandro Lazaric, and Maximilian Nickel.
System-2 recommenders: Disentangling utility and engagement in recommendation systems via temporal point-processes.
In *The 2024 ACM Conference on Fairness, Accountability, and Transparency*, pp. 1763–1773, 2024.
- Anderson (2008)
Michael L Anderson.
Multiple inference and gender differences in the effects of early intervention: A reevaluation of the abecedarian, perry preschool, and early training projects.
*Journal of the American statistical Association*, 103(484):1481–1495, 2008.
- Benjamini et al. (2006)
Yoav Benjamini, Abba M Krieger, and Daniel Yekutieli.
Adaptive linear step-up procedures that control the false discovery rate.
*Biometrika*, 93(3):491–507, 2006.
- Bonifacio et al. (2022)
Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira.
InPars: Data Augmentation for Information Retrieval using Large Language Models, February 2022.
URL [http://arxiv.org/abs/2202.05144](http://arxiv.org/abs/2202.05144).
arXiv:2202.05144 [cs].
- Bromley et al. (1993)
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah.
Signature verification using a ”Siamese” time delay neural network.
In *Proceedings of the 7th International Conference on Neural Information Processing Systems*, NIPS’93, pp. 737–744, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc.
- Cunningham et al. (2025)
Tom Cunningham, Sana Pandey, Leif Sigerson, Jonathan Stray, Jeff Allen, Bonnie Barrilleaux, Ravi Iyer, Mohit Kothari, Behnam Rezaei, Sanjay Kairam, et al.
Ranking by Engagement and Non-Engagement Signals: Learnings from Industry.
*Annals of the New York Academy of Sciences*, 2025.
- Curmei et al. (2021)
Mihaela Curmei, Sarah Dean, and Benjamin Recht.
Quantifying availability and discovery in recommender systems via stochastic reachability.
In Marina Meila and Tong Zhang (eds.), *Proceedings of the 38th International Conference on Machine Learning*, volume 139 of *Proceedings of Machine Learning Research*, pp. 2265–2275. PMLR, 18–24 Jul 2021.
URL [https://proceedings.mlr.press/v139/curmei21a.html](https://proceedings.mlr.press/v139/curmei21a.html).
- Dean et al. (2020)
Sarah Dean, Sarah Rich, and Benjamin Recht.
Recommendations and User Agency: The Reachability of Collaboratively-Filtered Information.
In *Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency*, pp. 436–445, January 2020.
doi: 10.1145/3351095.3372866.
URL [http://arxiv.org/abs/1912.10068](http://arxiv.org/abs/1912.10068).
arXiv:1912.10068 [cs].
- Ekstrand & Willemsen (2016)
Michael D. Ekstrand and Martijn C. Willemsen.
Behaviorism is not enough: Better recommendations through listening to users.
In *Proceedings of the 10th ACM Conference on Recommender Systems*, RecSys ’16, pp. 221–224, New York, NY, USA, 2016. Association for Computing Machinery.
ISBN 9781450340359.
doi: 10.1145/2959100.2959179.
URL [https://doi.org/10.1145/2959100.2959179](https://doi.org/10.1145/2959100.2959179).
- Feng et al. (2024)
K. J. Kevin Feng, Xander Koo, Lawrence Tan, Amy Bruckman, David W. McDonald, and Amy X. Zhang.
Mapping the Design Space of Teachable Social Media Feed Experiences.
In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*, CHI ’24, New York, NY, USA, 2024. Association for Computing Machinery.
ISBN 9798400703300.
doi: 10.1145/3613904.3642120.
URL [https://doi.org/10.1145/3613904.3642120](https://doi.org/10.1145/3613904.3642120).
- Friedman et al. (2023)
Luke Friedman, Sameer Ahuja, David Allen, Zhenning Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, Brian Chu, Zexi Chen, and Manoj Tiwari.
Leveraging Large Language Models in Conversational Recommender Systems, May 2023.
URL [http://arxiv.org/abs/2305.07961](http://arxiv.org/abs/2305.07961).
arXiv:2305.07961 [cs].
- Gao et al. (2023)
Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang.
Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System, April 2023.
URL [http://arxiv.org/abs/2303.14524](http://arxiv.org/abs/2303.14524).
arXiv:2303.14524 [cs].
- Gao et al. (2025)
Zhaolin Gao, Joyce Zhou, Yijia Dai, and Thorsten Joachims.
End-to-end Training for Recommendation with Language-based User Profiles, February 2025.
URL [http://arxiv.org/abs/2410.18870](http://arxiv.org/abs/2410.18870).
arXiv:2410.18870 [cs].
- Graham et al. (2019)
Scott Graham, Jun-Ki Min, and Tao Wu.
Microsoft Recommenders: Tools to Accelerate Developing Recommender Systems.
In *Proceedings of the 13th ACM Conference on Recommender Systems*, RecSys ’19, pp. 542–543, New York, NY, USA, 2019. Association for Computing Machinery.
ISBN 9781450362436.
doi: 10.1145/3298689.3346967.
URL [https://doi.org/10.1145/3298689.3346967](https://doi.org/10.1145/3298689.3346967).
- He et al. (2023)
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley.
Large Language Models as Zero-Shot Conversational Recommenders.
In *Proceedings of the 32nd ACM International Conference on Information and Knowledge Management*, pp. 720–730, October 2023.
doi: 10.1145/3583780.3614949.
URL [http://arxiv.org/abs/2308.10053](http://arxiv.org/abs/2308.10053).
arXiv:2308.10053 [cs].
- Huang & Chen (2024)
Chao-Wei Huang and Yun-Nung Chen.
PairDistill: Pairwise Relevance Distillation for Dense Retrieval.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 18225–18237, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.1013.
URL [https://aclanthology.org/2024.emnlp-main.1013/](https://aclanthology.org/2024.emnlp-main.1013/).
- Huang et al. (2024)
Chengkai Huang, Tong Yu, Kaige Xie, Shuai Zhang, Lina Yao, and Julian McAuley.
Foundation Models for Recommender Systems: A Survey and New Perspectives, February 2024.
URL [http://arxiv.org/abs/2402.11143](http://arxiv.org/abs/2402.11143).
arXiv:2402.11143 [cs].
- Izacard & Grave (2021)
Gautier Izacard and Edouard Grave.
Distilling knowledge from reader to retriever for question answering.
In *ICLR*, 2021.
- Jahanbakhsh et al. (2025)
Farnaz Jahanbakhsh, Dora Zhao, Tiziano Piccardi, Zachary Robertson, Ziv Epstein, Sanmi Koyejo, and Michael S. Bernstein.
Value alignment of social media ranking algorithms, 2025.
URL [https://arxiv.org/abs/2509.14434](https://arxiv.org/abs/2509.14434).
- Jia et al. (2024)
Chenyan Jia, Michelle S. Lam, Minh Chau Mai, Jeffrey T. Hancock, and Michael S. Bernstein.
Embedding Democratic Values into Social Media AIs via Societal Objective Functions.
*Proc. ACM Hum.-Comput. Interact.*, 8(CSCW1), April 2024.
doi: 10.1145/3641002.
URL [https://doi.org/10.1145/3641002](https://doi.org/10.1145/3641002).
- Karpukhin et al. (2020)
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
Dense Passage Retrieval for Open-Domain Question Answering, September 2020.
URL [http://arxiv.org/abs/2004.04906](http://arxiv.org/abs/2004.04906).
arXiv:2004.04906 [cs].
- Khattab & Zaharia (2020)
Omar Khattab and Matei Zaharia.
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.
In *Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval*, SIGIR ’20, pp. 39–48, New York, NY, USA, 2020. Association for Computing Machinery.
ISBN 9781450380164.
doi: 10.1145/3397271.3401075.
URL [https://doi.org/10.1145/3397271.3401075](https://doi.org/10.1145/3397271.3401075).
- Kim et al. (2024)
Jieyong Kim, Hyunseo Kim, Hyunjin Cho, SeongKu Kang, Buru Chang, Jinyoung Yeo, and Dongha Lee.
Review-driven personalized preference reasoning with large language models for recommendation, 2024.
URL [https://arxiv.org/abs/2408.06276](https://arxiv.org/abs/2408.06276).
- Kleinberg et al. (2024a)
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Manish Raghavan.
The inversion problem: Why algorithms should infer mental state and not just predict behavior.
*Perspectives on Psychological Science*, 19(5):827–838, 2024a.
- Kleinberg et al. (2024b)
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan.
The challenge of understanding what users want: Inconsistent preferences and engagement optimization.
*Management science*, 70(9):6336–6355, 2024b.
- Kolluri et al. (2026)
Akaash Kolluri, Renn Su, Farnaz Jahanbakhsh, Dora Zhao, Tiziano Piccardi, and Michael S. Bernstein.
Alexandria: A Library of Pluralistic Values for Realtime Re-Ranking of Social Media Feeds.
In *Proceedings of the International Conference on Web and Social Media (ICWSM)*, 2026.
- Lazar et al. (2024)
Seth Lazar, Luke Thorburn, Tian Jin, and Luca Belli.
The Moral Case for Using Language Model Agents for Recommendation, October 2024.
URL [http://arxiv.org/abs/2410.12123](http://arxiv.org/abs/2410.12123).
arXiv:2410.12123 [cs].
- Li et al. (2025)
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou.
From Matching to Generation: A Survey on Generative Information Retrieval.
*ACM Trans. Inf. Syst.*, 43(3), May 2025.
ISSN 1046-8188.
doi: 10.1145/3722552.
URL [https://doi.org/10.1145/3722552](https://doi.org/10.1145/3722552).
- Lin et al. (2024)
Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Hao Zhang, Yong Liu, Chuhan Wu, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang.
How Can Recommender Systems Benefit from Large Language Models: A Survey, July 2024.
URL [http://arxiv.org/abs/2306.05817](http://arxiv.org/abs/2306.05817).
arXiv:2306.05817 [cs].
- Lukoff et al. (2021)
Kai Lukoff, Ulrik Lyngs, Himanshu Zade, J. Vera Liao, James Choi, Kaiyue Fan, Sean A. Munson, and Alexis Hiniker.
How the Design of YouTube Influences User Sense of Agency.
In *Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems*, CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery.
ISBN 9781450380966.
doi: 10.1145/3411764.3445467.
URL [https://doi.org/10.1145/3411764.3445467](https://doi.org/10.1145/3411764.3445467).
- Lukoff et al. (2023)
Kai Lukoff, Ulrik Lyngs, Karina Shirokova, Raveena Rao, Larry Tian, Himanshu Zade, Sean A. Munson, and Alexis Hiniker.
SwitchTube: A Proof-of-Concept System Introducing “Adaptable Commitment Interfaces” as a Tool for Digital Wellbeing.
In *Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems*, CHI ’23, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9781450394215.
doi: 10.1145/3544548.3580703.
URL [https://doi.org/10.1145/3544548.3580703](https://doi.org/10.1145/3544548.3580703).
- Malki et al. (2025)
Omar El Malki, Marianne Aubin Le Quéré, Andrés Monroy-Hernández, and Manoel Horta Ribeiro.
Bonsai: Intentional and Personalized Social Media Feeds, 2025.
URL [https://arxiv.org/abs/2509.10776](https://arxiv.org/abs/2509.10776).
- Milli et al. (2021)
Smitha Milli, Luca Belli, and Moritz Hardt.
From Optimizing Engagement to Measuring Value.
In *Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency*, pp. 714–722, 2021.
- Milli et al. (2023)
Smitha Milli, Emma Pierson, and Nikhil Garg.
Choosing the right weights: Balancing value, strategy, and noise in recommender systems.
*arXiv preprint arXiv:2305.17428*, 2023.
- Milli et al. (2025)
Smitha Milli, Micah Carroll, Yike Wang, Sashrika Pandey, Sebastian Zhao, and Anca D Dragan.
Engagement, user satisfaction, and the amplification of divisive content on social media.
*PNAS Nexus*, 4(3):pgaf062, 03 2025.
ISSN 2752-6542.
doi: 10.1093/pnasnexus/pgaf062.
URL [https://doi.org/10.1093/pnasnexus/pgaf062](https://doi.org/10.1093/pnasnexus/pgaf062).
- Morewedge et al. (2023)
Carey K Morewedge, Sendhil Mullainathan, Haaya F Naushan, Cass R Sunstein, Jon Kleinberg, Manish Raghavan, and Jens O Ludwig.
Human bias in algorithm design.
*Nature Human Behaviour*, 7(11):1822–1824, 2023.
- Mozilla (2023)
Mozilla.
Youtube regrets: A crowdsourced investigation into youtube’s recommendation algorithm.
[https://foundation.mozilla.org/en/youtube/findings/](https://foundation.mozilla.org/en/youtube/findings/), 2023.
Accessed: 2023-10-17.
- Mysore et al. (2023a)
Sheshera Mysore, Mahmood Jasim, Andrew McCallum, and Hamed Zamani.
Editable User Profiles for Controllable Text Recommendation, October 2023a.
URL [http://arxiv.org/abs/2304.04250](http://arxiv.org/abs/2304.04250).
arXiv:2304.04250 [cs].
- Mysore et al. (2023b)
Sheshera Mysore, Andrew Mccallum, and Hamed Zamani.
Large language model augmented narrative driven recommendations.
In *Proceedings of the 17th ACM Conference on Recommender Systems*, RecSys ’23, pp. 777–783, New York, NY, USA, 2023b. Association for Computing Machinery.
ISBN 9798400702419.
doi: 10.1145/3604915.3608829.
URL [https://doi.org/10.1145/3604915.3608829](https://doi.org/10.1145/3604915.3608829).
- Paischer et al. (2024)
Fabian Paischer, Liu Yang, Linfeng Liu, Shuai Shao, Kaveh Hassani, Jiacheng Li, Ricky Chen, Zhang Gabriel Li, Xialo Gao, Wei Shao, Xue Feng, Nima Noorshams, Sem Park, Bo Long, and Hamid Eghbalzadeh.
Preference Discerning with LLM-Enhanced Generative Retrieval, December 2024.
URL [http://arxiv.org/abs/2412.08604](http://arxiv.org/abs/2412.08604).
arXiv:2412.08604 [cs].
- Penaloza et al. (2025)
Emiliano Penaloza, Olivier Gouvert, Haolun Wu, and Laurent Charlin.
TEARS: Textual Representations for Scrutable Recommendations, March 2025.
URL [http://arxiv.org/abs/2410.19302](http://arxiv.org/abs/2410.19302).
arXiv:2410.19302 [cs].
- Piccardi et al. (2024)
Tiziano Piccardi, Martin Saveski, Chenyan Jia, Jeffrey T Hancock, Jeanne L Tsai, and Michael Bernstein.
Social media algorithms can shape affective polarization via exposure to antidemocratic attitudes and partisan animosity.
*arXiv preprint arXiv:2411.14652*, 2024.
- Pradeep et al. (2023)
Ronak Pradeep, Kai Hui, Jai Gupta, Adam Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Tran.
How does generative retrieval scale to millions of passages?
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 1305–1321, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.83.
URL [https://aclanthology.org/2023.emnlp-main.83/](https://aclanthology.org/2023.emnlp-main.83/).
- Radlinski et al. (2022)
Filip Radlinski, Krisztian Balog, Fernando Diaz, Lucas Dixon, and Ben Wedin.
On Natural Language User Profiles for Transparent and Scrutable Recommendation.
In *Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval*, pp. 2863–2874, July 2022.
doi: 10.1145/3477495.3531873.
URL [http://arxiv.org/abs/2205.09403](http://arxiv.org/abs/2205.09403).
arXiv:2205.09403 [cs].
- Rajput et al. (2023)
Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, Maciej Kula, Ed Chi, and Maheswaran Sathiamoorthy.
Recommender Systems with Generative Retrieval.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 10299–10315. Curran Associates, Inc., 2023.
URL [https://proceedings.neurips.cc/paper_files/paper/2023/file/20dcab0f14046a5c6b02b61da9f13229-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/20dcab0f14046a5c6b02b61da9f13229-Paper-Conference.pdf).
- Ramos et al. (2024)
Jerome Ramos, Hossen A. Rahmani, Xi Wang, Xiao Fu, and Aldo Lipani.
Transparent and Scrutable Recommendations Using Natural Language User Profiles, July 2024.
URL [http://arxiv.org/abs/2402.05810](http://arxiv.org/abs/2402.05810).
arXiv:2402.05810 [cs].
- Rathje et al. (2024)
Steve Rathje, Claire Robertson, William J Brady, and Jay J Van Bavel.
People think that social media platforms do (but should not) amplify divisive content.
*Perspectives on Psychological Science*, 19(5):781–795, 2024.
- Rezk et al. (2024)
Anna Marie Rezk, Auste Simkute, Ewa Luger, John Vines, Chris Elsden, Michael Evans, and Rhianne Jones.
Agency Aspirations: Understanding Users’ Preferences And Perceptions Of Their Role In Personalised News Curation.
In *Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*, CHI ’24, New York, NY, USA, 2024. Association for Computing Machinery.
ISBN 9798400703300.
doi: 10.1145/3613904.3642634.
URL [https://doi.org/10.1145/3613904.3642634](https://doi.org/10.1145/3613904.3642634).
- Sanner et al. (2023)
Scott Sanner, Krisztian Balog, Filip Radlinski, Ben Wedin, and Lucas Dixon.
Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences, July 2023.
URL [http://arxiv.org/abs/2307.14225](http://arxiv.org/abs/2307.14225).
arXiv:2307.14225 [cs].
- Smith (2021)
Ben Smith.
How TikTok reads your mind, Dec 2021.
URL [https://www.nytimes.com/2021/12/05/business/media/tiktok-algorithm.html](https://www.nytimes.com/2021/12/05/business/media/tiktok-algorithm.html).
- Tan et al. (2025)
Felicia Fang-Yi Tan, Ashwin Ram, Moritz Alexander Messerschmidt, Hasini Amanda Dissanayake, and Suranga Nanayakkara.
Curious shorts: Curiosity-driven exploration and learning on short-form video platforms.
In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems*, CHI ’25, New York, NY, USA, 2025. Association for Computing Machinery.
ISBN 9798400713941.
doi: 10.1145/3706598.3713951.
URL [https://doi.org/10.1145/3706598.3713951](https://doi.org/10.1145/3706598.3713951).
- Twitter (2023)
Twitter.
Twitter’s Recommendation Algorithm - Heavy Ranker and TwHIN embeddings, Mar 2023.
URL [https://github.com/twitter/the-algorithm-ml/tree/main/projects/home/recap](https://github.com/twitter/the-algorithm-ml/tree/main/projects/home/recap).
- Wang et al. (2024)
Qi Wang, Jindong Li, Shiqi Wang, Qianli Xing, Runliang Niu, He Kong, Rui Li, Guodong Long, Yi Chang, and Chengqi Zhang.
Towards Next-Generation LLM-based Recommender Systems: A Survey and Beyond, October 2024.
URL [http://arxiv.org/abs/2410.19744](http://arxiv.org/abs/2410.19744).
arXiv:2410.19744 [cs].
- Williams et al. (2024)
Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dragan.
On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback, November 2024.
URL [http://arxiv.org/abs/2411.02306](http://arxiv.org/abs/2411.02306).
arXiv:2411.02306 [cs].
- Wu et al. (2020)
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou.
MIND: A Large-scale Dataset for News Recommendation.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp. 3597–3606, Online, July 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.331.
URL [https://aclanthology.org/2020.acl-main.331/](https://aclanthology.org/2020.acl-main.331/).
- Wu et al. (2024)
Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen.
A Survey on Large Language Models for Recommendation, June 2024.
URL [http://arxiv.org/abs/2305.19860](http://arxiv.org/abs/2305.19860).
arXiv:2305.19860 [cs].
- Yang et al. (2019)
Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin.
How intention informed recommendations modulate choices: A field study of spoken word content.
In *The World Wide Web Conference*, WWW ’19, pp. 2169–2180, New York, NY, USA, 2019. Association for Computing Machinery.
ISBN 9781450366748.
doi: 10.1145/3308558.3313540.
URL [https://doi.org/10.1145/3308558.3313540](https://doi.org/10.1145/3308558.3313540).
- Zhang et al. (2022)
Mingrui Ray Zhang, Kai Lukoff, Raveena Rao, Amanda Baughan, and Alexis Hiniker.
Monitoring Screen Time or Redesigning It? Two Approaches to Supporting Intentional Social Media Use.
In *Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems*, CHI ’22, New York, NY, USA, 2022. Association for Computing Machinery.
ISBN 9781450391573.
doi: 10.1145/3491102.3517722.
URL [https://doi.org/10.1145/3491102.3517722](https://doi.org/10.1145/3491102.3517722).

## Appendix A Usage of LLMs.

We used LLMs to help with rephrasing individual paragraphs of the paper for improved clarity, and writing code for the experiments.

## Appendix B Prompts

### B.1 Movie Summary Generation

To ensure consistent and informative movie descriptions, we pre-generate summaries for all movies using the following prompt:

### B.2 Genre User Request Generation

For generating user requests focused on specific genres, we use the following prompt:

### B.3 Open-Ended User Request Generation

For generating more complex, open-ended user requests, we use the following prompt:

### B.4 Distilled Model User Request Generation

For generating diverse training data for the distilled model, we use the following prompt:

### B.5 Feed-level LLM Judge Prompt

For evaluating how well a set of recommendations matches a user’s request, we use the following prompt:

## Appendix C Extracting Scores from LLM

To obtain more calibrated granular scores from LLM judgements, we follow the methodology of Williams et al. (2024). Rather than directly requesting a single rating on a scale of 1-5, we leverage the model’s underlying probability distribution. Specifically, we extract the logprobs for tokens “1” through “5”, normalize them into a probability distribution $P(r)$, and compute the expected rating as $\sum_{r=1}^{5}r\cdot P(r)$. To verify model comprehension, we ensure the total probability mass on these five tokens exceeds 0.9.

### C.1 Computational Efficiency and Quality of Distillation

Before considering the results of the experiments described in [Section 4](https://arxiv.org/html/2510.12742v1#S4), we discuss the efficiency and quality of our trained distilled model.

Figure: Figure 7: Computational efficiency: the distilled approach fares far better than the direct approach at scale. The distilled approach requires many orders of magnitude less computational resources than the direct approach for generating recommendations. Moreover, the speed-up factor for the distilled approach grows both with the number of candidate items and concurrent user requests. With $10^{7}$ items and $1000$ concurrent user requests, we estimate that the distilled approach would likely be at least $10^{8}$ times more efficient than direct LLM scoring. All evaluations were done on a single NVIDIA RTX A6000 GPU.
Refer to caption: x5.png

Distilled Model Computational Efficiency. We benchmarked the computational efficiency of the distilled approach compared to the direct LLM scoring approach. As seen in [Figure 7](https://arxiv.org/html/2510.12742v1#A3.F7), the computational resources required for the direct approach grow linearly with the number of candidate items and the number of concurrent user requests. Given the high starting cost even for small item sets and a single user, it is clearly infeasible to deploy at scale without massive computational resources and parallelization. While we expect the distilled approach will also grow linearly at large scales, it’s far lower starting cost makes it much more applicable to real-world recommendation scenarios.

## Appendix D Reachability Experiment Details

Figure: Figure 8: Screenshot of Letterboxd controls interface showing the traditional filters (genre and decade) that we compare against CTRL-Rec in our reachability experiments. We do not allow users to order by rating or popularity, as we automatically order movies using the engagement-based recommender system.
Refer to caption: figures/letterboxd_controls.png

### D.1 Technical Details of Filter Selection

Genre Selection. We analyze the target feed’s genre distribution by tallying how often each genre appears across the top-10 movies, then sort genres by frequency. We construct filter combinations by incrementally adding the most common genres (e.g., Action → Action+Adventure → Action+Adventure+War), using conjunctive (AND) filters where movies must satisfy all specified genres.

Decade Integration. We identify the three most frequent decades in the target feed and test each genre combination both alone and with each decade individually. For example, ”Action+Adventure” would be tested as: (1) Action+Adventure, (2) Action+Adventure+1990s, (3) Action+Adventure+2000s, etc. We avoid combining decades since ”1990s AND 2000s” would return no results.

### D.2 Training Data Categories

Our training data spans 10 categories of user preference requests designed to capture diverse ways users express movie preferences:

- 1.
One-time: Immediate, situational requests for specific moments (e.g., ”need something short”, ”want to cry”)
- 2.
Long-term: Persistent preferences and standing rules (e.g., core values, content boundaries)
- 3.
Aspirational: Requests for personal growth and cultural enrichment (e.g., ”help me develop appreciation for…”)
- 4.
Changing: Evolving tastes transitioning from old to new preferences (e.g., ”I used to like X, now I want Y”)
- 5.
Ambiguous: Poetic, impressionistic requests using metaphor and mood (e.g., ”something gentle”)
- 6.
Similarity-based: Comparative requests referencing specific films (e.g., ”like Blade Runner but comedic”)
- 7.
Smart-filtering: Highly specific requests requiring deep content knowledge (e.g., cinematography techniques)
- 8.
Smart-filtering-easy: Clear, specific requests in simple language (e.g., plot elements, character types)
- 9.
Logical-filtering: Precise logical filtering with boolean operators (e.g., ”Horror from 1990s BUT NOT zombies”)
- 10.
Refinement: Adjusting recommendations relative to recent patterns (e.g., ”more diverse than lately”)

## Appendix E User study details

Participants. We conducted a human study with 19 Letterboxd users to evaluate the effectiveness of CTRL-Rec in a movie recommendation task leveraging users’ own Letterboxd data. We recruited participants through social media posts, university Slack channels, and random sampling of personal connections that met the study criteria (18+ years old, have 10+ movie ratings on Letterboxd). Participant demographics are shown in Fig. [9](https://arxiv.org/html/2510.12742v1#A5.F9). 6 participants identified as female and 13 as male. The mean participant age was $27.0\pm 4.3$ (min: 21, max: 36). Most participants watched 3–5 movies per month on average. Our study was approved by our institution’s IRB and informed consent was obtained from all participants.

Figure: Figure 9: Distribution of age, gender, and average number of movies watched per month among Letterboxd users in our human study.
Refer to caption: x6.png

Setup. Our study had a within-subjects design with two cunterbalanced conditions: (a) CTRL-Rec(^8^88We used a $w_{\text{control}}=0.995$ for the trade-off between engagement and stated preferences ([Equation 2](https://arxiv.org/html/2510.12742v1#S3.E2)).) + genre and decade filters, (b) genre and decade filters only. We generated an initial set of recommendations using engagement-based recommender feed based on their ratings using the recommender in [Section 4.2](https://arxiv.org/html/2510.12742v1#S4.SS2). Participants were then given 8 minutes per condition to use the interface to find ideal movie recommendations. For each recommended movie, we provided two buttons participants could use to indicate their engagement to the movie: “Interested” if they were interested in watching the movie or “Watched” if they had already seen it. After each condition, participants answered Likert scale survey questions (on a 7-point scale) to rate their experience with the system. After both conditions, participants participated in a brief, semi-structured exit interview about their experiences across both conditions. Figures [10](https://arxiv.org/html/2510.12742v1#A5.F10)-[13](https://arxiv.org/html/2510.12742v1#A5.F13) show screenshots of our custom user study interface.

$p$-values and $q$-values. We performed Wilcoxon signed-rank tests to compute raw $p$-values for each measure. To account for multiple comparisons, we applied the Benjamini-Krieger-Yekutieli (BKY) two-stage procedure (Benjamini et al., 2006), following the approach described by Anderson (2008). The BKY method provides more precise control of the false discovery rate (FDR) compared to more conservative methods such as Bonferroni correction. For each measure, we report the corresponding $q$-value, which indicates the minimum FDR at which the result is considered significant at the $\alpha=0.05$ level. [Table 1](https://arxiv.org/html/2510.12742v1#A5.T1) presents the $p$-values and $q$-values for all measures. All measures with $p<0.05$ are also significant at an FDR threshold of $q<0.05$ (specifically, $q\leq 0.039$). Significant results are highlighted in green.

**Table 1: Raw $p$-values and $q$-values for all measures. All results that are significant with $p<0.05$ are also significant at a FDR threshold of $q=0.039$ (highlighted in green).**
| Measure tested | $p$-value | $q$-value |
| --- | --- | --- |
| \rowcolorgreen!20 Satisfaction in recommendations | 0.046 | 0.039 |
| \rowcolorgreen!20 Controllability of recommender | 0.046 | 0.039 |
| \rowcolorgreen!20 Expressivity of recommender | 0.0002 | 0.001 |
| Effort required for quality recommendations | 0.634 | 0.268 |
| \rowcolorgreen!20 Value of filters | 0.002 | 0.006 |
| Ratio of “liked” movies to total recommended | 0.825 | 0.308 |
| \rowcolorgreen!20 Ratio of “watched” movies to total recommended | 0.016 | 0.027 |

Survey questions. We asked users six Likert survey questions, where participants could pick from 1 (Strongly Disagree) to 7 (Strongly Agree). The following five were asked after each condition:

- 1.
Controllability. “It was easy to control my recommendations”
- 2.
Expressivity. “This system allowed me to articulate preferences in an expressive way.”
- 3.
Effort. “I had to put in a lot of effort to find recommendations I like.”
- 4.
Satisfaction. “I am satisfied with my recommendations.”
- 5.
Value of filters. “I found the genre and/or decade filter valuable.”

Finally, at the end of both conditions, one with the natural language controls and one without, we asked participants how valuable they thought having the natural language controls were.

- 6.
Value of natural language. “I found the natural language controls to be valuable.”

Figure: Figure 10: User study, consent form.
Refer to caption: figures/user_study/consent_form.png

Figure: Figure 11: User study, movie import page.
Refer to caption: figures/user_study/movie_import.png

Figure: Figure 12: User study, CTRl-Rec condition.
Refer to caption: figures/user_study/language_arm.png

Figure: Figure 13: User study, filters condition.
Refer to caption: figures/user_study/genre-arm.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See Weakness.
soundness: 3
strengths: 1. CTRL-Rec pioneers the distillation of open-ended natural-language requests into a lightweight dual-encoder that can directly steer large-scale retrieval, bridging conversational preference and traditional engagement-based recommenders.

2. The authors test the proposed method with large-scale reachability simulations, genre and subjective open-ended requests, and a within-subjects user study, offering both quantitative metrics and qualitative user feedback.
summary: This paper proposes CTRL-Rec, a framework that injects fine-grained, real-time natural-language control into traditional engagement-driven recommenders. By distilling LLM-simulated user-item preferences into a dual-encoder that scores requests against pre-computed item embeddings, it enables retrieval-stage steering with only one LLM forward pass per query.
weaknesses: 1. The paper lacks an overall system-level comparison: it shows that CTRL-Rec can steer recommendations, but never benchmarks its final top-k quality against standard engagement-optimized or LLM-based recommenders under the same task. Have the authors considered adding an overall comparison and comparing it with existing recommendation baselines?

2. It is unclear why Figure 2 discusses the **potential capability** of the proposed method in the main text. If the proposed method possesses this capability, the authors need to explicitly demonstrate it through experiments.

3. The paper only validated the algorithm in the movie recommendation scenario. Have the authors considered conducting experiments in more scenarios? This would better demonstrate the algorithm's generalization ability.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: Please see details in Strengths and Weaknesses.
soundness: 1
strengths: (1) Writing is easy to follow.

(2) The human study is valuable (although it would be better if the sample size can be larger).
summary: This study introduces a framework (CTRL-Rec) that integrates users' textual requests into traditional recommender systems. One of the major findings of the paper is that they find LLMs can be used to simulate how well items align with user requests, and how these simulated judgments can be distilled.
weaknesses: (1) The differences between the proposed method and conversational recommender systems are not explained clearly. From lines 151 to 161, the paper discussed the differences, but the reasons are not compelling. "Our work contrasts from other research on providing recommendations limited to a chat interface that do not affect traditional recommender systems" From this statement, it seems that one contribution is that the proposed method affects traditional recommender systems. However, why it is necessary to affect traditional recommender systems is not clear. Why can't conversional recommender systems achieve similar effect (having controllable recommendations; experimental comparisons should be made)? Filter-only baselines are simple which may not be sufficient to demonstrate the effectiveness of the proposed method.

(2) The novelty of the method which was claimed as scalable for integrating language-based controls into traditional recommender systems could be limited. It is common these days to use LLMs to generate either embeddings or natural language descriptions of users or items, and then train traditional recommender systems with them. This line of works normally only makes one LLM call per request as well. It is easy and straightforward to add user request into the original user side augmentation too.

(3) The validity of the reachability experiment can be strengthened. Two random users are selected to serve as the "scale". Would the original difference in preference between these two users affect the experimental results? It seems expected that it is easier to adjust recommendations for the two users who have similar preferences. I understand that if all compared baselines are facing the same set of testing data, they are still comparable. However, my concern is that whether the findings would be meaningful. For example, it is possible that model behaviors changed significantly when the testing data are difficult. Another concern is that such selection is not "authentic". In other words, the "request" was engineered instead of observed.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Refer to Weakness
soundness: 3
strengths: S1: **Scope**: This paper focuses on a crucial problem as well as a promising research direction in the recommendation field: incorporate nature language intention to interact realtime with recommend system.
S2: **Experiments**: The paper provides both empirical validation and a user study, demonstrating that LLM-based embeddings can capture human interests while maintaining engagement quality. The empirical experiment simulates how users adjust their recommendations to resemble those of a target user, comparing traditional genre/decade filters with the semantic embedding–based natural language control of CTRL-Rec. Furthermore, the authors conduct a human-in-the-loop simulation to validate the effectiveness and usability of their approach.
summary: This paper focuses on leveraging natural language as an interface for directly controlling recommender systems. It proposes CTRL-Rec, a framework that incorporates a request-aware recommendation score—computed via the embedding similarity between user intentions and items—into conventional recommender models. To achieve computational efficiency, the authors distill LLM-generated preference judgments into embedding models. The motivation behind CTRL-Rec is that natural language can express users’ reflective intentions beyond those captured by engagement-optimizing algorithms. The authors conduct empirical experiments on the MovieLens dataset and a user study with Letterboxd participants, demonstrating that natural language requests can steer recommendation outcomes.
weaknesses: W1: Lack of Quantitative Evaluation of User Intention Embeddings.
The paper does not provide a clear quantitative analysis of how well the learned embeddings capture user intentions. Most experiments rely on heuristic or qualitative observations, making it difficult to assess whether the model truly encodes intention-related semantics beyond simple content similarity.

W2: Methodological Validation
The experimental validation does not fully support the claimed methodological contributions, as no ablation or baseline comparison is provided. CTRL-Rec comprises two core components: (1) the Request-aware Recommender Scoring mechanism and (2) the distilled embedding approach for efficiency. However, the first component’s effectiveness has already been validated in prior work [1,2,3]. Moreover, intention-based studies [3] have demonstrated that advanced language representations can already capture user preferences and refine recommendations. Therefore, it is crucial to include an ablation on distill approach or baseline comparison [1,2,3] to isolate the contribution of the distillation mechanism and to demonstrate its empirical efficiency and necessity.

W3: Lack of Experimental Transparency
The experimental section lacks sufficient detail for replication. The authors do not release implementation details, such as the specific embedding model architectures, training objectives, or hyperparameters used. The absence of released code or model specifications makes it difficult to reproduce results or verify design choices.

[1] Bi, Keping, Qingyao Ai, and W. Bruce Croft. "A transformer-based embedding model for personalized product search." Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020.

[2] Ai, Qingyao, et al. "Learning a hierarchical embedding model for personalized product search." Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2017.

[3] Sheng, Leheng, et al. "Language representations can be what recommenders need: Findings and potentials." arXiv preprint arXiv:2407.05441 (2024).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How much performance degradation does CTRL‑Rec exhibit compared to directly using an LLM for filtering?

2. Does this approach fail to handle cold-start items and rely on domain knowledge introduced by the pretrained LLM?
soundness: 3
strengths: 1. At deployment, CTRL-Rec only requires a single LLM embedding computation per user request (instead of repeated queries for massive items), enabling real-time control of recommendations.

2. Unlike traditional structured controls (e.g., genre toggles), CTRL-Rec supports free-form natural language requests (e.g., "movies like Dune but with more humor" or "gentle content for relaxation"), covering diverse needs such as abstract "vibes," group preferences, and logical filtering.
summary: The paper proposes CTRL-Rec, a method integrating natural language controls into traditional recommender systems to solve users’ lack of fine-grained control. It uses LLMs to simulate user-item approval based on language requests during training, distills these judgments into an efficient dual-encoder model, and requires only one LLM embedding computation per request for real-time deployment.
weaknesses: 1. The evaluation is only conducted in the movie recommendation domain (using MovieLens and Letterboxd data), serving as a proof-of-concept but not verifying effectiveness in more complex scenarios (e.g., real-time social media feeds or large-scale e-commerce catalogs) where efficiency gains are more critical .

2. The MovieLens dataset mainly includes well-known movies that LLMs likely encountered during training. CTRL-Rec is not directly tested for generalization to truly novel items (a common limitation in retrieval systems but still a gap for practical deployment) .


Decision: Reject