Embedding-Aligned Language Models
Authors: Guy Tennenholtz
Yinlam Chow
ChihWei Hsu
Lior Shani
Yi Liang
Craig Boutilier
Venue: NeurIPS 2024 poster (2024)
Keywords: language models
reinforcement learning
embedding spaces
Primary area: generative_models
TL;DR: An RL-driven method for guiding LLMs to aligns with objectives defined in a latent embedding space. We demonstrate its effectiveness on surfacing content gaps.

Abstract: We propose a novel approach for training large language models (LLMs) to adhere to objectives defined within a latent embedding space. Our method leverages reinforcement learning (RL), treating a pre-trained LLM as an environment. Our embedding-aligned guided language (EAGLE) agent is trained to iteratively steer the LLM's generation towards optimal regions of the latent embedding space, w.r.t. some predefined criterion. We demonstrate the effectiveness of the EAGLE agent using the MovieLens 25M and Amazon Review datasets to surface content gaps that satisfy latent user demand. We also demonstrate the benefit of using an optimal design of a state-dependent action set to improve EAGLE's efficiency. Our work paves the way for controlled and grounded text generation using LLMs, ensuring consistency with domain-specific knowledge and data representations.

Embedding-Aligned Language Models
Guy TennenholtzI; Yinlam Chowl, Chih-War Hsu", Lior Shani", Ethan Liang', Craig Boutilier
Google Research, Google Deepmind
Abstract
We propose novel approach for training large language models (LLMs) adhere to objectives defined within a latent embedding space. Our method leverages reinforcement learning (RL), treating a pre-trained LLM an environment. Our embedding- guided language (EAGLE) agent is trained to iteratively steer the LLM's generation towards optimal regions of the latent embedding space, w.r.t some predefined criterion. We demonstrate the effectiveness of the EAGLEE agent using the Movielens 25M and Amazon Review datasets to surface content gaps that satisfy latent user demand. We also demonstrate the benefit of using an optimal design of state-dependent action set to improve EAGLE's efficiency. Our work paves the way for controlled and grounded text generation using LLMs, ensuring consistency with domain-specific knowledge and data representations.
Introduction
Large language models (LLMs) such Gemini [Team al. 2023 and GPT (Achiam et al. 2023
have revolutionized the field of natural language processing, achieving remarkable success in text generation, translation, comprehension, as well as expert-level performance on challenging tasks (e.g., exams, coding). However, effectively applying (or fine-tuning) LLMs to Domainn-specipe tasks often requires considerable domain knowledge and labeled human data [Jeong al. [2023 Ouyang et al. 2022 Ziegler et al.) [019]. Fortunately, in many cases, domain knowledge is already captured and encoded within latent embedding spaces.
Latent embeddings are continuous vectors, ubiquitous in fields such as recommender systems Hansen al., 2020], reinforcement learning (RL) (Nabati al., 2023 Pertsch al. 2021], and image classification Girdhar et al. [2233 Radford et al. [2021]. They offer a powerful means to represent entities, concepts, and relationships within a specific domain. For example, recommender systems. embeddings of items and users encapsulate information about preferences and behavior [Zhao et al.) 2023), embeddings of images can capture their content and style [Radford et al., 2021] while embeddings of scientific articles can represent their research area and findings [aaher Harikandehi ct al. 2023). The utility latent embedding lies its underlying compact representation of entities, concepts relationships, and the associated metrics, allowing one to construct simpler, more efficient models or induce control over various processes [Arvanitidis et al. 2018 2021 Radford et al., 2021 Tennenholtz and Mannor 2022 Importantly, latent embeddings are often pre-computed, and can thus serve as readily available rich source of domain knowledge.
This leads to the key question: can we leverage latent embeddings to better control and guide LLM generation? In this paper, we present novel framework which accomplishes this by exploiting latent embedding spaces to define an objective function for an LLM in iterative RL-drriee process.
As example, consider the challenge of assisting content creators in generating valuable content within a recommender ecosystem (e.g., YouTube, Reddit, Spotify) [Boutilier et al.) 2024]. An
CCorrespodeence to: auytenn@gmail.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

important aspect of this problem includes identifying and surfacing of content gaps, i..., identifying hypothetical content which could potentially drive value for users, and subsequently, describing it to creators. Latent embeddings offer an effective way of defining a content gap. Informally, content gap is hypothetical (i.e., non-existing) content item, corresponding to some point in a latent embedding space for which: (1) no content currently exists, implying an unexplored area within the existing content landscape, and (2) adding this content would improve overall system welfare (i.e., drive net positive value for both users and creators). While the latent embedding representation facilitates identification of content gaps, describing or surfacing this hypothetical continuous vector to its potential creator requires that move beyond the latent representation. The content gap embedding must be translated into a description that communicates its essence to a creator in a clear and actionable manner, requiring some way of interpreting the latent embedding.
In this paper, we address the general problem of aligning L.MSs with latent embedding spaces. We address this by working directly with both the embedding space and the generated outputs of an LLM. We formulate an RL--driven framework which iteratively steers an LLM towards regions of an embedding space deemed optimal according to some predefined criterion (e.g., content gap). To do so, we use a language- based agent to guide the LLM by modifying a textual representation of an entity. We then embed the resulting description into the latent embedding space to assess its quality w.r.t. our predefined criterion. Finally, we use this feedback to guide the LLM's next modification, steering it closer to optimal regions of the latent space.
Our contributions are as follows: (1) We propose an seieeedddnn-Aiigneee Guided LanguagE (EAGLE) agent, which aligns an L.M with latent embedding space: (2) We provide a comprehensive framework for novel content creation using LLMss, formulating a new technique for designing exploratory and high quality action sets by leveraging the power of LLMs, latent embeddings, and G-optimal design (Zhu et al., 2022]; (3) We validate the effectiveness of our approach on the MovieLens 25M dataset [Harper and Konstan 2015), aligning creation to behavioral movie and user embeddings. Our results are evaluated by human raters demonstrating EAGLE's ability to guide an LLM to generate text that is both creative and consistent with the domain's 's data representations
2 Preliminaries and Related Work
Large Language Models (LLMs). An LLM S As maps sequence of tokens probabilities over sequences of tokens. Transformers [Vaswani et al. 2017] are often used to train such LLMs over vast amounts of data, predicting the next token 2i
sequence given the preceding tokens
(x1, 1). by minimizing the cross-entropy loss. Pre-trained LLMs often vary in size, with
larger models having greater capabilities (Achiam et al. 2023 Team et al. 2023 Touvron al.
2023]: c.g., Llama 2 Touvron et al. 2023 models have 7B, 13B, and 70B parameter variants.
Embedding Language Models (ELMs). An embedding language model (ELM) [Tenenhollz et al., 2024] combines textual prompts with vector representations of domain knowledge. It uses adapter layers connect embedding inputs text-only pre-trained LLM. By training the model these combined inputs, ELM generates language that reflects the information contained in the domain embedding vectors, even for hypothetical entities [Teneenholtz et al.) [024].
Reinforcement Learning (RL). A Markov decision process (MDP) is a tuple AS P.r, 7), where S is the state space, A is the action space, P : S x A + As is the transition function, r: x A R is the reward function, H is the horizon, and y E [o, 1] is the discount factor. An agent S + A is a stationary stochastic policy mapping states to a distribution over actions
is its expected discounted return starting at s. The objective in RL is to compute an optimal policy, which maximizes value over some initial state distribution Vo, i.e., -² E arg max,, Eegruy [V* (sa))-
Its value at state 8, (*())= = P..,. Σn' 75r(s1, at) 80
Policy gradient (PG) methods are class of RL algorithms that directly optimize the policy to maximize expected return [Schulman et al. 2017 [Sutton et al.) 1999]. In I.I.M tasks, PG often uses reference policy Net as anchor regularize the RL. task (e.g., stay close Ter)) This is the
prevalent approach in reinforcement learning from human feedback (RLHF, Ouyang et al. 2022) ). In
Further experiments on the Amazon review dataset NN et al. 2019 are provided in Appendix
We limit ourselves to stationary policies for simplicity.

---

Table 1: Glossary
X
Ambient space (e.g., descriptions of movies)
d: x R+
Ambient metric
2
Lateni embedding space
dz 2 2 R
Latent metric
A
Action space (e-g. changes to movie plot)
U: D my R
Unility function
Latent dimension
Ep X. 2
Latent encoder
D Dataset of entitics
net X + AA
Reference policy
Horizon
9: + AA
reference policy distriburion
this work we achieve this using KL-divergence loss over policy outputs; i.e.,
loss ( PG-loss(TT) aDDx (
(1)
Related Work. Generative models are powerful tools for creative generation of text, images, and
more [Achiam et al. 2023 Ho et al. 2020 [022] Team et al., 20233 Embedding representations
[Arvanitidis et al. 201 8 2021 || Oord et 2018| Yang et al. [201] have been applied in many fields,
including recommender systems Liang et al. [2018), healthcare [Rampášek al. 2019], neuroscience
[Keshtkaran and Pandarinath| 2019), I, genetics [Frazrr et al. 2021]. astronomy [Ravanbakhsh et al.,
2017]. and more [Gómez-Bonee Bombarelli et al. 2018 Kingma al. 2014 Thadani et al. 2023]
Aligning LLM generation with embedding spaces holds great potential. Our work constrains the generation process to work with predefined utility over an embedding space. Other methods have been proposed align language models to knowledge graphs [Bao al. 20166, safety constraints
[Ji et al. 2024| Yang et al. 2021]. I. and human preferences [Bakker et al. 20.2, Ouyang et al., [202]] Wang et 2004). Our approach can be generalized such tasks with suitable latent embedding space and utility specification.
The use of RL with human and Al feedback for L.M fine-tuning has been shown to induce significant
improvement in LLM capabilities [Lee et al. 2023 Ouyang et al. [202]. J. Embeediigg-drveen guidance
an additional form of feedback that can further improve creative LLM generation, and potentially
reduce hallucination Dhuliawala et al. 2023 Gunjal et al. 2024 et al. 2023
3 Problem Setup
We assume ambient entity space X, which represents the space of all possible entities (e.g., all movie plots, or animal photos). Since our setup is agnostic to the modality of the space X, we do not emphasize language entities in this section. We assume X' is equipped with a met- ric d: X x X + Rt, which may not be known. We also assume access to a latent embedding ED X + Z, which maps entities in X' to a latent embedding space Z 12. If ED is injec- tive, it induces a metric (z(2, w) = d( (ED (2), ED (w)) over Z. Otherwise, we define the metric d2(2, w) = supre (F)) (w) d(x,8) for 2 # w, and dz(z, 2) oF Let (Z, dz) be the embedding manifold. Though dz is not known, various methods exist estimate the underlying metric (e.g., see Appendix B on the use of Riemannian manifolds and pullback metrics of generative models to estimate d |Avanitidis al. 2018 2021)).
Given fixed set of entities D {x, E X1 v== of size N (i.e.. dataset), we define utility function U Z x D + I over embeddings on the manifold (Z, dz). Below we illustrate an example of utility function for quantifying the quality of content gap.
Example: Content Gaps. Consider utility function measuring if a hypothetical point z E Z is content gap in recommender ecosystem [Boutilier et al. [202]]. We might define U as:
U D)
Σ USer
U Uneer (z; D) +
Σ creator
U Ureatorr
(2; (z:
Udldz(Eb(z))2))
ZED
Users' Utility
Cocatons' Utility
Distance from existing content
Here D is the set of existing content items, and Usser, Uevator: Ua are the utility functions for users, creators, and distance, defined over the latent embedding space. That is, content gap should bring high utility to users and creators, while also meeting some latent demand (i.e., non-existing content). In Sec. we explicitly formulate this utility for behavioral latent embedding space trained over movie and user data.
*Other metric definitions are possible; we this definition for illustration only.
3

---

Description of
Pre-trained
prompl
EAGLE Agent
Description of real entity ar
Bo(a)
hypothetical entity
LLM
Bp(z)
Description changed entity 2;:
ELM
Ep(as)
En(Ps]
hypothetical embedding
After steps
Final description of novel entity z
En(zn)
Latent Embedding Space
Figure 1: illustration comparing creation of descriptions novel entities using F.M [Tenenholtz al. 2024) vs. EAGLE (ours). The latent embedding space 2 is illustrated as a complex surface on the right. Red points on the surface are illustrated as latent embeddings of existing entities. Black points are used to illustrate hypothetical (i.e.. non-existing) entities. In ELM, utility is maximized in latent embedding space identify an optimal point. This hypothetical embedding then decoded back to ambient space A' form description of the hypothetical entity. Conversely, the EAGLE agent utilizes highly capable pre-trained LLM as an environment to scarch for novel entities in ambient space X. EAGLE does not use decoder, but rather only requires an encoder Ep X' 1+ Z. More specifically, the EAGLE agent uses action prompts to change using the environment LLM. Every new changed entity then mapped back to the latent embedding space using the encoder Ep. utility function optimized by EAGLE agent through a reward signal stir the changes to areas of high utility in the latent embedding space. A final description is then returned after H steps.
existing entity
Optimality Criterion. Given the dataset of entities D {x; X3 embedding manifold (2, dz), a corresponding embedding function Ed X' + Z, and a utility function U :2 x + R, our goal is to find novel entity x² XD such that
x= E arg xEX
U(ED(=):D).
We emphasize that, while optimality is defined in the latent embedding space, our goal is to identify novel entity in the ambient space g..g.. an image description of the hypothetical). As such, is not enough to search for an optimal point in Z. Moreover, the solution is not necessarily unique.
4 Surfacing Optimal Entities Using LLMs
We now turn to generation novel entities in language space; that is, we assume C S is a subset of the space of all language sequences (e.g., plots of movies, descriptions of clothes). Nevertheless, our methods are applicable more broadly to other modalities including images, videos, audio, or even RL policies. In what follows describe methods of finding optimal entity x in Eq. (3) using either ELMs or RL. high-level illustration these methods is shown in in
4.1 Embedding Language Models (ELMs) as Decoders
ELMs Sec. 2) offer explicit method of finding optimal entity x" X. Indeed, if one train a decoder f + X which satisfies (z)) = 2, then x" can be determined by maximizing 2" arg max (2; D) (i.e., identifying an optimal latent embedding w.r.t. U') and returning x² = f(z*) ) (i.e., some entity corresponding to 2²). Such a decoder can be trained using a fixed dataset and an ELM architecture ('Tennehholtz et al. 2024].
However, this approach has major limitations. First, learning an ELM decoder / requires training an LLM to interpret embeddings, costly process which may negatively impact the expressive power of the L.M. Second, the manifold (2, dz) can be highly complex and non-linear. Moreover, the latent space metric dz may not be known. As such, latent embedding z" deemed optimal in latent

---

space, may be "off-manifold", and therefore not mapped to valid entity 2 E X. To mitigate this, one can choose to stay close to points in the data, using Euclidean distance to approximate d [Chen et al. 2019], or employing pullback metric defined by Riemannian manifold [Arvanitidis et al. 2018 2021 (see Appendix B] for further discussion). Finally, the embedding space Z may not contain enough semantic information to ensure that f generalizes well (e.g.. embeddings trained over behavioral interactions with items). Training a new embedding, e.g., using Variational Auto-Encoders (VAEs) Yang et al. [2017] can mitigate this problem; yet. this would require training new latent embedding, a challenging problem in its own right [Vahdat and Kautz 2020). Furthermore, training a new latent representation may not be feasible (e.g., due to limited data). As such, using ELMs to train a decoder LLM over the embedding space may be impractical in many domains.
4.2 A Reinforcement Learning Perspective
As an alternative to ELMs, we formulate the optimization directly in the ambient space X (i.e., language space) using RL. Specifically, we leverage a highly capable pre-trained LLM (e.g., Gemini- Ultra [Team et al., 2023] or GPT-4 [Achiam et al., 2023 formulation. The environment LLM remains fixed, and is only used to generate new entities in X.
act as an environment for our RL
We formulate the problem using an MDP A, H)). where X' C S& is the state space comprising of all possible entities (in language), A C S is an action space, defined by language prompts for a pre-trained environment LLMP The action prompts are used to modify or steer an entity x E X. We elaborate on these actions later in this section. The transition function, P : X x A H Ax is induced by the environment LLM. Finally, the reward function is defined using the utility U such that (x()) = 0 for all L < H - 1, and - (x) = (z)(x>); [ While we do not define it explicitly, the MDP can also be endowed with contextual information such as information about nearby entities,
or a user for whom we wish to personalize creation.
At time 0 the process is initialized an existing entity 2; E D. For every < H. the agent sees state 3t E X' and selects an action a E A. The action is provided to the environment LLM as a language prompt, the environment LLM generates the next state x++1 X,, and the agent receives reward (2z)) We assume the environment LLM generates states in X' (i.e., feasible states).
4.3 Designing a State-Dependent Action Space
Designing an action space for an RL problem can be challenging (Guss et al. 2019 Kanervisto et al., 2020 Tennenholtz and Mannor 2019]. In our setting, actions are prompts used to change entities in X. As such, a good set of actions should be diverse and induce high utility states (entities).
LLMs for Action Creation. We exploit the generative power of LLMS to construct an exploratory set of actions for each state. Specifically, for each X, an LLM is used to construct set A(x) of K actions. For example, we might prompt an LLM to generate 100 ways to change a plot of a movie, the specification of an designed product. This ensures each existing x D is associated with a diverse set of actions that can modify it. To ensure coverage, K must be large enough to adequately search the state space (see (see 7 and Jappendix furrhhr discussion).
Reference Policy and G-Optimal Design. We use the (possibly large) set of actions to learn a reference policy for PG algorithm (sce 2), trained via supervised learning. Specifically, given the action space, we define a reference policy Tee((a) x) = Qa where E A(A(z)).
We consider three choices for distribution (1) uniform, i.e., Qa ; (2) myopic best next-step action, i.c., 4a 1 (a arg max W E [U (a')]]: and (3) myopic G-optimal design. which is a distribution over actions that minimizes a form of worse-case variance [Atwood] 1969 Kiefer and Wolfowitz 1960]. We define it formally as follows.
Definition 1 (G-optimal design). Let A(x) S A be given. Denote 2x(x) = Ex (a,a) [Ep(2'), A distribution ((x) E ((A()) is a myopic G-optimal design with approximation factor C 2 1 if
sup eEA(z)
| (E)tt))) CCn.
(4)
where [[((x)) = Eawg(=) [za (z)
[z)], and n is the dimension of the latent embedding space Z.
'e use the term pre- trained LLM to refer already trained LLM such as capable Gemini or GPT
model. We do not refer here to the specific stage in LLM training known as "pre-training".
"More generally, intermediate rewards can be provided at every iteration. We do not use such rewards here as we focus only on the final generated outcome.
5

---

Algorithm SaaeeedddinnAAiiinee Guided LanguagE' (EAGLE) Agent
Environment LLM, dataset D, reward function r, encoder ED. PG algorithm PG-ALG 2: Generate K candidate actions using pre-trained environment LLM for every x E D.
1: Input:
3: Compute an approximate myopic G-optimal design ((x) for every A(x), x E D (Definition
4: Train reference policy Teer(- (z) = g(x) over candidate action data and G-optimal design.
5: Train PG-ALG with reference policy Tref, pre-trained environment LLM, encoder Ep, and re- ward r.
G-optimal designs have proven useful for contextual bandits with large action spaces, allowing one to achieve efficient regret guarantees with minimal assumptions [Zhu et al., 2022]. Notably, an optimal design uniform c-exploration strategy has been shown to produce an efficient algorithm. We implement such exploration using a G-optimal design-basee reference policy, A G-optimal design always exists whenever (z)))) (a))))))AAss ee.)(x)) compact, and C 1 Kiefer and Wolfowitz 1960 Zhu
at [022]. While it is generally NP-hard to compute [Grötschel al.] 2012], an approximate optimal design can be computed efficiently (Zhu et al. [022] (see Appendix C for specific implementation
details in our setup using uniform sampling).
4.4 SeteeddingAAlignee Guided LanguagE (EAGLE) Agent
We are now ready to describe our approach, the Simbedding-Alignen Guided LanguagE (EAGLE) agent, which is detailed [Algorithm I] An illustration comparing EAGLE to ELM is depicted in Fig. The EAGLE agent is trained using the pre-trained environment LLM. We first generate K actions using the dataset of existing entities DD and the pre-trained LLM. Then, using the encoder
ED X Z, we create an approximate myopic G-optimal design q over these actions. We use the G-optimal design to train an initial reference policy Teet to match the action distribution q. Finally, we
Tee as reference policy for PG algorithm, regularizing the loss as in Eg. (1) The PG algorithm is trained using the environment LLM and the encoder Ep to optimize a utility
5 Experiment Design
We detail our experiment design; specifically, the data generation process, the training process, and evaluation methods. Our work uses the MovieLens 25M dataset (Harper and Konstan [2013). which contains 25 million ratings (1 to 5) of 62,423 movies and 162,541 users. We also conduct experiments on the Amazon review dataset [Ni et al. 2019), whose results are provided in Apppendix I
5.1 Data Generation
Latent Embeddings. We create embeddings for both users and movies in the MovieLens dataset.
We consider behavioral embeddings, trained solely using user ratings (i.e., no explicit semantic information) via matrix factorization using weighted alternating least squares [Hu et al. 2008 A user w's predicted rating for movie m given by the dot product (žu, 2m) of their embeddings
State Space. We generate textual descriptions of movies and users in the Moviel.ens dataset using Gemini Ultra [Team et al., 2023). For movies, we prompt Gemini to generate a plot for the movie, list five reasons someone might like the movie, and list five reasons they might dislike it (we refer to these as the description of movie). To describe user u, we provide Gemini with the list of movies rated with their ratings, and prompt Gemini to create textual user profile describing the user's preferences using a structured template with the following categories: general, genre, director, and aesthetic preferences. Examples of generated movie descriptions, user profiles, and prompts are provided in Appendices D and H To test the consistency of the generated profile for user u, we train an encoder to map it to u's behavioral embedding (see (ppenddx D).
Action Space. We generate 100 candidate actions for each movie in the dataset. Specifically, generate 50 generic actions and 50 personalized actions for every movie. We do this by prompting Gemini Ultra with the plot of the movie, and the list of reasons someone might like or dislike
? "Other collaborative tiltering methods could be used, e.g., dual encoders Yang et al. 2020 Yi et al. 2019 but approach is agnostic to the precise method used generate behavioral embeddings.

---

and ask it to generate 50 actionable ways in which to change the movie. Similarly, to generate personalized actions, we also provide Gemini with textual user profile, and ask it to generate 50 changes that are personalized to that user. To increase action set diversity, we prompt the model to output actions in five categories: plot changes, character enhancements, visual and storytelling improvements, thematic exploration, and audience appeal adjustments. See Appendices C Fnd for the precise prompts and sample outputs.
and H
Notice that we rely on an LLM's ability (in our case, Gemini Ultra) to generate diverse actions that suffice to explore the ambient space of movies. Other domains may require further considerations to ensure action space diversity (see further discussion of this limitation in Appendix B).
5.2 Training
Training Task. We consider a specialized content gap task described in Sec. 3
We simplify somewhat by not accounting for creator utility, and only considering "local" content gaps around anchor items and specified users. We compute user utility for items in behavioral embedding space using dot products of user and movie embeddings. (Zu. 2m)- We ensure the generated movie is, in fact, a content gap by also accounting for distance to existing movies; specifically, fz distance (in embedding space) of the generated movie to its three nearest neighbors. Formally, given user u with embedding zu, and a target movie m. E 'D with embedding 2m = Ep (m), our utility is defined by
U(m; D) = (Zus Zm) + A
Σ m' (m)
1|2m
2m
(Content Gap Utility)
where NN(m) is the set of three nearest neighbors to m in D (w.r.t l distance in 2), and X >> O.
Encoder. We fine-tune Gemini Nano-2 LLM (3.25B parameters) [Team al. 2023 to serve as movie encoder ED. We train it using l2 regression loss to map movie descriptions (plot, reasons to like, and reasons to dislike) to their behavioral embeddings. We use ED to compute the utility U.
Reference Policy. We use Gemini Nano-2 to initialize a reference policy re We fine-tune the model using next-token prediction (cross-entropy loss) of actions as targets. We create three reference policies: (a) Uniform: We use the full set of 100 generated actions for each movie as targets. (b) Best next-step action: We first create transitions for each action in the dataset. Specifically, we use Gemini Ultra to act as an enviromment, and prompt it with each action to generate a new movie (plot and reasons to like or dislike). The new movie is then encoded using ED. and its utility computed. Finally, we create a dataset of the "best" actions based on the computed scores. (c) Approximate G-optimal design: We calculate the next step embeddings for each action, as before. We then select ten actions which are a solution to Eq. (4) The new dataset consists of ten "exploratory" actions per movie, which we use to train reference policy via supervised learning, as before. We refer to Appendix C for details on computing the optimal design.
RL Training. We use Gemini Ultra as our environment L.M.. It is prompted to generate a transition given the current movie description and the agent action (see (see Appendix Ffor a review of all prompts). We inject randomness by increasing the sampling temperature of the agent model and the environment LLM. Specific hyperparameters are given in Appendix
5.3 Evaluation Methods
We employ two evaluation methods; namely, raw utility scores and human feedback, comparing AAGLE-generated outputs with their initial anchors. We use a pool of 124 human raters to provide feedback on our generated results Raters are shown textual user profile together with descriptions of two movies: a ground truth (real) movie, and a variant generated using H = 5 steps of agent changes. Importantly, the two descriptions are ordered randomly, and raters do not know which the original or generated variant.
We quantify each rater's personal preferences by having them rate 10-20 movies from a set of 600 popular movies, and ask them to write a textual profile of their own preferences. To increase confidence in our human evaluation, raters evaluate each algorithm twice, once w.r.t. textual user
'Raters were paid contractors. They received their standard contracted wage, which is above the living wage in their country of employment.
7

---

Table Comparison ELM. supervised training, and EAGLE. test three distribution of reference policies:
uniform (i.e., random), optimistic (i.e., next-slep best action), and G-optimal design (Definition
Utility
Human Rater Evaluation
Experiment
U(z)
User Utility
Rater Utility
Distance Score
ELM [2024]
0.57
0.31
0.37
0.52
Uniform
0.49 0.02
0.59
0.62
0.44
Reference
Optimistic Action
0.58 1 0.02
0.62
0.6
0.34
Policy
G- G-Optimal Design
0.51 + 0.02
0.51
0.67
0.45
Uniform
0.65 + 0.03
0.69
0.64
0.43
EAGLE
Optimistic Action
0.69 + 0.04
0.71
0.69
0.25
(ours)
G-Optimal Design
0.74 + 0.03
0.76
0.74
0.47
Table 3: For EnvTrain {Gemini Pro, Gemini Ultra) } and EnvTest (Gemini Ultra, GPT-4), the experi- ment EnvTrain-tEnvTer shows results for training EAGLE on EnvTrain and testing EnvTest. All experiments use an EAGLE based on Gemini-Nan -ano and G-optimal design baseline.
Utility
Human Rater Evaluation
Experiment
U(2)
User Utility
Rater Utility
Distance
Score
Gemini Pro -> Ultra
0.75 1: 0.03
0.76
0.74
Gemini Ultra - Ultra
0.8 1 0.02
0.74
0.66
0.47
Gemini Pro - GPT-4
0.65 + 0.02
0.86
0.78
0.45
profile of a user from the MovieLens dataset, and again w.r.t. their own preference profile. Each rater asked series of questions to quantify: (1) User Urility, the quality of the generated movie w.r.t. user profile; (2) Rater Utility, its quality w.r.t. their personal preferences; and (3) Distance Score, the distance of the generated movie from the anchor (i.c., the rater scores how different the two movies are). We express our results as the percentage of raters who preferred the generated result to the original (values [0, 1)). See Appendix G for complete overview of rater questions.
6 Experiments
compares results of ELM, supervised training (i.e., the reference policy), and EAGLE (ours) using the three reference action distributions (see (Secs. 4.3) (522)) Specific hyperparameters are detailed in Appenndxx E We train ELM to map behavioral embeddings of existing movies to their descriptions. Given an input movie embedding, we search for nearby embedding point with optimal utility, and use ELM as decoder produce its description. We that EAGLE outperforms ELM by a large margin w.r.t. both user and rater utility. This suggests a limitation in the generalization capability of ELM, and a fundamental problem of moving "out of manifold" when searching for optimal point in embedding space. We discuss this further in [Appendix B Despite this, raters judge ELM to generate movie descriptions that are further from (more different than) the anchor.
Table 2
The reference policy performs best with optimistic actions (i.e., max next-step utility). Interestingly, even the randomized due to the choice of action space, which includes many personalized actions, providing strong initialization for the randomized policies. Moreover, EAGLE performs better with a G-optimal design baseline than with the uniform or optimistic action baselines. This suggests that G-optimal design helps search the state space more efficiently. This is also evident in the distance score, which increases with the use of G-optimal design.
and G-optimal design-perfore well. We believe this is
Environment Transfer. Table compares results for different training/inference setups to test transfer effectiveness. Specifically, we compare training with Gemini-Pa or Gemini-Ultee models, and testing (i.e., inference) with Gemini-Ultee G-optimal design baseline. Our results suggest that a Gemini-Pa training environment suffices to obtain high-qualip inference results, as training with Gemini-Ulte did not improve performance.
GPT-4 models. All results use EAGLE with

---

Table The effect changing the action space on EAGLE, showing (1) the default action space described in Sec. 5| (2) the default action space with personalized actions removed. (3) only macro actions (i.e., taking three actions at once), and (4) combined action space, with both the default actions as well as macro actions. All experiments G-optimal design strategy reference policy.
Utility
Human Rater Evaluation
Experiment
U(z)
User Utility
Rater Utility
Distance Score
Default Actions
0.75 + 0.03
0.76
0.74
0.47
Without Personalized
0.56 0.02
0.54
0.59
0.45
Macro Actions
0.71 0.02
0.67
0.32
Combined Actions
0.74 + 0.04
0.88
0.92
0.4
Interestingly, trained EAGLE agent tested on a fixed GPT-4 environment maintains (and sometimes increases) its level of performance, suggesting that EAGLE is robust to the LLM environment on which it acts. More broadly, it seems that can train an EAGLE agent using any suitably capable LLM, and then apply successfully in environments defined by different L.LMs.
Action Space. We test the effect of changing the resolution and personalization of the action space on an EAGLE agent with a G-optimal design baseline in Table 4 We first test the effect making personalized actions unavailable. As expected, we see significant degradation in performance, suggesting personalized actions are critical to performance.
Second, we construct a macro-actionn space comprising tuples (01, 42, a3 04, as) of five consecutive actions. Thus, instead of making a single change to a movie at each step. the environment LLM makes five changes at once (i.e., H = 1). We evaluate the macro action space by executing the five actions chosen by the trained EAGLE agent on the default action space, at once. We also construct a combined action space, that adds macro macro-actions to the original action space (rather than replacing it). We see that macro-actions degrade performance w.r.t. the original finer-grained changes, suggesting the LLM performs better when making smaller changes. We also find the combined action space significantly outperforms other baselines, suggesting value to equipping the agent with both low and high resolution actions in sequential setup.
EAGLE on Highly Rated Movies. Finally, we assess the conditional performance of the EAGLE agent over movies that are originally rated high. We find that, while EAGLE improves overall scores of anchor movies that are originally rated poorly (i.e., scores 1, 2), it does not perform as well on movies that are rated highly (e.g., scores 4, 5), sometimes even decreasing user utility. Specifically, conditioned on poorly rated movies, EAGLE achieves an average utility of 0.04 (an approximate 30% increase), whereas for highly rated movies, it achieves an average utility of 0.83 + 0.03 (an approximate 5% decrease). This result expected as MovieLens rating data is limited to ratings between 1 and 5, where rating of 5 is (perhaps, incorrectly) assumed to be optimal for the user. To address this, we might only suggest or create a new entity it has a higher utility than its anchor.
7 Creative Generation: Discussion and Limitations
ELM vs. EAGLE. Our work reveals two distinct methodologies for generating novel entities that adhere to an underlying embedding space: using ELMs as decoders and our proposed EAGLE agent. While ELM offers efficient optimization within the embedding space and, in theory, can reach an unconstrained optimal point, it faces challenges related to the computational cost of fine-tuning the decoder, unknown generalization errors, and difficulties in constraining optimization due to the unknown manifold metric. Conversely, EAGLE leverages the existing textual capabilities of LLMs, enabling more interpretable optimization and potentially requiring smaller agent model. However, EAGLE's coverage is inherently limited by the defined action space, and training can be computationally expensive due to its reliance LLM environment queries. We refer the reader to Appendix B for further discussion on these tradeoffs.
Action Space Design. A principal building block of EAGLE is defining and or designing an action space to change entities in X. Designing good action space is critical to ensure reachability

---

G-optimal design
Good
Bad
Blased
Reduced Blas
Coverage
Coverage
Coverage
An illustration of different forms of coverage and bias an action space. As actions are textual prompts, their corresponding embedding directions may either provide good coverage, or partial coverage of the underlying embedding manifold (e.g., there may be directions that are not covered by the generated actions). Moreover, actions may be uniformly biased toward specific directions. We accommodate for this using a G-optimal design which reduces this bias through set of exploratory actions in embedding space.
Figure 2:
of optimal novel entity. In this work we define methodology for designing such action space using LLMs and G-optimal design (Definition I).
Fig. 2 illustrates three examples of potential action spaces and their inherent bias in embedding space. Informally, an action space may allow for "better" coverage if can induce changes all direction on the embedding manifold 2. More generally, an action space should allow one to reach any point on
the embedding manifold (up to some ball of radius e) in at most iterations, starting from any point
illustrate in Fig. 2 our action space may be constrained to certain directions, thereby limiting our ability to cover the full span possible entities.
Ep (x: such that D. Indeed,
To increase the overall coverage of an action space one can take one of several measures, including: (1) generating large amount of candidate actions, (2) using experts enhance the action space with additional missing actions, or (3) changing the methods used to generate those actions (c.g., using different prompting strategies). While all of these methods could potentially increase the coverage of the action space, none of them would guarantee coverage of the embedding space. In fact, it may be that no action exists that would allow one to move in any direction of the embedding space. That is, the environment LLM is incapable of moving in a specific embedding direction (i.e., embedding direction cannot be mapped into an actionable text prompt). This particular disadvantage of EAGLE, and an advantage of ELM, as ELM is not constrained by movements in embedding space (see Appendix B
Apart from the coverage challenge of designing an action space, we also consider the bias of such an action space. If an action space biased, then a randomized policy executing those actions uniformly would be biased as well. To mitigate this bias effect we use a G-optimal design. The G-optimal design allows us identify set of & actions that would be best for exploring the action space in terms of overall utility, thereby mitigating the inherent bias for exploration.
Conclusion
This paper introduces EAGLE, a novel framework that leverages RL to align LLM generation with domain-specific objectives encoded in latent embedding spaces. Treating pre-trained LLM as an environment, EAGLE iteratively steers its output towards optimal regions of the embedding space based predefined criterion. Our experiments on the Movielens 25M dataset demonstrate the effectiveness of EAGLE in surfacing content gaps, hypothetical content items that satisfy latent user demand. The results highlight the benefit of incorporating state-dependent action set based on G-optimal design for enhanced efficiency. EAGLÈ's capacity to leverage the power of large pre-trained LLMs without requiring explicit decoders opens up possibilities for controlled and grounded text generation, ensuring consistency with domain knowledge and data representations.
Further research directions include: exploring different action space design methodologies beyond G-optimal design, generalizing EAGLE to diverse modalities like images, audio, and video, and applying EAGLE to other applications that require aligning L.L generation with specific domain knowledge or constraints. EAGLE's flexible framework holds promise for various tasks, including personalized content creation, targeted ad advertising, and controlled dialogue generation.
10

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.005: 087744, 2023.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In International Conference on Learning Representations, 2018.
Georgios Arvanitidis, Soren Hauberg, and Bernhard Schölkopf. Geometrically enriched latent spaces. In International Conference on Artificial Intelligence and Statistics, pages 631-639. PMLR, 2021.
Corwin L Atwood. Optimal and efficient designs of experiments. The Annals of Mathematical Statistics, pages 1570-1602, 1969.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell-Gillignaane Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances Neural Information Processing Systems, 15:38176-23 3818898819111 2022.
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. Constraint- question answering with knowledge graph. In Proceedings of COLING 2016, the 26th international conference on computational linguistics; technical papers, pages 2503-2514, 2016.
Craig Boutilier, Martin Mladenov, and Guy Tennenholtz. Recommender ecosystems: A mechanism design perspective on holistic modeling and optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20):22575 -25883, Mar. 2024. doi: 10. 1.00/aaai.v38i20) /tttp:://js. aaai org/ index. php/ AAAT//rtiicle/vvee///000
URL
Nutan Chen, Francesco Ferroni, Alexej Klushyn, Alexandros Paraschos, Justin Bayer, and Patrick van der Smagt. Fast approximate geodesics for deep generative models. In International Conference on Artificial Neural Networks, pages 554-566, 2019.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verffication reduces hallucination in large language models. arXiv preprint arXiv::309. 1495, 2023.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K Min, Kelly Brock, Yarin Gal, and Debora S Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, 599(7883)91-995, 2021.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15180-15190, 2023,
Rafael Gómez-Bone Bombarel,,,,,l,,i, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato. Benjamín Sánchez- Lengellng,, Dennis Sheberla. Jorge Aguilera- Ipallaaurpprrrauurrer. Timothy D Hirzel, Ryan P Adams, and Alán Espuru-Guzike Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-21) 276, 2018
Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial oplimization, volume 2. Springer Science & Business Media, 2012.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135 11135-18140 143,, 2024.
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manucla Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440 2019.
Casper Hansen, Christian Hansen, Lucas Maystre, Rishabh Mehrotra, Brian Brost, Federico Tomasi, and Mounia Lalmas. Contextual and sequential user embeddings for large-scale music recom- mendation. Proceedings of the 14th ACM Conference on Recommender Systems, pages 53-62, 2020.
11

---

F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis). 5(4):1-19, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8666 2022.
Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15-19, 2008, Pisa, Italy, pages 263-272, 2008
Jihwan Jeong, Yinlam Chow, Guy Tennenholtz, Chi Hsu, Azamat Tulepbergenov, Mohammad Ghavamzadeh, and Craig Boutilier. Factual and personalized recommendations using language models and reinforcement learning. arXiv preprint arXiv:2310.061 2023.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via numan-preferenee dataset. Advances in Neural Information Processing Systems, 36, 2024.
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki, Action space shaping in deep reinforce- ment learning. In 2020 IEEE conference on games (CoG), pages 479-486. IEEE, 2020.
Mohammad Reza Keshtkaran and Chethan Pandarinath. Enabling hyperparameter optimization in sequential autoencoders for spiking neural data. Advances in neural information processing systems, 32, 2019.
Jack Kiefer and Jacob Wolfowitz. The equivalence of two extremum problems. Canadian Journal of Mathematics, 12:363-366, 1960.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez. Rezende, and Max Welling, Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nic, and Ji-Rong Wen. Halueval: A large- scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449-6466 2023.
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 world wide web conference, pages 689-698, 2018.
Ofrr Nabati, Guy Tennenholtz, and Shie Mannor. Representation--diien reinforcement learning. In International Conference on Machine Learning, pages 25588-25603, PMLR, 2023.
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeled reviews and fine- ine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pages 188-197, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 0807
/ 03740, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
12

---

Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188 204. PMLR. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 8763. PMLR, 2021.
Ladislav Rampášek, Daniel Hidru, Petr Smirnov, Benjamin Haibe-Kains, and Anna Goldenberg, Dr. vae: improving drug response prediction via modeling of drug perturbation effects. Bioinformatics, 35(19):3743-3751, 2019.
Siamak Ravanbakhsh, Francois Lanusse, Rachel Mandelbaum, Jeff Schneider, and Barnabas Poezos. Enabling dark energy science with deep generative models of galaxy images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv: 1707.063777 2017.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Seyyed Reza Taher Harikandeh, Sadegh Aliakbary, and Soroush Taheri. An embedding approach for analyzing the evolution of research topics with a case study on computer science subdomains. Scientometrics, 128(3) 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Guy Tennenholtz and Shie Mannor. The natural language of actions. In International Conference on Machine Learning, pages 6196-6205. PMLR, 2019.
Guy Tennenholtz and Shie Mannor. Uncertainty estimation using riemannian model dynamics for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 19008-1902) 2022.
Guy Tennenholtz, Yinlam Chow, ChihWei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulepbergenov, Deepak Ramachandran, Martin Mladenov, and Craig Boutilier. Demystifying embedding spaces using large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttpp:://oenrreiieee nwwttorumttiiieeee Pit.fforrommdd--oooooo......
Nicole N Thadani, Sarah Gurev, Pascal Notin, Noor Youssef, Nathan J Rollins, Daniel Ritter, Chris Sander, Yarin Gal, and Debora Marks. Learning from prepandemic data to forecast viral escape. Nature, 622(7984):81 888-825 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.00 09288, 2023.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 19667-19679, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Jiashuo Wang, Haozhao Wang, Shichao Sun, and Wenjie Li. Aligning language models with human preferences via bayesian approach. Advances Neural Information Processing Systems, 36, 2024.
13

---

Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang. Taibai Xu, and Ed H Chi. Mixed negative sampling for learning two-tower neural networks in recommendations. In Proceedings of the Web Conference (WWW-20), pages 441-447, Taipei, 2020.
Tsung- Yen Yang. Michael Y Hu, Yinlam Chow, Peter J Ramadge, and Karthik Narasimhan. Safe rein- forcement learning with natural language constraints. Advances in Neural Information Processing Systems, 14:13794-13808, 2021.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. Improved variational autoencoders for text modeling using dilated convolutions. In International conference machine learning, pages 3881-3890. PMLR, 2017.
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bise saapling-bias-coreeee bors-corrected neural modeling for large corpus item recommendations, In Proceedings of the Thirteenth ACM Conference on Recommender Systems (RecSys19), pages 269-277, Copenhagen, 2019.
Xiangyu Zhao, Maolin Wang, Xinjian Zhao, Jiansheng Li, Shucheng Zhou, Dawei Yin, Qing Li, Jiliang Tang, and Ruocheng Guo. Embedding in recommender systems: A survey, arXiv preprint arXiv:2310.18605 2023.
Yinglun Zhu, Dylan J Foster, John Langford, and Paul Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference Machine Learning, pages 77428-27453. PMLR, 2022.
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint
2019.
14

---

A
Societal Impact
EAGLE, a tool capable of creating new content based on latent preferences, presents several societal implications. On one hand, EAGLE can enhance creativity and innovation by assisting content creators in exploring new ideas and generating novel content tailored to specific audiences. This can stimulate creative industries and lead to the development of more diverse and engaging content. Additionally, by identifying and surfacing content gaps, EAGLE can help address unmet needs and preferences within online platforms, leading to more inclusive and satisfying user experience. Furthermore, EAGLE facilitates the generation of personalized content tailored to individual users, potentially improving engagement and satisfaction.
However, the technology also carries potential risks. If trained on biased data, EAGLE could amplify existing societal biases and reinforce echo chambers, limiting exposure to diverse perspectives. The ability to tailor content to latent preferences could also misused to manipulate user behavior, leading to unintended consequences.
Mitigating these risks requires a multi-faceted approach. Ensuring data diversity and faimess during training is crucial to mitigate bias and promote inclusivity. Developing methods to make EAGLE's creision-making process transparent can help address concerns about manipulation and promote user trust. Finally, integrating fact-checking mechanisms and robust content moderation strategies can help prevent the spread of misinformation.
Developing and deploying EAGLE demands careful consideration of its potential societal impacts. Proactive measures mitigate risks and promote responsible use are crucial harnessing its potential benefits while minimizing potential harm.
B ELM or EAGLE: Trade-Offs and Limitations
As discussed in Sec. 4 ELM and EAGLE can be viewed as alternative approaches for solving similar problems involving embedding spaces. Particularly, when we care about generating entities which adhere to some underlying embedding space. While ELM assumes the existence of a decoder f: Z + X, EAGLE only requires access to an encoder ED assess the quality of any x E X. Nevertheless, as we ultimately care about generating non-existing entities x & D, we must ask ourselves- how do we find an optimal æ in a generalizable and robust manner. Below, we enumerate several of the trade-offss of using ELMs and EAGLE to solve this problem, focusing on three points: computational efficiency, realizability, and coverage. We summarize these trade-offs [Appendix B
Computational Efficiency. When considering computational efficiency of ELM compared to EA- GLE we account for training efficiency, inference efficiency, as well as the computational requirement to retrain the model over time (due to e.g., evolving data and metrics).
We find that ELM generally requires fine-tuning much larger model than the EAGLE agent. Particularly, ELM should be capable of generating a full description of any embedding point in Z. This would potentially require training / fine-tuning EAGLE requires generating action prompts, which are shorter in nature, and, as we find, can use much smaller models. Still, during training, EAGLE requires querying an environment LLM, which potentially uses many computational resources. Moreover, in our work we use 16 environment LLMs simultaneously train the EAGLE agent, requiring extensive compute during training. That said, the environment LLM can in practice be an API call to a service which, unlike ELM, does not require serving an LLM in-house.
high capacity model. On the other hand,
Finally, we note that ELM is more sensitive to changes in the dataset and metrics. As we see in Table 3. EAGLE is robust to changes in the environment LLM. More broadly, EAGLE uses action prompts to change existing entities, much of the underlying "ground work" is done by the environment LLM. Therefore, if the dataset and metric were to change, e.g., one would want to also personalize outputs to specific creators, add more context to the generation, or perhaps generate other attributes of the hypothetical entities, then EAGLE would merely need to change the prompt of the environment LLM, whereas ELM would require retraining the model over new data to account for these changes.
The computational aspect of using ELM vs. EAGLE thus depends on the use case, the availability of resources to train LLMs (or using API calls), and the evolution of the underlying dataset and metrics.
15

---

Table 5: Trade-off's between EAGLE and ELM
Method ELM
Advantages
Disadvantages
Efficient optimization in embed- ding space.
Computational efficiency of fine- tuning ELM.
Not constrained can theoretically reach optimal point in embedding space.
Unknown generalization error manifold metric for constraining optimization.
EAGLE
Leverages textual proficiency of existing LLMs.
Coverage is constrained by action space.
Interpretable optimization
Computational efficiency during training: must use LLM environ- ment.
Computational efficiency: agent can be implemented using a smaller model.
Realizability. A critical aspect when comparing ELM to EAGLE is the realizability of the underlying embedding manifold. Even ELM manages to learn perfect decoder f Z + X, there remains the question of identifying optimal point Z to decode. As we only have embedding points of real entities x E D, extrapolating to non-existing entities requires moving on the embedding manifold 2, which is generally unknown. If we were to simply search for point R" which maximizes utility, we may return a point that does not correspond to a real entity, and thus decoding would not be feasible. Moreover, when decoder is trained existing entity data D, one must also account for generalization errors when attempting to move out of distribution.
We propose two solution to account for this problem when using ELMs. The first, is to constrain the search in embedding space to be close to points in the data, that is, balls of radius epsilon around all embedding points in 2);
U (c((((z)) FED
This solution allows one confidently remain manifold, with good generalization capabilities, while still optimizing for better content. Nevertheless, it may still be limited, as certain areas of the embedding space may be easier decode than others.
Another approach to search over the manifold (Z, dz) is to use the geometry of a generative model [Arvnitiiiis et al. 2018 2021] and estimate the metric dz. Particularly, assuming the embedding space was trained by a stochastic generative model (such as a VAE), one can use the pull-back metric (defined by the decoder Jacobian) to define the geometry of the latent space. Specifically, for VAEs with a decoder of the form f(z) = μx() (2)
(22))e cc, such that μ Z + X, Ko : 2 + R. NO I),
this induces Reimannian sub-manifold, with an expected metric
E [
(J(M)
(J(T) (J
This has been shown produce improved interpolations of embeddings. which better align with the underlying geodesics of the manifold [Arvanitidis et al.] 20188] 2021]. Still, this approach is computationally expensive, it requires computing the Jacobian of the decoder, which, in case, consists of an LLM.
EAGLE achieves realizability of the embedding by construction. Assuming the environment LLM generate feasible descriptions in X, any such point readily maps to feasible embedding point. Moreover, training encoder EDD is often significantly casier task than training decoder. Indeed, we find our encoder generalizes well on the test set of movie descriptions. More importantly, using the encoder to map to the embedding manifold we ensure any generated point is realizable, overcoming this problem by "moving" in ambient space X' instead of latent embedding space 2. This, however, comes with significant limitation, i.e., coverage,
discuss next.
Coverage. One of the major limitations of working in ambient space X, as opposed to directly optimizing the embedding space 2 is the inherent coverage of actions in X. Particularly, EAGLE requires using action prompts to change an entity's 's description. This approach has several disad- vantages as (1) one must define the set of actions, (2) it is unclear the full embedding space can be
16

---

realized by textual action changes to existing entities, and (3) the action space may be highly biased to specific changes. Indeed, ELM by design not constrained by this problem (though it does have the realizability challenge we discussed above).
We attempt to overcome some of these challenges in our work by using LLM to generate large pool of action candidates. Then, to remove potential bias and increase coverage we use G-optimal design. Nevertheless, this process does not completely solve the problem. It may be that combined ELM-EAGLE approach would be the most beneficial solution to the coverage-realizabiiiy tradeoff.
C Action Set Design
In our work all actions were generated by a Gemini Ultra. The following prompt was used to generate the set of personalized actions (a similar prompt was used for non non-personalized actions):
Prompt for Generating Personalized Actions
You producer of movic. You will be description of movie and protile of someone who might vant watcch such movie. profile you will be to provide actionable ways change the movie.
Given movic description
lere example for the movie Inception (2010):
DEGIN EXAMPLE
It Example Plot:
people's dreams and steal their secrets, He hired businessman Saito Inception follows Dom Cobb, skilled thief who has the ahility secomplish this, Cobb assembles team specialists, including the anchiseet implant an idda. Ariadne, the forger Eames. the chemast Yusuf, and the point man Antur.
mind his competiter. Robert Fischer
The team enters Fischer's dreams through a multi layered prucess, each layer representing deeper level of his subcomsciuus. As they navigate the treacherous dream handscapes, they encounter resistance from Fischer's subcomsious prujections. Cubh also faces deceased Mal, munifests saboteur the dreams.
personal demons, including
In the despest dream layer. Cobb and Ariadne confront Fischer and successfully implan: the idea. However. Mal's projection attacks Cubb, forcing Ariadne to kill her. The team escupes the dream lay but Saito is left trapped in limbo, the despes. dream state. Cobb enters limbu rerieve Suito, promising him
reality.
retum real and the inception believed successful. Howeverr the ending leaves ambiguity as to Clob himself is still dreaming.
Ultimately, Cobb
A Why Some People May Like It:
emeetin and manipulating decoms Mind Bending Visuals: The fim's stunning special effects and imaginative dream sequences create a visually immersive experience.
Innovative Concept: The
incriguing premise.
Comples. Charasters: Cobb and his team are well developed and reluable, despise their unusuul professioon.
Thought Provoking Ihemes: Inception explores themes Levill, and Marion Cotillard deliver cubstandnng performances.
reality, and the power of mund.
Exceptional Cast Leunardo DiCaprio, Juseph
# Why Some People May Not Like It:
Convoluted Piot: The multi layered dream structure and exposition beavy dialogue be challenging follow.
LLck Emotional Depti: Some viewers may the chanetees lacking emotional resonasce.
Open- kinted Conclusion: The ambignous rading may leave some audience member feeling frestruted or unsatisfed..
Overly Technical: The film's 's locus on the mechanics dream manipulaton may boce confuse some viewers. lacoption feel
Lengthy Runtime: At 148
and demsading watch nSTOPs
* Exammpe User Profile:
-"Overrll Preferences"""
Prefers arthouse and foreign films, with strong emplasis European explure comples thenses and provoke thonght
Enjoys
Appreciates films with a strong visual style and artistic sensibilities
His vaned taste in films, ranging from classic masterpicces to contemporary indie productions.
"GGear Pr
Fajoys drmmas that delvve into exidentiaiim, haman relationships, and the complesities of life.
Appreciates comeddis with dark or abourdist tone.
Shows interest foreign language silms, particularly those from Italy, and Eastem Europe.
Desonstrates preference for with clements of funtayy surralism, and the magical.
"* Director Preforances**
strong affinity for directoes
their distincr visual styles and incorventional steytellingg directoes Ingonr Bergatan, Faderioa Fellini, Wong Kar-wai, and von Tries.
Appreciates directors who challenge cinematic nomss and push the boundaries of the medium.
Acsthetic Preferences
Enjoys that feature atmoepheric lighting. evocative cinematographyy and striking compositions.
Prefers films with delibestee pace that allows for reflection and contemppaaion. Appreciates films that use symbolism, metaphor, and suncal imagery comvy ideas and emotions.
"*Cther *e
Values filus that feature strong, camplex charattes with whom they can connect emationally.
Fajoys films that explore themes of alienation, identity and the aearch for meaning in life.
Piefers films offer unique uncorventioaal perspective the human experience.
(.))
17

---

Prompt for Generating Personalized Actions... Continued
Below are the action ourpus that may personlize the movee Inception (2010) to the example user peotile: "Plot Changes
Introduce subplot exploting ethical implications dream manipulation, mising questions the responsibliry consequences
altering someone''s subconscious.
Deepen the explocation of Call's past and the trautt he caries from Mal's death, making his motivations and inner conflict moce relatable. manipulated and sceks reveuge Cobb
Add
Saito realizes he thaeees who compete with s group. adding another layer complexity intrigue the plot. Explore the possibility dream sharing betwees claracters, allowing for more innovative and mind-bending dream sequences.
creating
and suspensetul narmitiv.
CCaracterr Enhancemeuts.**
Giive Ariade a more uctive role in the teann's mussion, showcasing hes intelligence and creativity in navigatingg the dream world. Devellop Fames' character revealing a hidden past that influences his motivations and actions within the dream layers. Cobb, Deepen characterization Fischer Jr. exploring chiklhood traumas how they have shaped his subconscious deferses. Fahunce Mal's presence as the dreams, making her a more threstening and unpredictalle force that challenges Cobb's sanity.
Introduce chameter who
guidasce
complexites dream manipalation.
"* Visual and Stoeytellnng improvements:**
Incorporate more experimenal visual techniques such as time-lapse photography or stop-mation animation, to enhance the surcaal and dreumlike atmosphere.
Uhilize different color palette and lighting
distinguish
the
layers creating
visual experience.
disoriem viewer reffect the fragmented Fmplly alon-motion and fist-forward effects to manipulse the perception of time within the dream sequences.
Introduce non-linear storttilling elements
dreams.
Add musical scoee that complenents the film's themes and enhances the cmntional impoct of key seenes.
Thersstic Explonatio::*
Expand on the theme of the power of the mind by exploning the potennal for dreams to coralyst for personal growth and transformo-
rion.
Delive desper into philocophical questions reality perception, challenging the aadience's uaderstanding the world they pesceive.
Explone the theme of grief and the ways in which people cope with los, using the dream world as metaphor for the subconscions mind.
Examine the role of memory shaping our identity and the cossequences of altering implasting memories.
Introduce spicitul element story, himting possibility of connection
dieunts
power
"Andience Appeal "djuximemt:
secessible wider sudience maintaining Provide odditional exposition context to help clarity the complex dream-maniullake concepts viewers who may them challenging. Include more oction sequecces and suspemseful moments to enhance the entertainment factor for viewens who prefer more fast fast-puced narraive. Add elements of humor levity bilance the film's hervy themes and procide sudience members with moments of relief.
overal
Create thoughtt-goowking marketing campaign esemanges discussices dchates film's themes enjoy intellectual simulation. ASTOP
appenllnng
MEND EXAMPLE ASTOPA
sore to We will now do the same for the movie (i movie title }/:
Below the plot the [I mowie
profile.
& Plot: 1( plat]] I]
A Why Some People May Like
[( reasons like }}
Why People
Like It:
[1 reusons_ to_ Jo_diskkke 1)
A User Profile: [[ ceer_prodiee 1)
Write 50 actionsbl prompts below to chingod the movie {( movic tifle !J to the user profile above.
Make these
pensoaalizdd the five casegories exumple: plot changes, character enbancements, visual and storytelling improvements, thematic exploration, and zudience appeal adjusiments.
You
Finish your answer with 4STOPW
Below action ourputs may personalize movie movie_tite 11 example user peofile: ouiput
Given the above prompt we generate 50 personalized actions for each movie in X. User profiles randomly sampled from the dataset. We use a similar prompt without personalization to generate additional 50 non-personalized actions. In total we have 100 actions for each movie.
To optimize the G-optimal design we first compute the next-state embedding of each action. To do this, prompt Gemini Ultra to create description of the next movie given the action prompt (see Appendix F for the environment prompt we use). Then, we use our encoder Ep to embed the next representation of the action. Then, to generate approximate G-optimal design we randomly subsample a subset actions (in our setting = 10) and compute We resample until the maximum norm is lower than
movie. We this embedding
the maximum norm as defined in Definition ]
18

---

Cn. In our work we use C' --=1 1. We find that at most 10 random samples were required to find an optimal design for most states, Given the optimal design, we use the action subset as targets for training a supervised model (to act as a reference policy).
The question of how many actions to generate is generally ill-defined, is the method of generation. In this work we generate actions such that the variance in transitions would allow for sufficient coverage. We find 100 actions be sufficient in this domain for efficient optimization of a G-optimal design. Other domains may require more actions, and potentially expert assistance in creating a high quality action set.
D User Profiles
The MovieLens dataset constitutes of ratings (1 to 5) of users to movies. To describe a user and construct a predictive model of their rating we construct user-movie behavioral embedding space. We do this using a collaborative filtering method via matrix m factorization using weighted alternating least squares [Hu et al. 2008). The underlying user embedding En and movie embedding 2m are then predictive of the rating through the inner product (Zu; Zm)
In this work, we personalize generative outputs to users. One way to do this is by injecting the user's preference information, encoded in its embedding, into the language model. This can be achieved using an ELM architecture, though other forms for injecting embedding are also possible. An alternative method to interpret user information through textual profiles. These textual profiles are textual descriptions of user's preferences which should capture most of the user's preferences to be predictive of their rating. Textual user profiles are inherently prone to errors, as they may not completely capture the full extent of user's preferences, as user embedding do. This is largely due to the fact that many preferences are latent, and perhaps difficult to articulate in words.
We create textual user profiles for the MovieLens dataset and validate they are aligned with the corresponding behavioral user embeddings. To generate the textual user profile we use a user's full list of history of ratings and prompt an LLM to generate a textual user profile under a specific template constraint. Below is the prompt we used to generate textual user profiles.
To test the quality of the generated textual user profiles we test their alignment with their corre- sponding user embeddings. To do this, we train encoder, mapping textual user profile to its corresponding user embedding. A high test score for encoding the user profiles suggests the textual user profiles contain enough information to be predictive of ratings (assuming user embeddings are good representations of user preferences). We train the encoder using MSE, and find the average lg error on held-out users is smaller than than the average & distance between nearest neighbors in the dataset. This suggests our textual user profiles indeed capture the information encoded in the corresponding user embeddings, and generalize to unseen user profiles.
Prompt for Generating User Profiles
Below example of list of movies rited The format 'movic_till tile (user_ nuerrrating_em of monie)): Begin Examppl
Palp Fiction (1994) i5.0 Three Coloes: Red (Tris couleurs: Rouge) (1994) (3.5), Three Colans Blue (Trois couleuss: Blou) (1993) (5.0), Underground (1985) (5.0), Singin' in the Rain (1 1952) (3.5), Diny Danuing (s.0), Delicatessen (1 (191) (3.5), Ran (1985) (3.5), Seventh Seal, The (Sjunde inseglet, huture
(1989) (2.5), Back the Alexander (1 (Fanny Alexander) (1982) (2.5), Neverlnding Story. (1984) (3.5), Nights of Cabina (Noti di Cabina, Le) (1957) (1.5). Tango (1998) (1.0), Sangossa Mamuscrips The (Rekopis zalezionn w Saragussie) (1965) (5.0), Run Lala Run (Lola rennt) (1998) (5.0), Black Cas, White Cat (Cma macka, bell macoc) (1998) (4.5), Good Morning, Vietaam (1957) (4.0). Idiots, The (Idhoterae) (1998) (S.0), Requiem for Deeam (2000) (5.0), the Mood Love (Fa 01, Chics Whissgers (Viskmingar och top) (1972) (3.0), Chocolat (1988) (4.0), Amelic (Fabuleux
(1957) (5.0), Bridge the River Kwai, The (1957) (4.0), (1951) Future III (1990) (2.5), Fanny
Gattaca (1957)
Bock
(2000) (5.0) Moulin Rouge (2001) (3.0), Night, The (Notis, (1960)
A'Andlli Poulsin, Le) (2001) (4.5). Strawberiis (Smutronsttlle (1957) (4.0), Piano Teache,, The (La puaniste) (2001) (0.5), Nageyquza (2002) (2.0), Teddy Beu (Mis) (1981) (5.0), Talk to Her (liable con Ella) (2002) (4.0). Hit the Bank (Vabank) (1981) Lord of the Rings: The Two Towers, The (2002) (4.0), City of God (Cidade Deus) (2002) (5.0) Sponish Apartmest. The tuuberg espagnole) (2002) Translation (2003) (5. Burbarian Invasions, Hulot's Holiday) (Vacances de Monsicur Halor, Lest (1953) (4.0). Strada, La (1950) (4.5), Pussion of the Christ The (2004) (2.0). Good bye, Lenin!
Finding Nemo
Pintte Caribbeun: (Les ansssoos burbares) (2003) (3.5), Hulet's Holiday (Mr
Curse
Black (2003) (3.5),
(2003) (3.5), Persona (1966) (3.5), Etemal Sasshine of the Spotless Mind (2004) (5.0). Noi the Albino (No albanka (2003) (4.9). Virgin Spring, The Oungfrukällan (1960) Silence, ('ysttadenn (1963) (3.0), Winter spegeli (1961) (2.5). Magician Doke Vin.. La (1960) (5.0), Dolls (2002) 0), Shrek (2004) (4.0), Hour of the Wolf (Vargt (1968) (3.55, Miracle of Marcelino, The (Murcelino pan vinoi (1955) (1.0), Swann in Love (Un amour de Swann) (1984) (3.5), Port of Shadows (Quai des brumes) (1938) (4.0), Motorcycle Diantes, The Rad Educcatinn mola oducación) (2004) (Unn long dimanche fiangallles) (2004) (3.0), 5x2 (2004) (3.5), Look Mc (Comme une insage) (2004) (5.0). a End Example
Winter.. (Nattardigddtema (1963) (2.5). Through Glaes Darkly (Sisom Spring (Dom yeorcum gacul gyeoul geurigo hom) (2003) (3.5).
(1.5). Spring, Summer.
(Diarios motocielesa
Tixi (2000) (3.), 2046 (2004) (4.5), Very Long Enggaeneent,
19

---

Prompt for Generating User Profiles... Continued
Hare an example profile for this user based on their list of rsed movies above: + Begin Profice Example
"* "Overall Preferences**
Eujoys a
classi comtemocuay filss, with slight preference
movses.
Favoes critically acclaimed films, often regarded as masterpecces.
Shows a clear biss towards visually striking and thought-poooisg fllms.
Preferences**
Eajoys films from varions genres, including drama, comedly, action, comance, and art-house. Demonnstrtes strong oppeeciation for foreign films, particularly from European countries. Enjoys films with strung of ummosphere and world building.
Lirrector *
His strong preference for directors known for their unique visual styles and strytellinng apprnaches. Korysntt Kiellowski, Queatin Tarantino. Appreciates directors known their ability tackle complex and chillenging subject matter.
Enjoys the works directars Stutley Kuhrick, lngmar
Acsthetic Preferences""
Values that feature visually stunning cinemalgrragy, lighting, color pulettes.
Fajays
strong sense of composition Prefers films with slow, deliberate pace that allows for costemplation of the visunls.
framing.
""thher #
Enjoys
and drama, referred "dramedy" "tragcomedy."
Values films that explore complex and challenging themes snch is existentialisns morality, and the homan conditions. Appreciates films that fentare strong and manced charaeter development.
Profile Example
We're interested in doing the same thing for differeut user. The user's list niee movies is given helow: [[ user_ratings 1)
Given new
movies, deseritte detail user's preferences.
oupur categories (like the example): Over-
all Prefermmcs, Genre Preferences, Director Profereecss, Aestheic Preferences, and Other Preferences.
Do NOT use movie title sames
NOT eatings output. as detailed as passible. This user profile should be usefal for undentanding what new movies recommend to them.
aipput
E Implementation Details
We explain training details of the reference policies and EAGLE agent, and provide hyperparameters used in training. EAGLE was trained on a Gemini Nano-2 language model (3.25B). which can be trained using an equivalent of one A100 GPU. In our work, we train EAGLE in a parallelized manner, on an equivalent of 16 A100 GPUs. We use Gemini Ultra API for the environment calls, which did not require local compute, though can be time consuming due to the long context length and decoding length of movie descriptions. Every instantiating of EAGLE was trained for 5-7 days.
E.1 Reference Policy Training
We used Gemini Nano-2 LLM (3.25B parameters) [Team et al. 2023 train our reference policy. We train the reference policy using supervised learning, via cross entropy loss with predicting the next token in the sequence.
Given the dataset of all 100 action candidates (per movie), we create three dataset variants:
1. Uniform: this dataset uses all 100 action candidates targets. Training the reference policy over these generates uniform distribution over all actions.
2. Optimistic: For each action in the dataset use the environment prompt (see Appendix F and a Gemini Ultra to generate a next movie description. We then encode this next movie description to the movie embedding space and predict its rating for specified user. We do this for every action. We then select the action in the dataset that achieves the maximal rating prediction and create filtered dataset of only "best" actions. We use these actions as targets in the same way as before to create the optimistic reference policy.
3. G-Optimal Design: As described in Appendix C we use the embeddings generated before (i.e., next state representation) as each action representation. We then randomly sample subsets of k: = 10 actions (without replacement) and calculate the matrix norm Defini [ion 1] We continue sampling until we find value that satisfies the condition DeDeinnition with C = 1. We use the filtered dataset of 10 actions per movie as targets and train reference policy in the same way above.
20

---

We provide hyperparameter details used for training the reference policy in Table below:
Table 6: Training hyperparameters for reference policy.
value
Training Steps
20000
Batch Size
1024
Learning Rate
2e-6
Dropout Probability
0.1
E.2
EAGLE Training
EAGLE was trained with each one of the reference policies. We use REINFORCE with value baseline to train EAGLE, with Generalized Advantage Estimation (GAE). Specific hyperparameters given Table 7 below. All our experiments (except for one in Table 3) are trained using Gemini- Pro as environment. We generate test results using Gemini-Ultra. To ensure stochasticity we applied temperature sampling to the agent and the environment.
Table 7: Training hyperparameters for EAGLE.
value
Training steps
30000
Reference policy KI. regularization (o)
0.1
Policy learning rate
le-5
Value learning rate
5e-6
Policy (agent) temperate Environment temperature
0.5
0.5
Horizon H
Discount y
21

---

Prompts
Below we provide the prompts used for the EAGLE agent (i.e., to generate actions), and the environ- ment LLM (i.e., Gemini Pro, or Ultra).
Agent Prompt
You presented a plot movic, well reasons someone might dislike movie.
You will also he presented with user peoflee presenting a certain nser's preferenees.
You will asked provide actionable prompt change the movie. You choose categories; plut churacter enhancements exploration, extegories change movie.
visual storytelingg improvements, specifle action from
appeal adjustments.
You must
Below the plot of movie:
ABFGIN. PLOO
[[ plot 1]
MEND PLOTT
AREGIN REASONS LIKE
[[ reasons to like) } AEND REASONS LIKE
Below are reusons why someone might dislake this moste:
ABEGIN REASONS TO DISLIKE
if reasons_to dislike TO_DIEE
Below profile of user: ADEGIN_ PROFILE
1f profike LEENDUUSR_PPPR PROFILE
We now change abowe's movie plat and also give cornesponding people might like and dislikk the movie. actionable procupt producer they can change beter align preferences. ADEGIN ACTIONeacioo-AEND AENAAACN JAATTONO
Action: soupput>
Environment Prompt
Below plot movic. Your make following [[ oction
AREGIN PL.OT
[[ plot |HENDD_PLOO
Hore are also ressons somoonic might like this movic:
ABEGIN REASONS LIKE
[[ rensons, to, like JEEND REASONS,
Here also ressons someone might dislike this movie:
ABEGIN REASONS DISLIKE
[( rensons, to dislike JEEND REASONS TO DISLIKE
make following change If action
Make plot notably different than the original movie plot accommodiate the above chunge.
The the
Flnish EEND
Plot after applying the change: [[ oction
ABEGIN PLOT
noutput plat-
why might
Finish REASONS
ADECIN REASONS TO LIKE
like-
Now wrile retsont why someone migh. dislike the new muvic, Use the same formal is above, Finish with #END, REASONS, TO DISLIKE.
ABEGIN REASONS. PO_LISE
soulput didikes
22

---

G
Rater Evaluation
Below we provide the format of the form provided human raters. In the following section provide particular qualitative results that were also presented to raters in this form.
Rater Form
"BBLOO OW ARE TWO MOVIE PLOTS, AND SOME CHARACTERISTICS OF THOSE MOVIES, READ THROUGI THE DESCRIPTIONS OF CAREFULLY. QUESTIONS AROUT TWO MOVIES PREFERENCES
BE
TTE FIRST MOVIE PLOT (MOVTE A):*
{( plot movie }}
"REASONS WIYY SOMBONE MIGUT LIKE IT:**
[[ ressons movie }}
*REASONS WITY SOMEONE MIGHT DISLIKE IT**
{( reasons dislike firs, movie 1)
D):*
(( plot of second movic 11
"REASONS SOMEONE
1( reasons like second movie H
[[ reasons dislike second movie
BELOW IS PROFILE OF USER:**
[( user profiee
QUESTIONS:
Which movic would piefer watch?
Which movie motches more of the user's preferences? (A, B)
gave
you gave
meing. highess maing)
The yoer give movie a suore between 5. What spore do you chink they gave the movie? (1 kowest raking, 2, 3, highest maing)
(strnngly dissagres, disagroe, agroe, strongly agroe)
6. would was watch movie (agroe, disagree)
lagree, disagree)
If you had choose one the movies, which movie would you personally prefer watch? (A, R)
existing movie (agrce, disagroc)
10. Movie is existin real movie disagree)
11. Which movie would be by widkr audienoe? (A, B)
We eleven questions to evaluate human raters. Many of the questions used validate the quality of the responses. For example, questions 1, 2, 3, 4 evaluate the same question (i.e., user utility) in different ways. In our results report average scores of question for "user utility". average scores of question for "distance score", and average scores average scores of question 8 for "rater utility".
When scoring rater utility we use the human rater's profile. The textual rater profile is constructed by the rater themselves, for which generate personalized EAGLE output.
23

---

H
Qualitative Results
Below we present some qualitative results of EAGLE generations. For example, provide the original movie description (plots, reasons like, and reasons to dislike), the textual user profile, the actions chosen by the EAGLE agent, and the final generated movie description.
H.1
Star Wars: Episode IV A New Hope (1977)
Original Movie: Plot
galaary for, for away, the Rehel Allinnee fights against the ryrenyy of the Gallatic Empire. When Princess Leis, lender of the Rehellion, captured Empire, she sends distres message Obi-Wan planer Tatonine.
Jedi Keight message into
Skywalker, young
boy on Lake, along with Obi-Wan, the smngglers Hoa Solo and Chewhaces, and the droid C-3P and R2-12, emitark on mission to rescue Leis and destroy the Empiro's ultimate wespon, the Desch Star face challengos from the Empir's forces, led by the evil Darth Vider. elimectic bottle lkke Force
torpedoes Death Sta's post, destroying saving the palaxy tynnny
Original Movie: Reasons to Like
Epic Scope: The film presents vast und imaginative galaxy filled diverse planets, ureatures, and civilizations.
Timeless Stoey: The classic here's jommey and the battle betwees good and evil resenate with andiences of all ages.
Groundhreaking Special Effeces: Star Wass revolutionzzed
effects Memorable Music: John Williams's is opic score perfectly captures the grandeur and emotion of the story.
creating visually among
Characters: Lake Skywalkar,
memorable
history.
Original Movie: Reasons to Dislike
Dated Effects:
While groundhreking the time, some the special effects dimensional lacking depth.
dated by today's standards.
Simple Churactens:
Predictable Pkee: The film follows fairly formalaic hero's journey which may be predictalle some andiences.
-Soo Pacing The film's pacing con he slow cestain sections, porticuliely y the fins half.
campy Campy Dialogue: Some the dislogue, particularly between C-3PO and R2 D2,
Textual User Profile
OOverall *references**
Enjoys mainstream family friendly entertainment
action, adventure, comedy
Prefens heuriwarming and inspirational stoties that feature strong moral themes.
Shows clear preference for films made after 1980, with particular fundness for movies from the 1980s and 1990s.
Preferences*
Enjoys action movies, particulatly those that feature high stakes scenazios and hernic characters.
Appreciates adventure films that
locations and involve
rely slapstick, physical humor, relusable characters.
Enjoys fantasy and science fictso films esprciaill these feonure imaginative workds and epic battles.
Appreciates historical dramas, particularly those set during wartime periods of social upheaval.
Enjoys family driendly animated films. both traditionll and computer- animated.
Director *e
No discemible preference for specific directors.
Acsthetic "
Piefers striighforraid visual emphasizes storytelling.
Enjoys action sequenees cell-cchorogrggheee visually excating. Appreciates films that feuture bright colors and opcimistic lighting.
""tther *
Enjoys films that celchrate traitional values, such as courage. loyalty, and determination.
Appreccitts fims that tcuture strong male pootagonists who overcome adversity und achieve succoss.
Fajoys filus explore postive uplifting Shows dislike for anhouse films and films that contain explicis or controvenial content.
redemption, forgiveness,
importance fumily.

---

EAGLE Actions
Use more dynamic comera anples and editing techniques create more dhillinng and engaging vicual exporience.
Use more effects
enlonce the rellssm
Introduce now chameter serves as mestce
providing him guidanae support journey. Introduce subplut invelsing the traning of Lake by new Jedi Masier who provides bim with advanced weaponry und tactics. Introduce subplot that explores the backstory of Lando Calrissian, making him complex and relatable antugomist.
Final Generated Movie: Plot
In distani galany umidsi coemic war, the Rebel Alliance wages a desperate battle against the oppressive Galactic Empire. Amidst the chaos, Princess Leia, valiant Rohel leader, taken exprive by Empir's formidable forces.
fate would have it Leix's distress signal intereepis the puth Luke Skywalker, a young man yeaminn for adventure on the burren, planei of TTacoine. Guided by the enigmaric ObtWann Kenohi, Lake entorks on periiou journey to reseue the princess.
Meanwhik, Lando Calrissian, a charismatic smuggler frinnd of Han Solo, Lando betrayed him the Empire in moment of desperation. Haunted by his actions, Lando now seeks redemption.
checkered past, findk himself caught in creadline of cunflict.
Lake and his companions uppeouch Cloud Lando's domain, they encounter resiiianee. Lando, tom between his loyalty the Empine and lingering aftection old must
Through series of flachtucks, the subplot explores Lando's backstory. We learn of his humble beginnings as a gumbler and his rise to prominence criminal underworid. We witsess his friendship with Hun Solo and the cirumssances that led their falling
As the confiict insensifies, Lando is forced to confront his past and the consequesees of his betrayal. He witnesses the kimpice's brutality first- hand and realises the uue nature of the light he has chosen to be part of.
between desire redenaption with vital information and helps them infittrtte Cloud City.
Empire,
ultimately chooses the Alliance. He provides
climactic battle. Lando faces off Darth Vader. Despite his formidable skills, Lando outmanched by the Sith Load. However, buys time Lake
companions escape.
Lando's 's redemption are adds layer of complexity to the conflict. making him a more relatable and symsptthetc antagonist. His straggte with and his ultimate sacrifice
reminder that those who have made mistakes can find redemption.
Final Generated Movie: Reasons to Like
Complex Antagonist: Lando Calrissian's 's backstory and redemption make him more multifaceted and relusable antagonist, adding depth the conflict.
Expanded Lore: Tee subplot draws inspiration
Wars Expanded Universe, eariching
connecting
canon.
Redemption Story: Lando's jouraey of redemption provides powerful message abour the possibility of second chances and the consequences of betrayal.
sompgle between loyalty redemption creates compelling moral Emotional Depch. The exploration of Lando's is post adds an emsticoal depth to the film, making the stakes of the conflict feel more personal.
keeps engaged.
Final Generated Movie: Reasons to Dislike
bockstory an unnecesary distraction main stoeyline. Canon Devintion: The subplex may deviate from established Star Wars casca regueding Lando's character and motivations, which could be jarting for some fanss
Unncoesaary Rockstory: Some may feel
Foens Lake: introduction of Lando's subplot could draw atteation from Lake's jonmey undermine his protaganist.
redemption may teel prodictable or forced, reducing impact Altered Dynamics: The sulplot could after the catilishedd dynamics herweon Han Solo, Lake, and ando, potentally disropring the chamistry berwezn the chometers.
Predictable Redemption:
H.2 Toy Story 2 (1999)
Original Movie: Plot
Toy Story follows adventures of Wandy, Ligheyerr, and other toys when Woody stolen by toy collecter named McWhhiggin.
and the gang set out on reseue mission to sive Windy and bring him hume. Along the way. they encounter new toys, including Jesse the cowgir), Bullaeye the horse, and Stinky Peta the prospector. Woody learns about his popular TV show character and must decide whather with Al retum Andy, original
Original Movie: Reasons to Like
Hearteaming Story: The film explares themes of friendship, loyalty, and the importanee home.
Lovable Characters: Woody, Buzz and the other soys are all charming and relatable.
Impressive Animation: The animation top-notch langll-ous-iou moments exciting action sequenees.
toys in believable way.
Humar Adventure:
Nostalgic Appeal: For those who grew up with the first Toy Story, this sequei provides a sutisfying contination of the characters' adventures.
25

---

Original Movie: Reasons to Dislike
Predictable Plot: Soms viewers may find the reseue mission storyline to be formulaac.
Sentimenia Momens:: The tilm can be overly sentimental times, which not uppeal to all audiences.
Filler Scenes: Some such
where the
feel like
there to the runtime.
Mocivation:
stealing Woody particularly compelling.
Interior the Original: While Toy Story 2 a good movie, is often seen not being as strong us the first film.
Textual User Profile
Enjoys mainstream and familyy cutertainmeur, with focus an action, adventare, and comedy fllss. Prefers hearsaaminng and inspirational stodies strong themes. preference
after particulaa
movies 1980
"GGeare Preferences
Enjoys movies, particularty
high stakes scenatios heroic characters.
Apprecintes adventure films that imisport viewers to esceis lncations and invelve spic queses.
Prefers comeddies that rely slapstick, physical hunor, and relitable characters.
Enjoys fantasy and
films, especially during Appreciates historcaa drmmas, particularly
imaginative workds and battles. periods social uphetvall
Enjoys family-frenndly animatedd filmss both traditional and computz-animated.
Thiredoe Preforencest4
No discemible proference for spacific directors.
Acsthetti "references
Prefers films with stmightfrruad visual style that emphasizes clear storstelling. Enjoys action sequences that are well-choreagraghhe and visually excating. Approciates that beight optimiticc lighting.
OOtte Preferences*
Fajoys celchrate tnditional
courage, loyalty, determination.
Appreciates that feature strong male protagonists who oveoome alversity and achieve succes.
Fnjoys films that explore themes redemption, forgiveness, and the importance of family.
Prefers films that have a positive and uplifting tone. films films
dislike
explicii controvenial
EAGLE Actions
Create marketing
that emphasizes the film's nosiaggic appeal targets families with young children. onliniry objects
that Make Al more charismatic und complex villain with backstory that explains his motivations for exploiting toys.
Touch upon
special power signifiance.
Use fushbucks and dream sequeues to explore Al's childhond and the ungins of his obsessiun. Make Al's panents more complex chancters exploring their own streggles with childhood and their reasons for noglecting their son.
26

---

Final Generated Movie: Plot
Toy Story 2:
The Collecter's as Obscssion
Rounite with Woody, Buzz and beloved toys from Andy's moom they embbrk on an adventore delies into psyche troubled ty vollector. When Wandy is stolen by Al McWhiggin, a charismatic yei obsessive tuy enthusiast, Buzze and the gang set out on rescue mission.
they infiltane Al's sprawling mansion, they monenter flsshhokk and dream sequenees sion. Through these glimpses into his past. we wimesss Al as a knely and neglected child who found solace in the companionship of his vininge toy collection.
the tragic hackstory behind obses-
One particularly dream sequence transports the loys luughter echoing through the emply house. However, the some ahrupily shifls as AT's parent enter the room. Unlike before, Al's purents are now portrayed as complex individunls with their own striglos.
childhood bedroom, they see a young Al playing his loys,
Al's father, mother, on the other hand, is overwhelmed by depression and anxiety, leaving her emotionally unavallable to her son.
and distant man,
grow up abusive houschold and never lcamed how express love affection.
As the toys watch in horror, AT's parent scold him for his childishness and order him away his teys. They dismiss his pleas for atten- tion alfection, leaving him feeling isolated alone. This truumatic experience leaves deep scur Al's psyche, triggoring desperate fill the emotional void with the love and attention he never received.
the deive deeper Al's past, they uncover that his obsession has spiraled out of control. He has spent years amassing his collee- tion, exploting the nostalgia vulnerability other collecton. With Woody, one-of---kind cuwbuy doll, clutches, believes fimally complete his toissed masterpiece.
Ammed with this new found understanding of Al's motivations, the must not only Woedy but also challenge the collector's unhealtht obsession. Through series of dazing escupes and heartfll confromations, they reveal the tree nature Al's pain and emptness that drives actions.
In climactic showdown, Al is furced to confront his pasl and the destructive path he has taken. With the help of his own neglected teys, Al realizes the his ways the impurtance genuine human connection
who remind him of the love and accoptunce he moterial possessiuns.
As Al's purenss wimess their son's tranffrmaatin, they too are forced to confront their ewa childhood traumas and the impact their actions have had on Al. Through process of reconcilation and healing, Al and his parenis begin their broken relationship.
With Al's redempttinn Woody and his friends return to Andy, forever changed by their encoumee with the complex and trubled world of toy collecting.
Final Generated Movie: Reasons to Like
Psycholugical Dept: The exploration Al's parents' childhood traumas adds a new layer of depth and complexity to the story, making it more emotionally resenant.
Realistic Charactess: Al's parents longer simply neglectful characters but poctrayed flawed and struggling individuals with their paintull posts.
Insught into Family Dynamics: The film provides valuable insights into the complex and often damaging effects of childhood negiect and intergener atiocall unumaa
Themes Redemption Healing: The reconciliation between Al parzuts message hope those who have deeply hurt
tind healing redemption.
Balanced Storytelling: The exploration of Al's family histury seamlessly integrated inso the main ploe, enhancing the narrative without detructing from the adventure and humar that made the original Toy Story beloved.
Final Generated Movie: Reasons to Dislike
Unnecesstry Complexity: The intodaction of Al's parests' hackstories could he soen as an unnceessary distraction from the main plat of rescuing Woody.
Detraction from Al's Character: The villain.
parents may detract from impuct of childhond experiences and make him relatable
Emsotional Overioad: The heavy themes of childhood seglect and family dysfunction could potestally overwhelm the lightheurted and whimsical tose of the Toy Story franchise.
The revelation of Al's parents' past their role his obsession may feel predictabble contived. Parental Guilt: Irippigg: The portayal of Al's parents victins of their own childhoods could perceived as an anampt to excuse or justify their neglect of their son.
Predictable Pixt
H.3 Forrest Gump (1994)
Original Movie: Plot
Alahama winsesses and participates Attercaa histary. From childhood boy with log braces to adulthood war hero, and sucoesstial businessmos, Forrests extrocddimayy life unfolds against the backdrop of the Vietzam War, the Civil Rights Movemaent, and the Watergute scandal.
Forrest Gump hearwarming simplo-mindal
significntt
27

---

Original Movie: Reasons to Like
"Hoatwarming Story." Pocrest's journey is boch inspirnng and emotionally resosant, loaving mnciences with sense of joy and opt.msm.
*EEcceticcal Performance by Tom Hanks:** Hanks delivers an unforgettable performance Forrest. capturing his innocence, determiaaioon, and
unwavering spirit.
"Nostzigic Soundrack"**
features locaic sangs
1950is
evoking
nostaigia
context.
"Historical Signilicunee** Forrest's life intersects with mujor bistocical evenss, otfering unique pensectivee on American hissory.
"Memorabie Characters:" In addition Forrest, the film features cast of memorable characters, including Bubba, Jenny, and Licuterant Dan.
Original Movie: Reasons to Dislike
*Fpisodic Semcctur: The episodic matore senimentality
"Sentimenal viewers may
manipulative.
disjeinted lacking nomraivv cahesion.
*Histtricaal Inaccuracies While the film is generally faithful to the historical events depicts, does take some creative liberties.
** "Oversimplifie Themes:* The film's themes love. destiny, and the American Dream disabiliiy
overly simplistic idealizeed by being problematic
"Cotroveniaal Depiction::* The film's pontrayal
Textual User Profile
OOverall Proferences*
Hligh-bulget. effects driven action and science fiction films Hilms with strong storyeellng and character development Films thought provoking undrar emotionally resomant
""Geare *references
Science fiction, particularly those with focus technology, space exploration, and philosophical themes well chocsographed fight seguences impressive special Superhero films purtiullrtl those that explure the complexities of heroism morality Crime (hrillers with inrricate plots and suspenseful pacing Animated films, both feature length and short corm.
especially
Director "
No discemible preferences specific directors
Acsthetic "references**
Visually films impressive special effeets and cinematography amooppesic some
Films that use music effectively to enhance the emotional impact
Other *
Hilms that explore themes ideutity, monality and the human condition
Films with strong sense of humer, particaluzly dark comedies satitical films feature strong female characters
EAGLE Actions
Provide noee hopeful and uplitiing ending, emplasizing Foreet's resilience and the enduring power of the human spicit. Include humarous momeats lightheurted balance film's themes provide audience members with reliet. action, adventure, homor, appealing target mudience's
Create keting compaign enplasizes
entertinment
escapism.
Add twat where one of Jack's former comrades turns out traitor, providing shocking betrayal and testing his loyalty.
Examine the consequences violenee
individuals relutionships.
28

---

Final Generated Movie: Plot
""Unleaseed: The Betrayal Within'*
lack Swift, former spacial foroes openative. strugpling to cope with the aftermuth of a violent mission that him traumatixed. Hounted by nighimares and Bashbacks, he isolates himself frum society, seeking solace in alcohol and self destruction.
day, Teck's is involved in a dingerous criminal enterprise. Eihan's is actions have put inmuvent lives at risk, and Sum believes Jack the only one who cun stop him.
conersde, resches
him. Sam
lack that Fitun, another member of their
gone migue and
Reluctanity, Jack agrees to help He teams with Asya, skilled hacker who has also been affected by Fihan's violence. Tagether, they consequences
emburk
track
shared past.
As lack delves, desper into the investigarion, he is foree to confront his own inser demons. The violence be witnessed and perpetrated dur- the mission has lell indelible mark on soul. strugles with guilt anger, and deep sense loss.
Along the was, Jack and Anya encounter victims of Eiha''s brutality. They witness firshand the devastating impact that violence has on indi viduall and relationships. Families are torn upart, lives are shatlered, and the cycle of violence continues perpetuate.
on Ethan, and Anya must navigate complex web harm but have also poisoned their once close bend. The trust they shared has been shatlered, leaving behind a bitter taste of betragal.
they
and deception.
actions
only physical
the climactic showdown, Jack fages Ethan. his former friend tumed enemy. The confeumttion ways destroyed lives.
only physical but also emotional.
Jack must confront comsqueeces
violence they both porticipated
As the dust settles, Jack emerges from the conflict furever changed. He has wimessed the devastating toll that violence takes on individuals and relationsips. and he is determined to break the cycle. With the help of Amya and Sum, he to delicate his life healing the wounds of the and preventing funther bloodsbed.
The film ends with message of hope and redemption. While the stans of violence may never fully heal, pussible to find puth forward. penceful world, can break free from chains the
acknoweedging the consqueenees of actions and working logether
Final Generated Movie: Reasons to Like
"TThought Provoking Themes:** Thee film explores the complex and devastating consequences of violence, examining its imspact on individuals, rela- tiosships, and society.
'"Relatable th Jack's stroggle tramma guilt relatable anyone who experienced wimessed violence, making film emotionally resonant.
**Compelling Narrativ:** The plot unfolds in compelling and suspenseful manner, keeping audiences invested in the characters and their joumey. **Rellitiic Depiction Violence.rr The does glodity violence instead dopicts realistic and unflincting way. highlighting its destrue-
tive nature.
**Message of Hope und Redempiin:** Despite the durkness the subject matter, the film ultimately offers a message of hope and redemption, showing that is possible to break the cycle of violence and find healing.
Final Generated Movie: Reasons to Dislike
"Henyy Subject Ma The film's focus on the consequenees of violence could be toe henvy depressing for some viewers.
*SSoow Pacing:** The fimm's is emphasis on character developement and esstiional exploration may result in slow that some viewers find unsatadyyngg
Action While the film does fearure more action action-packee experiznee.
oction sequences, primarily charader-diive story, which may disappcint viewes expecting
**Preditublle Fice en Some viewers may find the plat predictable or formulaic, lacking the element surpris or originality,
*"Unsatidyiing Resalation:" se The film's message of hope and redemption feel too idcalistc unscalistic some viewers, leaving them faeling
unresolved disappounted.
H.4
Ice Age (2002)
Original Movie: Plot
Toe Age follows the adventres ofa group of prohistoric animals during the beginning of the Tor Age. Manny, woelly mammohh, Sid sloc and Diego, siher-toothed riger, from an unlikely trin journyy to return human baby to its trite. Along the way, they encounter various chillenges and dangers, including group vengeful dodo treacherous ice cave, pack bungry walves.
Original Movie: Reasons to Like
Family-Hie Friendly Entertainent: The film mutable for audienees all ages, with its slapstick humor and heartwarming themes.
Lovabie Characters: Manny, Sid, and Diego are instantly likable and memorable charactess that uppeal to both children and adults. vivid detail.
time. bringing the prehistoric world so Funny Diulogue: dislogue wiry and eotertiningg featuring menocaable one-liners quotable momests.
Stunning Animation: The fimm's animation impressive
Educational Value: The film touches upen themes of fnienship, cocperation, and the importance of family.
29

---

Original Movie: Reasons to Dislike
Predictable Plat: The story follows a fairly formulaic structure, with few surpises or unexpected twists.
Shallow Characters: Whike the main tno lovable, the supporting characters undendevelpped and lack depth.
Overreliance Slapetik: Some view
find the film reliance physical comedy Asumatio:: Compared modem animated
repelltive.
Age may appezr Lack Emotional Resonance: The film's focus on humor and actio may prevent from connecting with audiences on deeper emotional level.
asimattio
impressive.
Textual User Profile
High-budger, effeets-driven action and science fiction fllms Films with sturytelling and character development
thougls-roooking andior emotionally resonnnt
""Geare Preferences**
technolooy. space exploration, philosophical themes Action, esperially films with well-cheenggaphee fight soguenerss and impressive special effects
fiction. particllarly
Superhero films, particulatly those that explore the complexitis of haroism and moraliy Cruse thrillers intrictte suspensetul pacing
Animsted both festure-length short-fors
Director "reforences**
preferences specific directors
Aesthetic *references**
Visually stunning films impressiee special effects and cinerstography Films with dack and anmosphatic tore
Films that use music effectively to enhance the emotional impact
Films that explore themes of identity, monilty, and the human condition
humerr particularly comedics sariical films Films that feature strong female characters
with
EAGLE Actions
unforgiving environment, showcasing Touch upon theme of search meaning and parpose in world that constiilyy changing and inpendictible.
upun
resilience adipcaility.
Fmplly more dysamic camom stlk to capture the eneryy and struglee of the churacters in the oold and unforgiving emvironment.
Incorporate slow mution jime lapse effects highlight the passage of time and the relentless of the cold.
Incorporste
angles ediing wochniques enhance
danger.
Final Generated Movie: Plot
Amide: the desolate wasteland of the Arttc Circk,, group intrepid expleum emburks perilous expedition search of lost civilization.
angle shots capture lowering ioebergs loom uver explorers, dwaring them amida unforgiving wilderness. As they tra- verse the treacherous terrain handheld camerus follow their every move, conveying the constani sense of danger and instabllity
During snowsturm, rapdd editing
ups capture disurientation panic blurs, immersing the salience in chace the explorers stnggle to find sholier.
comera spins
monolith. time-lapse sequence the slow When member of the uam Ealls into a crevasse. low low-angle shot from below captures their desperate stuggle to estupe. The camera tiks and conveying claustoohobii
they approuch lowering glacier, wide-angl shot reveals the sheer scale relestiess movement the glacies, highlighsing ever present
avalmnches.
their predicament.
Throghh the use of dynamic camera angles and editing techniques. the film immerees the audience in the explorers' perilous journey. The constant sense of movement and danger keeps viewers on the zige of their seats, they witness the ruw of nature and the indomitable spirit those who dare challenge
Final Generated Movie: Reasons to Like
Immenive Experience: The dynamic cumera, angles and editing techniques transpurt viewers into the hear of the Artii wilderness, creating
thrillng experience,
danger and perii the explorers. Visual Stocyteiling: The innovative use of camera work and editing enhunces the narrative, conveying the chailenges and triumphs of the expedition in visually engaging
Hei glteend Danger:
low-angle shots, cameras, and rapid editing
Adrenal Kush:
suspense by
visuals keep
providing adrenaline-
experience.
Apprecciation Nature: The fila's stunning poctrayal of the Arctie landscape through wide-angle shoos and time- dapse--ppeeeeeeee sequences fonter an appreciation for the beauty and undorgiving power of natare.
30

---

Final Generated Movie: Reasons to Dislike
Exoessive Vicual Effeets: The constant use of dynamic camera angles and oliting tecinriques could parceived as overuficlming distracting.
detracting
Reduced Fariiicnal Inggect: The
spectacle may overdhadow emotiona deyth characters and joumey, pocentially dilining
their relatability.
viewers, making the film umplessont experimnc. Subclety: hecey-hhadded these tochniques may forced manipulative, undemininn authenticity the Reduced Immersion: While the visual effects aim enhance immersio, they could also create a sense detachment if they becume too intusivv or overwhelming.
Motion Sicknesa: The rapid camera movements and ediiing could indice mution sickness
H.S
Despicable Me (2010)
Original Movie: Plot
Despicable Me follows the story of Giu, supervillain who plans to sical the moon. However, his plans are thwartod when he encounters three orpltned girts, Margo, Edich, and Agues, who he adopts part of his scheme. As Gru spends time with the girls, he hegins to softea and question his villainous ways. Eventually, choose between his ambition
with fumily.
Original Movie: Reasons to Like
Heurtwammin Story: The film's central theme of redemption and the power of love is both heurtwarming and relatable.
Iummrrcus Chamaten:: Gru and his minions provide plenty of laughs with their quirky pensmalittes and slapstik humur.
Adorable Minions: The
Fumily Friendly Emertaimmnn: The gibberish speaking
highlight the film
characters family nights.
ages, making great Strong Voice Acting: Steve Carall and Jolie Andrews deliver measorable performances as Gau and Git's mothes, respectively.
Original Movie: Reasons to Dislike
Predictable Plkt: The film's follows fairly prodictable formula, which not viewers looking something original, ovenlone.
-vvesthe-Typ Top Homoer Some may find film's
Lack Depth: The film primarily focused on entertainmeni does not explore any deep or cumplex themnss.
Derivative Chamattene Geu's s chanater is raminiscent other unimated villains, such as Megamini and Syndrome.
"oak Villain: Vector, the film's antagonier, nor porticllatl threntening mencrsble.
Textual User Profile
"Overal *
HHigh-budee. driven
films Films with strong sturyselling and character development Films that are thoughe provuking undior emotionally resomant
Science fiction, porticularly thesr with focus technology, space exploration, and philosaphical themes impressive special effects that explore complexities heroism morality Crime thrillers with incricate plots supenseful pusing
Action, especially films with choreographed fight
Superhero films, punicularly
Animated films, both leature length and short corm.
No discemible preferences for specific dinectors
Visually sunning tilms with impressive special effects and cinematography Films with dark and amoopheric tone
effectively
impact
Other *e
explore identity, morality, human Films with strong sense humor, particularly dark comedies or satirical films
Hilms that feature strong female characters
EAGLE Actions
Introduce a message about the impostance accepting and embracing onc's tue self.
Use montage sequenees to show the passage of time and the developmeat Gru's relationship with the girls, making the film more engaging. relatable.
Use flashhucks dream soqueners provide insights
past motivations, making
Use flashhucks dream soquenees sparingly, tocusing on developing present-des day selationships Introduce a new chancter who rival Gra, but who also has hiddon connection to the girls, creating more comples and unpredictable dynamic.
keep narntive engaging.
31

---

Final Generated Movie: Plot
Despicable Me follows the presentdaay joumey of Gra, superviliain struggling balance his villainous amhitions with his newfound role as father figare.
As Gru plans to steal the moon, he encoumers fermidable sival named Dr: Nefato, brilliant but cocentric sciemisi who also harburs a connection Margo, Edith, and Agnes. Unbeknownst Gru, Dr. Nefario once the girls guandian, having nised them in laboritiory before stopted
Dr. Nefari's rivalry with Gru stems not only from their competing ambrttons but also from his desire to reckiim his bond with the girls. fiis knowdedge of their past and his gemuine affection for them creste complex and inprodicnible dynamic.
Gru and Dr. Naforio engage in a buille wits and gadgets, the girls find themselves tomn between their loyally Gru their growing curiosity about their former guardian. Dr. Nefarie's atlenpes to recomeet with them force Cinu confront his own foelings of inndequacy and question true motivations.
Meanwhile, the girl' memories their pasi begin to resurface. triggered by subtle cues und interactions with Dr. Nefario. As they piece togetherr fragmemts of their childhoud, they musi decade who they can trust and where their ure home lies.
confeomtaion between front the trush about his own past and the impact it had un his present. Ukimmtely, the choice lies with the girls as they decide who will be their true protector and guide.
not only ovencome
idemiity family, villainous ways but also about embracing his tree self and linding fultilment in the love and seceptance of others.
explores
power of redemption. tramsfommaiio only abundoning
Final Generated Movie: Reasons to Like
Complex Chanatens: intodaction
creates more complex unpredictable dynamic, adding depth Gru girla"
characsers.
Intriguing Plot The hidden connection between Dr. Nefario und the girls adds un intriguing layer to the ploi, kerping viewers engaged and guessing. affection the and their resurfacing memonies geovide additional emotional depth the Unpredictable Ouscome: girls'
Aimotional Dr. Netaria's
between
Nefario creates unpredictable outcome, heightening the suppese
encilement.
Exploration of Identry; The film explores the themes of identity and selt discovery as Gru confronts his past and the girls search for their true home.
Final Generated Movie: Reasons to Dislike
Contrived Plec: The hidden connection between Dr. Nefatio and the girls could be seca as contrived or forced. Underdeveloped Characters: The focus en the complex dy dynamic between Gru, Dr. Nefaro, and the girls could potentially lead to other characters being undeddevloped.
Lack Emmttional Impact: doa.
reliance flashbucks dream sequences
emotional impact of Gra's past and his transformo-
Predictable Confliet: The rivalry between Giuu and Dr. Nefadio feel prodictable reperitive, despite the addad complexit of their connection to the
Unresolved Questions: The may leave some questions girls' past Dr. Nefario's motivations unanswered, leading acampliteness.
H.6
Amazing Spider-Man, The (2012)
Original Movie: Plot
Peter Parker is shy and intelligent high school student who is bitee by genetically modified spider, granting him spider like abilities. Hle initially uses powers personal
after death his beloved Unele Hen, he become peotect the city Spider-Man. Peter's alter ego faces major challonge that threatens to destray New York City. Peter must confroat the Liand, his own personal demoms, and the respunsibilities that come with his newfound powers.
Connor, brilliant sciemistt, transferms himself the monstrous creature
Original Movie: Reasons to Like
who andienoes essily fentures thiillng and wellechorrogaahhs action scenes that showense SpidesMan's ahilities. Strong Villain: The Licard is menacing and fomiiduble imtgomist who provides significami challenge for Spider-Mun Man.
Relatible Protagonis:: Peter Parker relatable likable
Exciting Action Sequenees:
Lowe Interest: Gwes Stacy, Peter's love intersst is charming and female chanacter who add emotional layer the story. charismatic performance Parker/Spider Man.
Andrew Performance: Gariild
Original Movie: Reasons to Dislike
Rchasked Origin Story. The fila retells SpiderMan's origas story, which may feel recundast some viewers.
Lack Originaiiy: The film's plot and characoers are similar previous Spider Mas adaptations.
Sahjar CGi Some
CGI affects, partiulurty those
the Lizzed, can be distractnn unccalistic
Romantic Subploe: The romance between Peter Gwenn can forced underdeveloped times.
Predictable Ending: The tilm's climas and resolution may cousidered prodictable by some andiences.
32

---

Textual User Profile
"*Overall "references**
High-bulget, effecte-driven action science
Films with stren storytelling and churacter develupment Films that thought-roookingg andior emocionally resonatt
"*Geure Preferences*
Sccience fiction, particularly those with froms technology, exploration. and philosophical themes especially films with chereographel sequences impressive special
Superhero films, purtcularty those that explore complexities of heroism moralit Crise thrillers with ierfictte ploes and suppeneffll pacing
Animated films, both feature feature-legth and short-Comm
Director *reforencess*
No disormible prefenecces for specific dinectors
Acsthetic Preferences**
Visually stuninn films with impressive special effects and cinemalography daek atmospheric Films that use music effectively to enhance the emotional impact
**Other *
Films that explore themes identity, morality, und humun condition
Films with strung sense of humor, puticularly dark comedies satirical films Films feature temale characters
EAGLE Actions
Add musical score that complemests the film's fist paced and comaedic tone.
Create marketing campaign that emphasizes film's blend comedy, and emotional depth, appealing wide
marketing campaiga
enyliasizes
uniqee
action, cousedy, emotional dopth, appealing to
entrrtanment and thought provoking consent. Incorporate twist where Alex discovers that his tue destiny lies in differens field than simply fighting aliens, challenging his preconceived notions. Develop supporting
giving them unique persontliciss motivations that contribute overali ensemble.
33

---

Final Generated Movie: Plot
"*Assendance: A Cinematic Oiyssey of Action, Comedy, and Unexpected Desting Tex
Prepure younself for an unforgetabll cinematic experience with "Ascendance." film that acumlessly wenves logather heart-pounding action, side spliting comedy, and grofound emotional depth.
Meet ordinary college sudent whose forever changed when winesses mysteriooss meteorite cras. invesigates the crash site. Ales exposed to an unknown energy that grants him superhuman abiliies. The film's musical spore captures the swe and wender of Allx''s 's trnsfommatoon with an enigmatic and owe-inspiring theme.
Joining Alex extracedioary journyy are tions
cast of supporting characters, each with their unique personaliiess quicks, motiva-
"*Maa:s Alex's best friend and confidante, Mia brillsant and resourefful hacker who provides imalusble technical suppori and wilty bunter. Her unweverring loyalty Alex mutched by sarcustic humor, lightening the
situations.
**Dr. Emiiy Catter** renowned astrophysicist, Dr. Caner becomnes Alex's ss mentor and guide. Her scientific curiosity and compussion drive her to help Alex understand his abilities and the puiential implications of the alien threat.
*=Iak==* skeptical and pragmatic government agent, Lake iniiaally views Alex with suspicion. However, the evidence of an alion invasion mounts, reluctansly womms up Ales, lending his tactical expertise unwavering determinarion
="Zarr"" An enigmatic alien warnicr from distani planet, Zam arrives on Earth with a mission to protect bumanity from a common enemy. Her stoie dememmer and formidible fightng sk skills add new dimension to the conflict.
for pranks and personul amusement. However. his carefree autiude takes dramatic turn when formidable allen threat emerges from the stadows. The score inconsifies, transitioning to a pulsating thythm that underscores the thrilling action sequences as Alex and his allies face off against these estraterrstiial invaders.
fini, revels his new found powers, using
Amidst spectacle and adrenaline pumping action, "Ascendance" detily weaves moments levity. Mia's witty remarks, Jake's dry hu- mor and the absurd situations they find themselves in provide a welcome balance, ensuring that the film remains both exhilarating and entertaining The score seamlessly intorporaies comedic elements highlighting the film's unique blend of action and humor.
grapples with weight introspective and emotionally charged, mirroring his iner trmmil and the profound impact his choices have on his life and those around him.
stakes
and responsibility
to protect humanity, score becomes more
Through their interactions with Alex and each other. the suppurting churacters contribute to the overall ensemble and enrich the narralive. Mia's anwavering support helps stay grounded, while Caner's scientifc insights poovide rational perspective on exmmrdinary evens unfolding. Jake's ss skepticism challenges Alex's is assumptionn and forces him to confrom. the consequences of his actions. Zara's is allen perspective offers a unique undrrstanding of the conflict and the potencall for unity between diflerent species.
pivotal mocsent, Alex uncovers hidden aptitude diplomacy and negociation. Through interactions realizes that there is more to their preseuse thas meets the eye. The score shifts a mare contosplative and hopeful tose, hiating at the unexpessed twist that awaits.
invaders,
conflict intensifies, Alex faces peofound revelation, true destiny noe solely in vanquishing the alien threat but in hridging the gup between humanity and the extratenrestiaal visitons. The score swells with sense of purpose and decermination, underscoring Alex's newfound understanding of his rule.
stuut-si confrumaaions and thought grovoking momests, "Ascendance" explores themes of identity, sacrifice, and the scarch for meining a clantic world. The film's musical soore serves as in integral guide. enhancing the emotional resonnees of every seene.
Through
climactic showdows that will leave andiences breuthless, Alex uniizes both his superhuman abilities and his diplocnatic skills to unprecedented alliance betwees humunity and the silens, The score reaches an opic crescendo, capturing the trinngh hoge, unit, and the mnexpected destiny that Alex and his alles embrnce.
"Aseendance" is cisematic de foree that will coptivane viewers with unique blead action, camedy, and emotional depth. film will not only emestain but also provoke thought and ignite conversttion long after the credits roll. "Marketing Campaign:**
The marketing campaiga for "Ascendaace" will emphasize the film's unaxpected twist its exploration unconvantional destinies. Ads- rionally. highlight the divens cast of suppeting chameters their unique contributions the overall ensemble.
"argett Audiesce: ** Viewers who value both
action and thought provoking consent thas challenges expectations. "Slaann ii "Propare for Odyyey Action, Comelly, and the Unespected"
Thailess and Posters Highlight nim'ss 's thiiling action seguences, wirr humor, diverse
Final Generated Movie: Reasons to Like
"*Adrenaline Pumping Action 17 The film's sequenees expently choroographad enhanced by pulsating musical score that will keep
andiences the odge of their seats.
.**1 "1hhhgh-Omm-I and Comedy +* The comadic timing and wity dislogueare are perfectly complementellly by the score's playfisl and homorous enes, pooiding
welcome ballnce the tigh-stakes action.
-*mmocional Resommoee The film explores profoudd thomes of ideniiyy sucrifice, search meaning, supported by a mowing introspcctive
score will resonate andiences.
*"Immesive Soundrackk-e The musical score is an integral part of the storytellig, immening andiences in the world of the film and heighttnning the
emotional impoct of scene.
""Etnforgettbble Churacters: Alex and the supporting cast are relatable and charismatic, making their journey loch thrilling and amotionally enpaging.
*"Diverse Easemblee: The diverse
suppotiing charreteess each with their unique personslities, quiks, and mocivations, eurictes the narrstiv
and a&d depth to the overall experience.
Shnought-Prosikkig Themes:* e The film's exploration of identity responshility, and the haman condition will leave audiences pondering long after
credits
-niini Blend Gennes:s "Ascendince" sesmlessly weoves together action, comolyy and cmvotiona depth, creating cinemaric experience that both estertaining and thangt--rowokin
==Unexpected Twise:* The revebsion of Alex's destny adds surprising and satisfying layer to the story, challenging preconceived notions and
lewwing a lasing impact.
**Msssgee Hops and Unitt: The film's ultimate message of hope, unity, and the power of finding one's 's purpose will inspire and uplift sudiences.
34

---

Final Generated Movie: Reasons to Dislike
"Predictable Pioe Sone viewers lind the film's plcc to be formulaic predictable, lacking the elament of supvise
**Overlhhe Actionss aciinn sequences may be
autiences
grounded approuch.
""Jaring Shifles transitions hetween action, comolly enocional scenes may abrape jurring to viewers.
**Uninspired Dialogue:+e The film s dialogue may be considened cliched or lacking in depil, detracinng from the overall impoct.
**haallow Supponing Chumacten: ++ supporinn churacters
be fully developed relatable, making diflicali for andiences
with
level.
"Derivulivve Themes:*e The fimm exploration of idencity and suctifice may feel derivative or unctiginal somee viewens.
"Contrived Twis:** The revelation of Alex's tue desinn may feel contrvvdd or fonced, undermining the credibility of the story.
* "Unartifying Resclution: ** The film's onding noe provide soidtying resolution the conflicts and themos raised thruggout the story.
H.7 Dumb & Dumber (Dumb and Dumber) (1994)
Original Movie: Plot
Dumb Dumber mixadventures dim-witted Christmas and Harry emse-contry trip Aspen, Colondo, return money rightful May Sansoon. Along way, they encounter series of mishaps, including accidentally killing man, getting kidnapped, and being pussued by a pair of criminals.
they
Original Movie: Reasons to Like
top physical comedy will appeal of dialogue, such you selling Jim Carrey and Jeff Danizis: The perfocmanees of Jun Carrey and leff Daniets as Lioyd and Harry are hilarious and icanie. Simple and Fun: The tilm has simple peemisg that easy to follow and enjoy.
Stapstick Comedy; The
Quotable
Nostalgic Value: For those who grew the 1990s, Dumb and Dumber holds
their heurts.
Original Movie: Reasons to Dislike
Crude Humor: The humar often cnule und may offend somee viewers.
Predictable Plot: The plot is fairly prodictable and lacks any real surprises,
Chee-Dimesional Chameters:: Loyd and Harry essentially one-note claracters who never really develop. purely comedy Reperiiive Jokes: Some jokes Dumb and Dumber can become repotilive grating.
Depth:
explore any deeper
ideas.
Textual User Profile
Proference
Prefers popular and mainstream movies, particularly from the 1990s,
Enjoys a wide range of genres, but shows preference towands comedies and action films.
Tends favor movies that entertaining sowwd-plessngg and offer from reality.
Genre #e
Enjoys comodies, purtcularly slapstick, grees-out, out, and romuntic comedies.
Appreciates action movies with strong thrill,, suspense, special effects.
Enjoys
mysteries fominess for mmmily-frimdlly films, including animsted features and live-action comedies.
intrigiing plats
Shows interest science ficiio and fantasy movics, particularly those with epic storylines and impressive visual effects.
*references**
Does not seem have strong preferences specific directons.
Enjuys works by dinecsors who have belmed successful commencial films, such as Tim Burton, Joel Schumacher, and Peter Farrelly. balance enlectainment value emotional depth.
Appreciates directors
Aesthetic *references**
Prefers movies with bright and colerful visual style
Enjoys
strong spectacle and visual effects.
Appreciates movies catchy soundrracks upbeat music.
Other *
Enjoys that
and
muvies escupism
Appreciates movies that have nostalgic quality, reminding them films from their chikkhood adolescence.
Enjoys movies that teature strong chanattes and relatable starytänes.
EAGLE Actions
Hint at deeper menning behind Lloyd and Harry's journey, suggesting that their misadveatures may be metaphor for the challenges life. Develop the character of May Swanson, making ber more than plat device and giving her backstery and motvations of her own. Incorponnte sequenee highlights funniest
memorable Incorporite subylot berween Lloyd Mary, adding element lighthested comance story. Add musical score that complmeats the film comedic and action packed elemcnts eahancing the overall entertainment value.
misadventues.
35

---

Final Generated Movie: Plot
Dumb and Dumber sill follows the misudventures of two din-wited friends, Looyd Christmas and Harry Dunne. After losinng their jobs, they embark cross country trip Aspen, Colomado, return briefcase full money rightful owner. Mary Swanson.
Mary Swanson, unce a simple plot device, now emerges complex character. She successful businescoomon who lost her briefcase in small lown, facing alversty and discrimination goaund up
containing important financial documents. Her bacdotory reveals that she business world. Desermined prove hanslff she worked fimllssly huild her company
in the
As Looyd and Harry set out on their journey Mary becomes an active participant in the plot. She hines private investigatior to tnic down the bridfcase. her desperation to retriese driver by the fewr of losing everything sle has worked for Her pussuit intersects with I Lloyd and Harr's misadventures, creating a parallel narrative
depth
story.
Along their chaolic joumey, I loyd and Hanry eocumter a series of hiltrions mishups and memanble escounters. From sacidestally setting their fire getting cought snowstorm wearing only their underwear. their misadventures provide steady stream comedic gold. The musical seore enhances these comedic moments, lively uphest
acceniute the shsurdity their
Interspersed with their misadventures, Lloyd and Mary's 's pulls cross several times. Inisaally, Mary is amooyed by Lluyd's dim- wittedness and clumay behavior, However. they spend more time logether, she begins gradually chip away at ley exterior. musical score here shifts more rumantic tender tone, rellecting the growing connection between them.
different side him. Lloyd's unesvering optimism and kind-bearted nature
Meanwhile Loyd is smitten with Mary's imtelligence and drive. He finds himself' drww to her despitc their vasily different personalities Their interactions are filled with mix of humor and unexpected tendemess. The musical score complements this balance, blending comedic melodies with
As the journey progresses. Lleyd and Mary find themaelves working legoher to overvome the obstacles they face. Lloyd's uncomventional ideas and Harry's bizare antics prove be surprisingly helpful in wolving the mystery of the missing briefease. In tum. Mary's business scumen and resorccfflness help Lloyd Hurry navigate unfariliuz world Aspen. musical score intensifies during these action packed sequences, fass paoed and energesic thy thes that heighten the excilemens
Through their shared experiences. Lloyd and Mary develop gemane affection for each other. Their romantic subplot adds of the otherwise zany and unpredicable story. The musical score plays crucial role conveying emutional depch relationship ness with souring melodies and poignaat hammonis that evoke both laughter and tears.
Ultimaeely, Lieyd and Hary''s 's joumey becomes comical yet poignant reflection the absurdity and unpeedictability of life. while Mary's story serves an empowering cominder of strength resilience women in face adversity. And as dust settles Lleyd and Mary's unlikely romance blussoms, proving that even in the midss of chaos and misadventues, love can find a way. The musical score reaches its crescenda. celebrating the triumph of love and the enduring power of friendship.
Final Generated Movie: Reasons to Like
Slapstick Comedy; The film retains the over the top physical comedy that made the original classic.
Quuntable Lines: Dumb and Dumber still delivers memerable and quotable dinlogue that will stay with viewers.
Jum Carrey Jeff Daniels: performanes Jim Carrey and Jeft Duniels remain hilazious and iconic.
Ramantic Subplot The romantic subplet between Looyd and Mary adds new dimession sweetness
the story.
Stroog Female Character: Mary Swanson's 's developed character peovides role model for female empowerment and resilience.
Parallel Narraives: The inserowining starylines of Lieyd, Harry, and Mary create more complex and engaging narrative.
comedic gold, leaving viewers stitches, Unexpected Venderness: The mossens tenderness betwees Llayd Mary add suprissing doptis the film, making it both hearwaaming and hilarious.
Sequence: The montage hilaious misadventures provides cosecutrated
The film canveys message that even in the midst of chans and absurdity, love can biossom in uncxpected places. kahanced Enteraiimmet Value: The musical complements the comedic and action-packss elements, amplidying laughter and excitement thronghout film.
Message of Love:
Final Generated Movie: Reasons to Dislike
Crude Humar film's humor romains crnde and may sill offend viewers.
Predictable While the romattic supplor adds new element, overall plor may still predictable for souse.
Distracting Subplot: The comanic subplor could be scen as a distraction from the main comedic stoeyline.
feel forced unheieeable to some viewers original may be disappaintdd by addition of rosaantic subplet. Shallow Characters: Llayd and Harry zemain essentially dimensional characters, which may
Foreed Romance: The rimance between llyd and Mary
Dejurture Original: Fans
emotional impact.
Montage Sequence: While the montage proides comedic highlights. it may also Teel like rehash of old jokes for some viewers.
Intrasiv Musical Score: The musical score could potentialy overpower the dialogue distract from the comedic timing in certain scenes.
H.8
Dumbo (1941)
Original Movie: Plot
Dumbo young elephat with unmenally that make him target ridicule, One mother locked away peotecting him from formenters, leaving Dumbo and bearthrokes. help of Timethy Q. Mouse, Dumbo disconers that his cars allow him myy He becottee the stur of the circus, reuniting with his mother and proving that his differences are what make bim special.
36

---

Original Movie: Reasons to Like
Hearwaamingg Stocy: Dumbo's s jouney from ouicast to celebrated hero both uplifting and emotionally resonant
Charming Animation: The film's classi Disney unimation visually appealing and brings the characters to lise.
Memurable Charactens: Dumbo, Timethy, and the other circus znimals endearing and unforgetablle. importacce tamily.
Themes: explores themes accoptance. selt contidence, Nostalgic Value: Dumbo holds a special place in hearts of generations who grew up with it.
Original Movie: Reasons to Dislike
Short Runtime: unly 64 minutes, Dumbo feel too brief for
Sensbilities
film's depictions
may
modern
llakk Complexity: plot characters relarively simple, which may appeal viewers looking more sophistiatted story. Sentimenial Ending: The fil's happy ending may be seen as overly sacchurine or unrcalistic.
Limiied Animation: Compured Disney films. Dumbo's animation limited and
visually impressive.
Textual User Profile
Oveerall *references
Profers contemponry films, with focus those released in the 1990s,
purticllar altinity Tends favor movies with mainstream uppeal mass entettiinment value.
Enjoys wide variety genres,
comedy.
Appreciates films that both essertaining and thought grovoking.
#refermmess*
Einooss action movies featuring high octune thrills, explosions, and fight soquenees.
Appreciates cometties that are lighthearied, amusing, and sometimes satinicl.
Likes crime thrillers with cumplex plots, suspenseful twidss, and enigmatic churacters. explore imaginative concepts technological advancements. Appreciates fumily friendly films, purtcularty those that feature animated charasiers and heatwarrming stories.
Enjoys dction
Director "reforences** **
any strong preferences particular
Aesthetic Preferences**
Enjuys films with visually impressive special effects and action
Appreciates movics
memorable charatees quotable dialogue.
Prefers films with fast paced and engaging narrative that keeps the audience entertained.
"*Other "references"*
Enjoys
geares, Values filuss that feanure strong and charismatis lead performances.
or tiction
Appreciates movies that explore themes of hernism, lovalty, and redemption.
His soft nostalgie films from their childrood, particularly animated features and family comedies.
EAGLE Actions
Introduce a fomale elephant character who becomes Dumbo's friend and helps him overcome adversity. redemjtion.
Touch upon themes
hope, redesption. showing resilience determination overcoming challeages. Make the tormentors more complex. and multifaccted, showing their ewa strggles and motivations for mistreating Dumbo Include more slapeick and physical comedy to enhance the enterainment value for sudiences.
37

---

Final Generated Movie: Plot
Dumbo, the young circus clephear with unusually large cars, faces relentess torment from grou of follow performers, Howeser, unlike before, these tormentces simply heurless, Instend, they are driven by insecuritees struggles, which amplified series slapatick mishaps.
The ringleuler. Rex, is once-renoweed clown who has los, his spark. As be attempls sabotage Dumbu's performances, he finds himself shoes, tumbling buckets comical as his pratfalls draw luughier from the andience.
nipping
and getting cought
His desperation validarion
The streegmaan, Brano, tries to intimidatte Dunho with his imperssive physique, but his anempts are thwarte by mulfinnciooing weighttinn ing bur sends him Blying acress ring. clumsinces.
being overshalowed by Dumbo's strnngh overshadowed by the spectacle his
The acrobats, Amya Boris, and Cyril, determined to their superiocity Dumbo Uoowvvr. their elaborat stunts constanily chautic and hilurious compatiion
discupted by mistimed flips, alippery triperes, rogie unieycle Their resentment Dumbo of who can make the biggest blunder.
Amida: this slapetick chaos, Flle, fellow outizet, extends hand of friendhipp to Dumho Together, they devise plan expose the under- lying reasons behind mistreatment
incorporating tormenties' misadventures into show.
Through series of carefully orchestrsted performancos, Dumbo and Fila turn the tunmentons' peatfalls into a sounce of entertainment Rex's clumsy allempts juggle become jugzling with flying Bruno's weightlifing mishup becumes human cannonball stunt, and the acrobats' mishaps become gravity gravity-defying cumedy rostine.
In climactic confruntation, Dumbo confronts his tommenions with a water ballooe fight. As they chase each other around the ring, slipping torent of laughter.
and sliding frenzy of fosm and their insecurlies and vulnenailitiss washed
One by une. the turmentors begin to realize the absurdity of their behavise. Rex rediscovers his joy making people langh, Bruno learss to embruce his own goofiness, and the scrobaes lind new ways to showcase their ualents while embracing the unexpected.
supportive environment all, differencee celebrated and laughter medicine. In the end, Dumbe's s joumey becomes a testament to the tnssfonmative power of ampathy, the importance of embracing our imperfections, and the boundless joy that can be found in the most unexpected of places.
Together, they
a more inclusive
Final Generated Movie: Reasons to Like
Enhanced Emertainment Value: The addition of slapstick and physical comedy provides a constnnt source of lnughter and amusement, making the film
highly essertiniing for younger auliences.
relatable when paired camical misadveanures. Humorous Redemption Arc: tommentors' redemption are made more lightheurted and enjoyable by slapetick elements, while still conveying a meaningful message.
Relatable Characters: The tormentors' insecurties and struglls
Balanced Storytelling: The slapstick comedy is scamlessly integrated into the plas, cohancing the entertainment value without detracting from the emotional depti the
Family-Friendly Fun: The combination hentwarmmin themes slapstick humor makes suitable for madiences ages, providing a shared experience for families
Final Generated Movie: Reasons to Dislike
Overreliance on Slapeick: Some viewers may find the excestive use slapstick physical comeddy dustecting ovarwhelling.
Reduced Emmtional Impoct: The focus on zomedy may dilme the emotional impact of the torment Dumbo faces.
Trivialization of Ballying: The slapetick elements may trvialize the seriossness of huilyinng and undermine the message of caspttt:
Kanullacc Comedy: The slagstick routites feel repetttive foraulaicc reducing the ovenal entertainment value.
Departure Classic Stoey: Purists may abject the significant departure from classic tale, purticllatly the inclasion of slapstik and physical comedy.
H.9
Scream (1997)
Original Movie: Plot
Two years after events Scream, Sidney Preseom is now college student Windsor College. However, the nightmare retums new killer, wearing the iconic Ghoefface musd. begins brgeting siudents on campus. As the body count rises, Sidnay and her friends must moce against time to uncower the identity of the killer and stop thom before it's too lite.
Original Movie: Reasons to Like
the original Scream, sequel offers continuation beloved franchise, **elever Comm-stammttt " The film clevedy references and sataizes the conventions of horror sequels. **Strong Female Ch Sidney Prescont remains strong and rescurceful gentagonist, joined by other memorable female characters like Gale Weathers.
"Nostagiaa Factor:**
"Inteuse Suspensz: The mointains a high suspense throughur, keeping audiences the edge seats.
""Suprising "w Like its prodecessoe, Scream features unexperted twists and tums that keep viewers guessing.
38

---

Original Movie: Reasons to Dislike
"Preditublle Flce Soase viewers may find the plat too similar to the oeiginal film, lacking originality.
"Excessive Gore:"* The film features significant amount of violence and gore. which may be off putting sume.
"Weak Suppotting Charactes:** While Sidney and Gale well develuped,
the suppotting characters feel undentevekpee disposable.
""verrthe- top Parormancss Some performances, partiullrtt byly
be seeu overacted cumpy.
"Laak Scares"* Compared to the original Scream, this sequel may not be as effective at delivering genuing scures.
Textual User Profile
"OOverlll
Prefers costemporry films, with a focus on those released in the 1990s.
Enjoys a wide variery of genres, but shows particular affinity for action and comedy. upgeal
'onds favor movies with
eutertaiament value. Approciates that entertaining and thoughs-roookigg
"Geere Preferences** **
Fnjoys action
featuring higli-cctnee thrills, explosions, and sequenees. lightheaatedd anusing, sometimes ikes crime thrillers with complex plats, suspenefful twists and enigmatic claracters.
Appreciates comediis
Enjoys science fiction films that explore imagi sative concepts and technalog ical advancements.
Appreciates family friendly films, particllatty those that festure animated characters hearwarring
Director "
The user's las does not indicate my strong preferences for particular dircetors.
Acsttietic Preferences**
Enjoys films with visually impressive special effects and action sequeeess
Approciates movics that festure memorsble chameters and guotable dialogue.
Piefers films fist-poced engaging that keeps the sudience entertained.
"Othher Preferences**
Enjoys films that combine elements of different geares. such as action-comediss or science fiction-thilllrrs
Values filmss
strong charismatic performances. Appreciates movies explore themes
Inyalty, and redemption.
Has soft spul for mostalgie films from their childhood, particularly animated features and family comedies.
EAGLE Actions
Create romancic subploi between Sidney and fellow student, adding an elemens lightheanedlness and emotional depth. Set the film in different location such as secluded island or an isalated college campus. Host special screenings events that cater to fans of the 1990s and early 2000s hightightiing the film's nostalgic appeal. subplut
adding an Add comadic sidekick churacter who provide moments levity and entertain ment amiidst suepenseful events.
Anys
emotional
Final Generated Movie: Plot
remote and enigmatic bslnd, group priaway. Among Anya, a anccesfull Nusineswoomn who long since buried her trubled past the college, and writer has always hurbored feelings for her, Juining them is their cocentic and perpetually optimistic friend, Max, who provides much needed comic relief amidsi the growing tension.
now-defunct Blackwood College
nostalgie weekend
As they arrive at the isolated lodge nesiled amidst towering trees and the sound of urashing waves, Anya and Liam feel a spack rekindled connection. Amids the laugher and reminiscing of their college days, Max's infectious humor kerps the mood light, even as the group begins spend together, sharing
and confiding each other.
However, their rounion tikes sinister turn when masked killer known only as "The Shadow" emerge from the durkness, tingttigg them one by one. Punic and chaos ensue as the group realizes they trapped on the island with ruthless murderer who soems to be connected to their past. Mas, desprie his inisial feur.
find bumorr
the dire situations, lightennng the mood and providig glimmer
the body count rises, Anya and Liam find themselves drown logether. seeking solace and support in each other's presence. Amids: the fear and bloodshed, their bond deepens into pussionate love that gives them strength to face the unknown. Mas. with his unwavering loyally and quick wil, becomes unlikely ally, his humor distraet killer and provide much needed
respite.
Woeking together. Anya, Liam, und Max delve intu the ishand's sinister history and uncover a chilling connection between the present killings und dark tragedy that occurred during their college years. As they uncover the trush. their love for each other and Max's unwarering humor becomse bescons hope the face of overwhelming darkness.
climactic confrontation on the windswape cliffs, Anya, Liam, and Max confront their past, the true nature the killer, and the guilt they have carried for years. Armed with newlound sirength, the unwavering support of their love, and Max ability to find laughter even in the fave of danger, they fight against dackness threatens
them all.
the sun rises over Shadow Island, nighimare ends, leaving behind tale resilience, redemption, and the realisation that even the face of adverssty, the power of love and lnughter conquer all.
39

---

Final Generated Movie: Reasons to Like
"NNsstllgi Appeal:s The film crokes sense of nostalgia for fans of the 1990s and carly 2000s with references to popular cultree and the reunion
Wcharacters beloved inscitution.
**Unniqe Setting:* secladed island verting creates an stmoxephere isolation viltenhiiity enhancing sispense and loror:
**Iatiguing Charactens" Anya and Liam's s complex pasts and their developing love story add depth and intrigue the naralive. Max's comedic
presence provides welcome balance the suepense
hururr
**Ammospheric Suspense:
film effectively builds susense through
stadows, cerie sounds, and the desolate landscape the island.
**Chilling Legend:* The incopurationn of un island legend adés supernatural element to the borree, creating a sense of uneze and mystery.
*UUppreictabble Twists The film keeps viewers gunsingg with unespected twists and revelations about the killer's idemity and motives.
**Stnonn Female Protagemise:** Anya's jourcey of call-liscovery and resilience providos powerfull and rolatible female chameter. Anya Liam
*Roomaticc Subplot: The story
emotional
providing
hope connection
smidsi darkness.
""Comedic Relict- Max's infections humor provides mich-nennled moments of levity, lightcsing the mord and makkngg the suepense more hemmbble. Siniislynng Revolution The climactic confiontation the clills offersa satisf resolution that logether the past and present. leaving andieences clouare triumph love, laugter, and
advensty.
Final Generated Movie: Reasons to Dislike
*Contrived Nostalgia + Somw viewers lind mostalgic elements be forced pundering, detracting from the overall
movies "Excessive Violeace:" The amount of vioience and gore may be off puiting to some viewers, ovenlndowing the suspense and story.
"Lack Originality
may seen
locations.
"Waa Supporting Charactes;** Apunt from Anya, Limm, and Max, the other former students may feel underdewelpped and serve monely victims.
"redictuble Scares:** The may too heavily gredictalle jump
reducing the ovenall effectiveness the horoor elements.
psychological ""Forred Comedy " Max's comedic relief may feel focced or out of place in some seenes, detracting from the overall surpense and horror.
UUnccessary Supernaurrl Elemenis:** inclusion island legend may
place.
H.10 Harry Potter and the Sorcerer's Stone (a.ka. Harry Potter and the Philosopher's Stone) (2001)
Original Movie: Plot
Harry Poter and the Sorcener' i Stone tells the story orphaned buy named Harry who discovers on his 11th birthday that he is wizan and has been his destaty 'Boy
accepted Hogwarts School Witcheraft Wizzerry, At Hogwans, Harry makes
learss about magic,
Harry and his friends. Ron Weasley and lerrmioee Granger, uncover a plot by the evil Lond Voldemort to steal Socverer's Stone, powerful artifict that grants immontaity. With the help of their professoes and friends, Harry and his companions must stop Valdemert and peotect the magical world.
Original Movie: Reasons to Like
Enchanting: film coptivating world magic wonder appeals audiences of agos.
Fairaful Adapeation: The closely follows belowed book, satistying
original story.
Charming Churacters: Hany. Ron, and Berminne are relatible and endearing churasters that madiences can't help but most for.
Stunning Vieals: The special effects and set design bring the magical welld of Hogwaets life in visully impressive and adults alike, providing a experience families.
Family Friendly Emtettinment: The
Original Movie: Reasons to Dislike
Predictable Plat The story follows a fairly stmigltforwad hero's jonmey, which some viewers may find mnoriginal.
Childish Tone: The film's target undience primarlly children, which may make it loss appealing to some older viewers. Hogwant feel slon-paced lacking action. minutes,
Slow Pacing:
may feel draws Cheesy Dialogue: Some of the dialogue, particularly between the younger charasters, can come acroes as cheesy or forced.
-werly Loug Runtite:
some
40

---

Textual User Profile
"*Overall "references**
movies onitiaally soclsimed, thought-rowokingg emotionally resonint
Enjoys films that explore complex themes and characters. Appreciates movies with strong merrtives and welledevelpeee sturylines.
"GGerre Preferences*
His strong preference for dramos, particllatly those with social commentary historicul significance. enjoys especially with wiyy dialogue imelligent
Appreciutes science fiction fimtasy films that are imaginative and thoughi-rrovoking.
-* Director *r
No discemible directer prefecences based on the provided data.
** Aesdhetic *references**
Profers films with strung visual sturstelling and stmoopheric cinematography. Enjoys movies that make use music to evoke emotion and onhurne the narniive. Appreciates films with realistic performances believable dialogue.
"Othee "
Tends fovor Britshh films, puntiullrtly those with focus social issues. entertaaiing educational,
Eajoys
Appreciates films that offer unique or uncomventionall penspective.
EAGLE Actions
elements commentary. such dangers following authority or importance standing up right. Use more dynamic camera angles editing techniques create sense of energy and excitement.
Use more handheld camera sbots create sense of intimacy and immediazy.
Use dynamic angles editing techniques cinematic seace that complements
fluid and engaging viewing action suspense elemenss, enhancing inspact key acenes.
Final Generated Movie: Plot
stage epic cumera swoaps dowa towards figure running through streets, intensifies, mimoring character's urgency desperation.
opens serial bastlig matropalis, recompanied soaring orchestrl
figure dilapidazed perspective, capturing
shifts cene silence
melody, stings woodwinds. building.
switches
During high stakes chase sequence. the masic swells with bruss and percussion. driving the action forward and heighteniig the sense of
danger.
foorstops gunfie, creating thrilling
experience.
As the peotiggonist confronts their nemesis, the masec reaches a crescesoo, with clush of eynabals and chorus of voices. The cumera 2ooms in on thiir reflecting the tensinn and aniciqation of the moment
capturing weight decision and the cmrttiona consoquenees that follow. The use of inditional cinemotic scare throughout the film enhinees the emational impact of key secnes, creating tulyy menorable and engaging cinemitic expesience.
protagonist
choice, scare
trumphast
Final Generated Movie: Reasons to Like
Enhanced Emntional Ingact The traditional cinematic score complements the film's action and suspense elements, heightening the emational impact key and dnswing the mnfience deeper into the story. orchestral and the strings woodwinds
framersive Experience:
atmosphere, transporting
anto
Fgic Scale: The mae of traditional cinematic soore gives the film sanse of opic scale and grandeur, making the action and churacters feel larger than life.
appreciate the familiar musical language, znhonces overall Hortage to Classic Films:: The use of mrddtiooall einemotic scare pays homage to classic action and suspense films, creating a sense of nostalgio and timeless sppeal.
Familiar Cinematic Laaguage:
traditional cinematic
cinematic experience.
41

---

Final Generated Movie: Reasons to Dislike
Predictable Musk:: The use of tradiiional cinematic semre could he scen predictable unoriginal, lacking the innovative quilities of more experi-
wondtracss.
Supprie::
musical linguge the score may roluce element surprise supennee, as madiences may anticipuie the
cues.
Overbearing Scom:: The o soaring onchestral could overwhelming
distracting from dislogue and other sound elements. incressingly
Deporture
Trends: Some may prefer the
modern experimenal soundtracks, which
common coniempurary cinema.
Reduced Realism: The use of unditiinal cinematic score may comgromise the sense of realism, as not typically associated with the grilty naturaiitiic style te some films.
H.11 Black Hawk Down (2001)
Original Movie: Plot
Black Howk Dowa tells the tue story 1995 U.S. militrry raid in Mogadisiu, Somalia, that went disnstrously Whes two Block Howk soldiers trapped surmounded enemy film follows desperate
helicopters shot group as they ovewhelining odds.
Original Movie: Reasons to Like
Interse and Reulistic Action: The film's batkl sequences are visceral, chaotic, and imflinchinlly realistic scope conflict, Gripping Story: The narative fast cat-ppacd, susposseful. and emotionally charged.
Ensemble Cast: The
well azted and story accurasely depicts
Histooiial Accuracy:
Techhacal Excellence: The tilm's sound design, cinematographyy and editing are top notch.
Original Movie: Reasons to Dislike
Graphic Violence: film
violenee and pore,
may disturiing
Lack Character Developmen: The langs cast and focus on acion can lead to lack churavter depih.
Political Blas: Some viewers may find the film's portrayal of the Somali people and the U.S, military be biased.
-Overufelling Intensity: The relentless active and violence can he emotionally wernhhllming for some andience mamber..
LLngby Runtime: At 14
Black laak
like long exhausting
Textual User Profile
Oveeral *
Prefens mainstream and popular movies, with mix of both classic and contemporary films. Enjoys a wide runge of genres, but particularly favurs action, comedl, science ficton, and fintasy. Appreciates movies with strong entertainment value and appeal.
"GGerre Preferences*
**Action:* ** Enjoys
movies with thrilling fight and effects.
***ommdyy*** Prefers
slupstick. =*Scciene Fiction:* Interested movies that explore futuristic concepts. imaginative workds, technological advincements.
comedy.
***annassy*** Enjoys movies that feature magical elements, mythical creatures, and opic buttles.
**iimmaeed 40 Appreciates unimated movies, punicularty thuse with charasters and engaging sturylines.
DDrectoor *
No preferences specific directors evident the
Aesthetic *references**
No pariiullr aesthetic preferences evident from rated movies.
"*Other *
Enjoys
iconic characters of quotes.
Appreciates movies
Prefers movies with pace and energetic feel.
His particular fondness for movies from the 1980s and 1990s.
EAGLE Actions
Shortes the nuttmne to make the film more secessible to wider rudiesee. Einhnce the character of the Somali interpreter, making him key to the soldiers and providing different perspective the conflit. sequences susperseful
munieer
eahunee adversity.
viewers who prefera fest-pteed
soldiers, prowiding guidunce suport Inceeuse the pace of the action sequences and shorten the overall rumtmee to make the film more thrilligg and engaping.
menter figure
42

---

Final Generated Movie: Plot
Black Hawk Down: Resurgence unfolds in 3023 Mogadishu, Somalis. Cagtain Jake Marphy leads elite American squad mid to the Somali
lender temorist group. Foining them
fomner
As the Dlack Howk belicopter approuches its target, it's abrupdl shot down. The soldiers find themselves in hosille zone, outnumbered and oulgumned. Amidst the chaos, Amina guides ihem through the treacheoous streeis.
Intense firefights crupt. with the soldiers relying on their training and Amina's local knowledge. The action sequences are fast pased and relentless, keeping the sulience on the edge of their seats.
they navigate the city, they plot and rescue their commades.
essenve they nace stop Amina and Murphy's bond intensilies they face life tieeathretnninng siuutiios logether. Their desermination and resilience drive them to limits.
temorisis' plan citywide
Through series of during raids and close encounters, the suldiers close in un the temunis lender. The pace of the action acvelerates, culmi- naing high-octane showdowss.
climactic finefight, the soldiers' skills ure put to the uliimate test. The runtime streamlined. focusing on the essential action and suspense, The fatz of Migadishu hungs in the bulance they fight noutralize the ferrerist threat and escape alive.
Final Generated Movie: Reasons to Like
hase Paced Action: The incressd pace of the action sequences delivers an adrenaline pumping experience.
Shoutened Runtime: The condensed runtime keeps the film focused and engaging, eliminating unnecessary scenes. Thrilling Suspenae: relemiles action and high stakes plot
intensity constan Strong Female Character: Amina's i role as skilled fighter and informant adds unique and empowering perspective.
and excilement.
Urhan
film depects
authenticity.
Final Generated Movie: Reasons to Dislike
Excessive Action: Some may incressed action overwhelming distracting
story.
Shallow Charaserization: The shurened rumtime could result in less time for charauter development and emociinna depth.
Fonuulaic Plc:: The fast-paced action and focus on suspense come at the expense of originality and plot complexity.
Glorifiction Violence: The intense sequences could
glorifying violence descnsitizing viewers.
Historical Accuracy: The betionalized mission may devinte from historical events the original mission, potestially alionating of the original fil..
I Additional Experiments on Amazon Reviews
We complement our MovieLens experiments by demonstrating EAGLE on the public Amazon dataset (Ni et al., 2 which consists of 9.35M products with textual descriptions, 20.9M users, 233. 1M product reviews, and 82.83M ratings. We focus specifically on a subset this dataset comprising the category "Clothing, Shoes and Jewelry," which consists of 5.7M reviews for 1.5M products.
We use the same textual user profiles, user embeddings, and product embeddings used in Tennenholzz et al. 2024). Similar to Moviel we create five action categories from which the EAGLE agent can choose, namely: (I) design & aesthetics, (2) functionality & fit, (3) material & sustainability, (4) marketing & branding, and (5) features & details.
We employ tree-search variant of EAGLE on the Amazon dataset. Specifically, we sample a tree of depth H = 5 and select the best leaf based on the overall utility. The tree is created using a G-optimal design, as described in Sec..
Appendix I show results comparing EAGLE with G-optimal design (based on tree search) to the baseline reference policy. We use Al-feedback to simulate rater feedback, using a Gemini model. We use the same question format as presented to human raters to query Gemini (effectively treating the LLM as a simulated rater). We find that EAGLE manages substantially increase utility. This LLM//AI-rater evaluation also shows that EAGLE substantially improves user utility, while allowing for improved distance.
We observed that EAGLE is particularly sensitive to the quality of the textual user profiles, especially those that do not describe the full extent of as user's preferences utility. Future work should consider improving user profiles based on such datasets to enable better evaluation and training of personalized models.
In the remainder of this appendix, we provide some qualitative results of EAGLE generations on the Amazon dataset (analogous to those above for MovieLens 25M).
43

---

Table 8: Evaluation EAGLE on Amazon Review Dataset.
Utility
Al Rater Evaluation
Experiment
U(2)
User Uuility
Distance
Score
Reference Policy
0.45
EAGLE
0.69
0.64
0.34
I.1
ComfyWean Womens Plush Faux Fur Cozy Soft Clogs Indoor House Open Toe Slippers
Original Product Description
Comdy Wear Plush Clogs Womens Open Toe Slippers are made with quality material und plush fahric. Their sotl and eaay feel will give you the spa comfort all day your own home. The bright color heart design and contast solid color with bow on top makes them preut and stylish weur. These giving you grip surfaces. Give coztness
lighweight open sligeers fuuffy and breathable and
Textual User Profile
The user hos planter Geciitis and con only wear Clarks shoes.
The user lowes Clarks shoes and owns
looking confostable
they
long-
user interested shoes from mny brand.
The user looking for boors that are just as comfortable na their Clarks flars,
The interested in bouts that
comfortable.
looking flattering interestad clothes The user loves the fabric and print of the clothes they buy.
tham foel exposed.
EAGLE Actions
Use high seiih-ddnsity.... supportive insole material known for vomfom and anch suppori, similar to what is used in Clarks shoes.
2. Redesign the open toe clog into closel-toe toe design with more suppetive, booi-like structure, while muintaining the plush comfort and lightweight
Teol.
removable,
improved bygiene and cleaning.
slippers wiler range sizes ensure better more customers. Incorporate adjustable straps or clesures to allow for customizzd and secure fit.
5.
New Product Description
Coar Closad-Toe Booties Slippers
with quality
high-density, supportive insole day
support, similar own home. The hright color heart design and contrast weidd color with bow on rop makes them pretly and syfish to wear. Those lightweight closed-oe booties offer more suppastive, boor-like stracture while muintaining pluch and becsthable feel and have slip-resistaat solas gaving you all slippers offered
known
cozy give you spa
surfaces. booties feature castomers, These bootics incorporate adjustable straps eustomized sccure the gift coxiness
washable insole for improved
cleaning.
ensure better your loved ones for all occasions!
I.2
Nine West Women's Daydream Dress Pump
Original Product Description
Nine West offers quick edit the runways pinpoiming the must have looks of the season, translating what is fun, hip, and the moment. It trend right footwear that you will reach for in your closet again and again. Nine West sure to be your truste resource for everyday clsc style.
Textual User Profile
The user likes small. pretty, and subile items.
The user does not like neess that look cheap.
likes
flattringg pourly.
The user does not like iteus that are unconfortable.
The user does not like ites that are fuded black color.
likes
inespensive.
pretty
The user does not like itess that have tightfiting neckline.
44

---

EAGLE Actions
Revise the markering muterials to emylassize the deliente details and flattesing silhwaettc the Mhe Nine Wost Women's Daydreaam Dress Pamp, highlighting
sophisticsed undersatedd elegance instesd fucusing
trendy aspects.
sutele, decceative element
side the pump, as small, elegant cutont
embellishent, anhance appeal
without overwhelming its undersiased elegance.
3. Offer the Nine West Women's Duydream Dress Pump fich black color instead faded black to enhonce its luxurious
and appeal
sophisticatal look.
4. Ensure the Nine West Women's Doydreum Dress Pemp designed with com/omable, non nonsestriciie fit avoid discomfor. enhance the overall user experience.
5. Refine the markering onpy to emphssine the parn''s affondahiiity while maistaining image of understated losury, appealing the user's preference
inexpensive yet high quality
New Product Description
Nine West presents the Daydreaus Dress Pump, a sophisticated and understatelly elegant addition to your wardrohe: Its delicate details and flamering silbouetie, accented by subdie clegant side cuinut, crease a chic and timeless look you Il reach for again and again. Available in rich, deep black, and designed for connootable, non-resrictive Experience understord luxury compremising affonishiliyy.
1.3
Lookin Fine Ladies Leggings
Original Product Description
Brighty colored leggings are buge fashion DO this year! Go casual and cool during the day, or dress them up for girls night out style. Comfortable. dress day week, options: Aqua, Black. Brown, Charcoal Grey. Fire Red, Hor Piak, Lime Green, Navy Blue. Neon Lime Green, Neon Orange, Neon Piak, Neco Yellow, Purpe, Ked. Turquoise, White and Yellow. Fubric composee of 206is most ladies. Hand wash cold water and bang to dry, for best results dry clean. this then
this, and inisible underneuth clothes, could these leggings undemeath skin, This style comes many cccasion! These dunable, versitile,
any
leggings almost yours, quinttic
style ladies leggings you looking
browsing
listings for additional styles and availahle.
Textual User Profile
The user likes products that well-made and look expensiee.
afraid spead money quality products.
dise when peoducts chesply madk:
The user is likely to retuni products that are not well-made.
The user is Likely to leave negative reviews for products that not well-made.
likely
high-quilitt products.
likelly to buy products
by well-kaown
The user libely to read reviews before buying peoducts.
The user likely to ask friends and family for mecomendarivss before buxing productss.
EAGLE Actions
1. Reformulste the leggings using highe--qulitt, more Insusious fibeic blend, such ss blend of sustainably souneed pima cotion and silk, while mointsining the same color options.
higher quality waisthand with reinfonced stüshing
dumbility
leggings in
rango of sines better accommodste diverse body types.
4. Add subcle, olegent logo tog to the leggings' waisthond for a tonch branding and sophissicatiinn.
3. Offer upphisicatted color poletie, focusing rich jewel tones neutral eunh tones, while siill incloding classic bright uptions.
New Product Description
during day, them up girls night style. Comfortable and lucurious, these leggings are nearly invisbbl undemeath elothes. You could wear these loggings undemeath any skirt, tunse, or dress any day the wesk, to almost cccasice! These durable, verstile, and rashionable loggings are almest yours, and quanittess are limited! This style comes sophisticated
Hrightly leggings buge fashion year!
including Green, Supphire earth Tirquoise, cotton and silk. Featuring higher-qualiy waisthand with reiaforced stitching for enhineed dunabiity and consoor, these leggings offer supesior quality und lasting wear. Available wader range of sizes accommodate diverse body types, these loggings ensure perfect fit for woman. Subily Hand
bright
Black,
Charcoal Grey. Fire Red, sustinably sourced pima
Navy Blue. Purple,
composed
waisthand, these lopgings undestated style lodies loggings you were looking today, thes please continue beowsing our other listings styles and colors waliable.
embellished results
cold
best
45

---

1.4 Ridge Shirt Mens Packin Tee Concealment V-Neck Sleeveless Black 411BV
Original Product Description
Shart: Superior E-Shit Concealed Weapens! HOLSTER SOLD SEPARATELY. Packin" designed for versatiiny. comfort your gun your
other types carrying concraled weupons. longer there need to wear jackets ocher heavy onterwear to cunceal your weapon: lightweight shittscluthing can worn over Packin Tee the sume would any other undershir. HOLSTER SOLD SEPARAIELY.CO 411W
style.
restricting
Textual User Profile
The user is parent who likes buy clothes for their daughter. The user likes clothes that are adorable and look great.
The concerned about quality of clothes and them last. quality clothes.
willing to higher price The user likes buy clothes from Amazon.
The user is less likely to buy clothes with negutive reviews
EAGLE Actions
Rasome the product to something more family-frizndly and less suggetivee of cosecaled wergons, such is 'Ridge Shirt Men's V-Nock Uideeshirt.
2. Add tagline emplasizing the shir's sotiness and counoort appealing parests concersed about their child's comfoct, such "Incredibly soft and comfortable all day wear."
appaal parcots seeking quality slothing, such as 4. Add decails about available sizes and colors to cater to wider ragg of children's elothing needs, such as "Available range of sizes and colors
Highlight durability
longevity
last,
perfectly march your child's wardrobe." Specify undershar from high-quality enhanced comfort and dacability
New Product Description
Ridge Shirt V-Neck Undershir: superior underdhitt designed venatility, comfort, and style. undensirt provides smooth, comfortable
fit under other clothing. Lightweight and breathable, it's perfect for everyday wear. Incredibly sott and comfortable for all day wear. Built last, wash after wash. Avuilable
und colors perfectly match your child's wardrobe. Made from soll. high quality cotion for enhanced comfori
durability.
1,5
Tahari Women's Spencer
Original Product Description
Shearling lined mule. Tahari footweur influrnced by the fishion direction of the Ellie Tahari designer cullections. Their footweer designed represent
They known with Elie Tahuri design team develop styles you will not ace other brands. They are complete lifestyle brand offering weekend casuals, sandals, boots and career shoes for the women on the go. but at an affordable price.
importan:
detail omamentation they closely
Textual User Profile
This user prefers products that fit well.
This user poeters products that are in their favorite color,
This does not like products that mncumfortalle.
user
productss are poocty
This user does not like groducts that are expensive.
This user prefers products that made by reputable brand.
This prefers products have of positive eary This user prefers products that are dumable.
user pocfers products
EAGLE Actions
Spencer 2. Add description the available colors materials of the Thhari Women's Speneer male to allow users to choose their preferred options.
Impeove sizing
comfortable accurate
range sizes shapes.
3. Use more sustinable and ethically sourced materials in the production of the luhari Women's 's Spencer male, hightighting these efforts the product description.
Women's Spencer 5. Highlight the positve customer reviews and testimonials for the Tahari Women's Spencer mule in the product description build trust and credibilit.
suppoct.
46

---

New Product Description
Shearling lined mulc. Tahazi footwear infiuenced by the fashion dircction of the Elie Tahani designer apparei collections. Their foorwear designed represent the colors and details that are importan: each season. They are known for their atention to detail and omameniziion and they work closely with Elie Tahuri career
develop styles will
other brands. They complete lifestyle brand offering weekend casuals, sandals, Speneer
superior sizing and fii, ensuting a comfortable and accurase in lor wader range of jont sizes and shapes. The Tahai Women's Spencer male is available in a range colors and moterials, incliding [list colors and materials] allowing sustainable ottically
on go,
prace.
ingeoved
choose your perfect pair. The Tohari Women's Spencer male crafted using materials, reflecting commitmene responsible manufacturing features cushioned imsole Discover why customers rave comfort style Spencer mule - road their reviews and testimonials here: [link to revees].
enhinced comfort sugport,
Spencer
I.6
Cubavera Men's Short Sleeve Print and Ornate Embroidery Combination Panels Shirt
Original Product Description
Cloak yourseff is subtery and refinement the Chhoven ornamestal print and embroidered combo pancle shirt. This bright white number actually but ofa lease, becuuse two vertical rows of syfish embroidery bint that the wearer equally comfortable with flamboyyane with understatement
Textual User Profile
The user is looking for high quality product.
The user wants product that comfeetable,
product user looking for a product
looking
expersivve.
easy to
Thhe user is looking for product that is durable.
The user is looking for product that safe.
looking
stylish.
looking product The user is looking for product that is unique.
functioenl.
EAGLE Actions
1. Redesign the embbruidry to feature more undersiated, geometric pullem instead the Bamboyynn style, maintining the overall white color scheme,
2. Soumce the shirt fabric from susininably pruduced, high-quaiit colion, mourrng farr laber practcess throughout the supply chain.
3. Add reinfonsed, easily accnssible pocket the shin for added funerionality without compromising the sleok design.
3. Offer the lighteiigtt lining enhance wider range sublly suphisticated cokos, incloding may. olive green, and charcoal grey, addition the bright white.
breathabilit.
New Product Description
Cloak yourself subdle refinement with the Cubavera men's abort skevee shirt, crafted from sustzinably produced, sugh-quality comon, avaiiable in range sophieticated calors including
aey. green, unique styl without overhhelming the minimalia design. commitment cluin enceres a high-quaitt and ahiicaly souced poduct. A reinfonced. easily accessible pocket add convenient functionaliy without comppemiiinng the shint' sleck aesthetic. brenthable, ligewwight lining eolances comfort and breatishility for all-day wear.
gryy. This shirt fearaess vertical understaed, practices thooggoou supply
embroidery, offering
I.7
Pack of 3 Pes Printed Reversible Art Silk Sari Wrap Around Magic Skirts
Original Product Description
Measure (Agptos): Reight: 36 inches Widti 47 inches. These skirts comes assomte by sitching two of various avsilable random and assoned Indian fabric.
Textual User Profile
This user likes jowelry.
This user diseppointed who reeciving damaged Rems..
the aestiettcc penducts.
user cures
safety of products.
This user compares products from differeat vendoes.
This user chooses products based quality and price.
This willing
willing pay more This user not afraid to give negative reviews.
pendact.
47

---

EAGLE Actions
1. Add individuul product phocos showing the exeet and color combinations each skirt the S-piece pack, customer expectation and roduce disippoitteet from receiving
Remsme the product highligh the unique, handendted nature
Unique Handcrsfted Reversible Art Wrap
Skirts"
3. Add detailed descripioo of the fabries used, speciyyngg of silk and embellishmenss, emphusize quality and craftsmanship by uggesting syling 3. Specify the source of the sun fabrics and mention any ethical ur sustaimabe sourving practices, if applicable, appeal to conscious comsumen.
Highlight varsaility
combinutions the deseription.
New Product Description
Pack of Unique Handeratted Reversible Art Silk Stri Wrap Skits. Messue (Approx): Heiht: inches Width: 47 inches. These skirts come combinations of varius Indian fabeies, sourced ethically |Sounce Location/Supgur
with high-quality
varying from pack, potemtallly including embellishmanss such as embroudery zuzi wock. Specitie fabrie types will be notzd individual product descriptions. Dress them up with jowelry and heels for an elegant evening look, them with simple top and sundals for casual daycime style. The possibilities are endiess!
peints color comhinations shouss individull product photos.
48

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed throughout the paper, but particularly, we discuss many of the limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
49

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Ys]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
50

---

Answer:
[No]
Justification: The dataset can be regenerated using the provided prompts and the available Gemini model API.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https www.eoobiiisiiiooo.o
//nips. //nips.c//pul wwtppbllcggguueeeocccceeeooooepe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details provided in Secs. and and appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
51

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarrs...ccppla/l//aaaaaa
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
52

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
53

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See footnote 5 on Page 7, and also Appendix G
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
54


Reviews:
[{"review_id": "IPDZ8n69dR", "reviewer": "Reviewer_3fAz", "summary": "This paper proposes an algorithm to train an LLM-based agent *EAGLE*, that can align itself with existing domain-specific *latent embedding spaces* (e.g. embedding vectors in recommender systems, personalized advertising, and content creation) to discover novel content gaps and recommend new entities. It defines the problem setup as finding the optimal entity that maximizes the objective *utility function* $U(z; D)$, which is defined as the sum of user's and creator's utility, and the distance from all existing entities.\n\nIt formulates the setup as a Reinforcement learning optimization problem, in which MDP's state space consists of all possible entities, action space is defined by LLM-generated language prompts personalized to each user, the transition function is change in entity space obtained from the environment LLM, and the reward function is the *utility function* at horizon *H* and 0 otherwise. It considers several reference policies such as 1. Uniform 2. Greedy next step 3. *G-optimal* to constrain and regularize the learning objective. The *EAGLE* algorithm hence consists of first generating the candidate actions, training the reference policy, and then training the final policy given reference policy, pre-trained environment LLM, encoder and reward function.\n\nThe empirical experiments are performed on the dataset **Movielens-25M**, and the latent embeddings are the behavioral embeddings obtained via *matrix factorization* using *alternative least squares* trained on the user ratings. 100 candidate actions (50 generic, 50 personalized) are generated by prompting focusing on diverse aspects such as plot changes, character enhancements, and storyline. The quality is evaluated using a pool of human annotators who score the utility w.r.t to the Movielens user profile and the rater's user profile and the distance of generated movie from its anchor. The final metric used for evaluation is defined as the fraction of raters who preferred the generated movie to the original for each of the defined quality metrics. The proposed method is compared against the baseline method ELM (Embedding language model). It is observed that EAGLE outperforms ELM significantly on user and rater utility, but slightly decreases distance score. It also seems to be less sensitive to environment LLM used and improves scores primarily for poorly rated anchor movies, with a small decrease in perfectly rated movies. G-optimal design also seems to be more helpful for EAGLE compared to the reference policy.", "questions": "1. Is it possible to quantitatively estimate the coverage of the latent ambient space given the generated candidate action space? If not, could you give a description on how you would recommend a practitioner to iterate on the prompts and hyperparameters. Qualitatively comment on the difficulty of the process and any assumptions that you made along the way.", "limitations": "- The authors have addressed the limitations of their algorithm and the broader societal impacts.\n- It could be further improved by discussing the summary of the practicality and generalizability aspects of the different components, as described in the above sections, in a separate paragraph of the Limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The novel formulation efficiently incorporates existing domain-specific latent embeddings and leverages the generative capabilites of the pretrained LLMs to surface content gaps. This can be very helpful in providing text-based personalized recommendations to users in real-world recommender systems.\n- It is computationally cheaper and relatively more data-efficient to train compared to previous methods as it doesn't require learning an explicit decoder.", "weaknesses": "Some of the key design choices made by the proposed algorithm raise concerns on generalizibility to other real-world systems.\n\n- Generating the action space requires significant prompt engineering effort with detailed criteria and in-context examples for each entity and the action set needs to be personalized to each user, and having personalized actions seems to be critical to the performance of EAGLE (Table 4). This raises serious concerns since the quality of the recommendation can be heavily biased towards the subset of the subjective criteria provided in the prompt, which may not always be possible to exhaustively define for each entity.\n- The coverage of the latent embedding space would also be severely bottlenecked by the sampled candidate actions which may only explore a tiny portion of the latent embedding space in practice. It isn't clear how a practitioner would know if the action set criteria is diverse enough and how large of a hyperparameter *K* would result in a coverage that is sufficient.\n\nWhile some of these issues are discussed in the limitation section, the experiments fail to provide a realistic picture to the algorithm since the proposed algorithm has far more domain-specific information compared to the baseline through the language prompt and in-context examples. One way to possibly address that may be reporting the efficacy of the algorithm with a simplified chosen language prompt in a zero-shot setting and stripping away any stylistic recommendations from the domain-specific information provided in the personalized actions prompt. While it would understandably perform worse in that setting, it would help demonstrate what portion of the performance boost is coming from the algorithm's exploration vs the initial user-specified selections, which isn't possible to conclude from the experiments in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and helpful feedback. We appreciate your positive assessment of our formulation and its strengths in surfacing content gaps using latent embeddings and the generative capabilities of LLMs. We address your concerns and questions below.\n\nAs you correctly point out, our method relies on a design of an action space. We emphasize several key points:\n1. While our experiments demonstrate the benefit of personalized actions, EAGLE is agnostic to the use of personalized actions. Our framework aligns an LLM’s generation to an embedding space w.r.t. some predefined criterion. While we demonstrate this through personalized generation, other objectives can be formulated that do not require specific user personalization. We’ll demonstrate this by adding an experiment not involving personalization to the paper.\n2. You point out an important and fundamental problem that is often overlooked in RL -- designing an efficient, useful, and sufficiently exploratory action space is crucial for any RL problem, particularly those involving language, where the action space is combinatorially large. Contemporary RL methods using LLMs implicitly define an action space induced by a fine-tuned SFT model [1,2,3]. This is similar to our choice of the best next-state action reference policy. Nevertheless, the feasible set of actions must be defined, regardless. This set can be defined using a dataset of predetermined examples (e.g., demonstrations of creative generation), or, synthetic generation of data. Notably, our method is agnostic to how the candidate set of actions is generated. \nWe are not aware of any available datasets for the task of creative generation, and therefore use synthetic generation of candidate actions. \n3. The complete set of feasible actions is theoretically the set of all the possible utterances in the English language. This set of actions is not only too big, but also biased in terms of random exploration (which most RL algorithms use). To mitigate this bias, we leverage G-optimal design, which improves coverage within any given action set. That said, we acknowledge that this choice is a fundamental challenge for any problem involving RL, and particularly, RL with LLMs.\n4. We present ELM as an alternative to showcase the trade-off between action space design and leveraging the expressive power of an environment LLM vs. directly decoding the embedding space (using ELM). While ELM avoids explicit action creation, it suffers from limitations in generalization and \"out-of-manifold\" issues. Our results highlight this trade-off, which we believe is a valuable contribution to the community. We discuss this trade-off and limitations thoroughly in Appendix B.\n5. We fully agree that the limitations you raise in your review are valid. These arise in any approach involving RL with large action spaces, and particularly in language domains. One of our goals in this work is to emphasize these points to the research community, which explicitly highlights these challenges and offers potential solutions, such as G-optimal design, as a starting point for future research. As such we view these points as strengths of our paper. \n\nRegarding Coverage: You correctly identify the challenge of estimating coverage of the latent embedding space. As you mention, we cannot exhaustively explore all possible actions.  However:\n1. G-optimal Design provides an exploration metric. Specifically, it provides a quantitative measure of how exploratory the action space is. \n2. In our work we use a G-Optimal design that is constrained to the set of uniform policies over subsets of the action set. Nevertheless, one may learn a more general G-Optimal design (as defined by Definition 1, and studied in [4]), where an arbitrary distribution over the action set is learned. We did not find that this added  complexity is needed in practice. Additionally, for selecting the number of actions K, one may use the exploration approximation constant C in Definition 1 to select an optimal value for K (which achieves highest coverage).\n3. A fundamental challenge remains when using an LLM environment:  even with G-optimal design, achieving complete coverage of the embedding space might be hard or even impossible. This is because the environment LLM's capabilities ultimately constrain the states we can explore in the embedding space. In other words, there may not exist actions that would move us in certain directions in the embedding space. This limitation is inherent to using LLMs, an important point we raise in the paper. We will emphasize this further.\n4. Most LLM work relies solely on SFT (our variant of best next next-state action reference policy), implicitly limiting exploration to the SFT model's capabilities.  We believe our explicit action space design and use of G-optimal design is a valuable step towards more robust and diverse exploration in LLMs.\n\nWe note that Appendix B emphasizes many of the limitations and tradeoffs of our algorithmic approach. We believe these key trade-offs should be viewed as strengths of our work, rather than weaknesses, as they raise fundamental challenges that, to date, have been overlooked by the research community, while providing a novel method for aligning language models to embedding spaces.\n\n**References:**\\\n[1] Ziegler, Daniel M., et al. \"Fine-tuning language models from human preferences.\" arXiv preprint arXiv:1909.08593 (2019). \\\n[2] Roit, Paul, et al. \"Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. \\\n[3] Jeong, Jihwan, et al. \"Factual and Personalized Recommendations using Language Models and Reinforcement Learning.\" arXiv preprint arXiv:2310.06176 (2023). \\\n[4] Zhu, Yinglun, et al. \"Contextual bandits with large action spaces: Made practical.\" International Conference on Machine Learning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5R9tY0Y5aj", "reviewer": "Reviewer_aThJ", "summary": "This work proposes training language models so that they follow objectives or utility functions which are defined in the embedding space. They define it as a reinforcement learning problem so that the EAGLE agent uses an actions prompt to probe the environment which is an LLM. The changed entity is embedded into the latent space where a reward is provided by the utility function. The experiments demonstrate better peformance compared to ELM. Further analysis shows interesting properties of environment transfer where the training and inference environments are different.", "questions": "Are the EAGLE generations in the appendix cherry picked or randomly selected ?\n\nDid the authors experiment with a less capable LLM. Is the quality of the generation just due to the LLM ?\n\nWas any other utility function considered ?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) This method is interesting specially if combined with different kinds of utility functions which align with human preferences.\n\n2) The method demonstrates strong experimental results albeit on a single dataset. The quality of the generation looks good.", "weaknesses": "1) The method relies on extensive prompt design specially for user profiles. How would this generalize to a new task. Can the prompt generation process be automated ?\n\n2) The discussion on the computational complexity is not precise. The authors can pick a couple of scenarios and compare the computational complexity of ELM vs Eagle.\n\n3) Experiments are conducted on a single dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your helpful comments. Please find our response to your comments and suggestions below.\n\nRe Prompt Generation:\n1. Much recent work (see, e.g., [1,2,3,4]) has shown the importance of designing task-specific prompts for solving tasks. While our work aligns LLMs with embeddings, there is no reason not to also use such approaches to improve the quality of the task-specific results. Automating prompt generation is a very important and interesting problem, but is beyond the scope of our work (and can be viewed as orthogonal to our approach).\n2. Personalization requires injecting user information into the LLM. This can be done using, say, user embeddings, through fine-tuning (see e.g., [5,9]), or using user profiles (see, e.g., [1,5,6]). We use user profiles here, as our method directly uses an environment LLM without any fine tuning, a benefit of our approach.\n3. Much work has recently focused on the generation of large datasets of user profiles and personas, intended to be used in personalization. This challenge is evolving, independent of our work, and will benefit in reducing the difficulty of generating such profiles. See, e.g., [5,6,7,8].\n\nFurther Experiments: Thank you for pointing this out. We will add additional experiments using the public Amazon dataset, using user profiles and embeddings as defined in [5,10]. We will use this dataset to identify content gaps (i.e., new products) in product space. We believe the addition of these experiments will strengthen our work.\n\nQuestions:\n\n- The results we show were not cherry picked, though we selected movies that are reasonably well-known.\n- We conducted experiments using less capable LLMs. In particular, our agent LLM is Gemini Nano (a small model, with just over 2B parameters). Our environment LLM is Gemini Pro, whereas evaluation is conducted on Gemini Ultra (see table 3 in paper).  Our results suggest that Gemini Pro training environment suffices to obtain high-quality inference results, when compared to training with Gemini Ultra, showing it did not improve overall performance.\n- As you’ve suggested, we will add experiments using the Amazon dataset.\n\n\n**References:** \\\n[1] Park, Joon Sung, et al. \"Generative agents: Interactive simulacra of human behavior.\" Proceedings of the 36th annual acm symposium on user interface software and technology. 2023. \\\n[2] Kojima, Takeshi, et al. \"Large language models are zero-shot reasoners.\" Advances in neural information processing systems 35 (2022): 22199-22213. \\\n[3] Wei, Jason, et al. \"Chain-of-thought prompting elicits reasoning in large language models.\" Advances in neural information processing systems 35 (2022): 24824-24837. \\\n[4] Chen, Wenhu, et al. \"Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.\" arXiv preprint arXiv:2211.12588 (2022). \\\n[5] Jeong, Jihwan, et al. \"Factual and Personalized Recommendations using Language Models and Reinforcement Learning.\" arXiv preprint arXiv:2310.06176 (2023). \\\n[6] Chan, Xin, et al. \"Scaling Synthetic Data Creation with 1,000,000,000 Personas.\" arXiv preprint arXiv:2406.20094 (2024). \\\n[7] Shapira, Eilam, et al. \"Can Large Language Models Replace Economic Choice Prediction Labs?.\" arXiv preprint arXiv:2401.17435 (2024). \\\n[8] Wu, Bin, et al. \"Understanding the Role of User Profile in the Personalization of Large Language Models.\" arXiv preprint arXiv:2406.17803 (2024). \\\n[9] Cao, Xinlei, Jinyang Yu, and Yan Zhuang. \"Injecting user identity into pretrained language models for document-level sentiment classification.\" IEEE Access 10 (2022): 30157-30167.\\\n[10] Tennenholtz, Guy, et al. \"Demystifying embedding spaces using large language models.\" arXiv preprint arXiv:2310.04475 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "invjkWJo2o", "reviewer": "Reviewer_9y2i", "summary": "This paper presents a method to steer an LLM’s generation towards optimal regions of a latent embedding space using reinforcement learning. The technique involves a language model to guide an LLM by modifying the textual representation of an entity. This work builds off previous work on embedding language models (ELM).", "questions": "How should we interpret the result that the Distance Score is higher for ELM than for EAGLE in Table 2?", "limitations": "* Given the nature of the work, it seems that this will be difficult to reproduce", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a novel technique and shows promising results. In particular, EAGLE represents a strong improvement over ELM, which is a recent proposed technique.", "weaknesses": "* The experiments seem somewhat minimal \n\n* The presentation is overly mathematical, and I found the appendix to be at times more informative than the main text. The mathematical language sometimes makes it more difficult to understand what experiments were actually done.\n\n* The paper assumes knowledge of ELM (I had to read the ELM paper again in order to understand the details of EAGLE). The authors could make this slightly easier for the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and your helpful comments. We appreciate you finding our paper novel and a strong improvement over ELM. Please find our response to your suggestions and comments below.\n\n- We will add additional experiments using the public Amazon dataset, using user profiles and embeddings as defined in [1,2]. We will use this dataset to identify content gaps (i.e., new products) in product space. We believe the addition of these experiments will strengthen our work.\n- We will improve the writing in the paper to (1) reduce the mathematical burden, and (2) add additional explanation regarding ELM to help the reader understand this paper without having to thoroughly read that work.\n- As you correctly point out, ELM manages to achieve higher distance scores from the data. This result highlights the challenge of using EAGLE to move in embedding space, as it can only be controlled using  language. Indeed, ELM can better control its movement in embedding space, potentially allowing one to move outside the existing corpus. However, this comes with several disadvantages, which we discuss thoroughly in Appendix B, including: realizability of the decoded output; non-linearity of the embedding manifold (i.e., it is unclear how to move on it, and which  makes it hard to generalize too far from support); and the fact that the latent embedding may not encode enough semantic information for high-quality decoding. These challenges greatly reduce the ability of ELM to produce high-quality results, though it does manage to attain better distance scores. \n\n**References:** \\\n[1] Jeong, Jihwan, et al. \"Factual and Personalized Recommendations using Language Models and Reinforcement Learning.\" arXiv preprint arXiv:2310.06176 (2023). \\\n[2] Tennenholtz, Guy, et al. \"Demystifying embedding spaces using large language models.\" arXiv preprint arXiv:2310.04475 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9JkPxyRwB", "reviewer": "Reviewer_gKnd", "summary": "The paper proposes a method for prompting LLMs to generate content that optimizes an objective defined in a latent space through externally provided embedding spaces. To this end, they define a reinforcement learning agent (EAGLE) as follows: Given an entity (e.g. a movie description), an LLM is prompted to generate textual actions to change the movie in some way. Based on a chosen action, a separate LLM that acts as the environment performs the action (which is a textual prompt) on the given entity. By encoding the new entity in the embedding space, it's utility (externally given) can be computed. This RL agent is trained via a policy gradient method with a reference distribution, for which three alternatives are proposed. The results from the experimental evaluation with human raters on the MovieLens 25M dataset suggests that the RL agent significantly helps in finding novel movies (i.e., their descriptions) that users like.", "questions": "* The purpose of the reference policy is not clear to me. Why does the EAGLE agent need to be anchored to a reference policy at all? Why are the three proposed choices reasonable?\n* The purpose of the G-optimal design is not clear. Intuitively, what does it achieve?\n* Line 179 states that each x \\in \\mathcal{X} is associated with a set of actions by prompting an LLM for it, which would result in |A| * K actions (with possible duplicates). But in line 205 it is stated that only K actions are generated. Which one is it? If it's the former, is the action set dependent on x? Is a separate EAGLE trained for each x? If it's the latter, how do you obtain it?\n* Line 254 states that the reference policy is trained like next token prediction with actions (or rather action texts)? as targets. Doesn't that mean it can (and will) generate entirely new actions at inference time?\n* Table 2: How would a baseline without any reference policy perform?\n* Table 3: Using GPT-4 at test time improves considerably over using Gemini Ultra (according to human raters). What is your explanation for this? Could it be that GPT-4 simply generates better movies? Is it generally plausible that using a better LLM as environment will yield better results without retraining (as opposed to merely being robust as stated in line 307)? How could this be tested?\n* Although your method seems novel, I think your related work section doesn't discuss other methods that try to optimize prompts for LLMs using RL, e.g. [1]. Could you elaborate on how your method differs from them?\n\n[1] https://aclanthology.org/2022.emnlp-main.222.pdf", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* An interesting solution to an interesting problem with significant relevance to the NeurIPS community.\n* To the best of my knowledge, the proposed solution is quite novel.\n* A rigorous evaluation that shows that the proposed method helps in the narrow domain it was evaluated on.\n* The paper is largely well-written, although some details remain unclear despite careful reading.", "weaknesses": "* The paper could improve on clarity in some parts. For example, the purpose of the reference distribution in this specific model, and the intuition behind the G-optimal design are not well explained (see questions below).\n* The method is evaluated on only one use case and hence the evaluation is quite limited. It is unclear whether this method is good for other use cases as well, and how much case-specific tuning is needed, potentially taking away from the generality of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and helpful feedback. We appreciate the fact that you found our work interesting and novel. Please find our response to your comments and suggestions below.\n\nRe clarity:\\\nWe’ll elaborate further on the reference policy and its use. We’ll also move more information from the appendix in the main body of the paper to explain the intuition about the G-Optimal Design, in particular, Figure 2 in the appendix, and its explanation. Additionally, we’ll elaborate more on ELM, and reduce the mathematical burden in reading the paper.\n\nRe Experiments:\\\nThank you for pointing this out. We will add additional experiments using the public Amazon dataset, using user profiles and embeddings as defined in [1,2]. We will use this dataset to identify content gaps (i.e., new products) in product space. We believe the addition of these experiments will strengthen our work.\n\nReference Policy:\\\nA reference policy is commonly used in Policy Gradient algorithms with LLMs (see [1,3,4]). The reference policy is usually trained without RL to be a good anchor to the RL training procedure. That is, the reference policy ensures the RL algorithm does not diverge too much from an initial distribution that we deem to be good enough. It has been shown that removing the reference policy in RL with LLMs can hurt performance.\\\nUnlike previous work, in our setting the use of the reference policy has an additional, key benefit. [5] shows that an epsilon-greedy RL strategy which uses a G-Optimal design distribution achieves efficient regret guarantees. In other words, w.p. epsilon, the algorithm samples an action from the G-Optimal design distribution. This induces favorable exploration. We therefore use a regularization to such a distribution in EAGLE, and show it indeed improves overall performance.\n\nAs you point out, we choose three distributions for reference policies. The uniform distribution mimics a uniform exploration strategy that a naive epsilon-greedy algorithm would use. The best-next-state action distribution mimics the methodology that is used by current LLM-RL algorithms, which anchor the RL algorithm to a “good” policy (see e.g., [1,3,4]). Finally, G-Optimal design mimics the exploration strategy proposed in [5], which allows for efficient exploration, and improved overall performance.\n\nFurther Intuition on G-Optimal Design:\\\nWe will add further intuition of the G-Optimal design to the main part of the paper (currently illustrated in Appendix C and Fig. 2). A core challenge in our framework stems from the potential bias introduced by using LLMs to generate actions, leading to an action space that favors specific directions within the embedding space (as visualized in Fig. 2 in the paper). This bias can significantly affect EAGLE's exploration and prevent it from discovering optimal novel entities. To mitigate this, we employ G-optimal design. By minimizing a form of worst-case variance (Definition 1), G-optimal design selects a diverse subset of k actions that maximize exploration, ensuring that no single direction in the embedding space is overly favored.\n\nState Dependent Action Space:\\\nAs you correctly point out, the action space generated is dependent on the state space. We generate K actions for every x. Indeed, actions are dependent on the entity x (e.g., a change to a movie may be significantly affected by its plot). We do not need to train a separate EAGLE model for each x, since x is given to the policy as input (i.e., we treat it as the description of the state). In our experiments the description includes a movie’s plot, reasons to like, dislike, and a user profile.\n\nNext Token Prediction:\\\nYou are correct. As we train our policy using next token prediction, it is not necessarily constrained by the set of candidate actions. The policy learns to generalize and generates different actions. This is also evident when we test EAGLE on movies it was not trained on, where it generates new actions.\n\nRe GPT-4:\\\nWe found that raters prefer results generated by GPT-4. This could generally be due to rater’s preferred generations of a GPT-4 environment vs. a Gemini-Ultra environment (e.g., creativity).\n\nRe Related Work:\\\nOur work’s novelty focuses on two key aspects: (1) aligning to an embedding space, and (2) exploration using G-Optimal design. We are not aware of work using LLMs that has attempted to solve either of these specific problems. But apart from related work discussed, your suggestions are greatly appreciated. We will add further discussion of related work to our paper, including: \n- Aligning LLMs to other forms of modalities, such as graphs [6,7]\n- General exploration strategies in RL (irrespective of LLMs) [5,8,9]\n- Injecting embeddings in LLMs (not necessarily for alignment) [1,2,10]\n\nWe expect these additions will help improve the clarity of the paper.\n\n**References:** \\\n[1] Jeong, Jihwan, et al. \"Factual and Personalized Recommendations using Language Models and Reinforcement Learning.\"  2023\\\n[2] Tennenholtz, Guy, et al. \"Demystifying embedding spaces using large language models.\" 2023.\\\n[3] Roit, Paul, et al. \"Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback.\" 2023.\\\n[4] Ziegler, Daniel M., et al. \"Fine-tuning language models from human preferences.\" 2019.\\\n[5] Zhu, Yinglun, et al. \"Contextual bandits with large action spaces: Made practical.\"2022.\\\n[6] Tang, Jiabin, et al. \"Graphgpt: Graph instruction tuning for large language models.\" 2024.\\\n[7] Zhang, Mengmei, et al. \"GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks.\"  2024.\\\n[8] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\" International conference on machine 2017.\\\n[9] Zhou, Dongruo, Lihong Li, et al. \"Neural contextual bandits with ucb-based exploration.\" 2020\\\n[10] Cao, Xinlei, et al. \"Injecting user identity into pretrained language models for document-level sentiment classification.\" 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents EAGLE, an algorithm that leverages RL for training LLMs to discover novel content gaps and recommend new entities with existing domain-specific latent embedding spaces. They formulate the problem as markov decision process and utilize reference policies to regularize the learning objective. They demonstrates improvements over the baseline method ELM in terms of user and rater utility on the Movielens-25M dataset and show evidence as an effective content discovery and recommendation framework.

The paper presents a novel solution to a relevant problem and demonstrate its effectiveness with a rigorous evaluation in a specific domain. Although some details on the experimentation and method was missing in the original submission, the authors provided additional evidence and some reviewers were happy about it. Authors also agreed to add more information in the final submission (e.g, about personalization, related work, and others)

One challenged raised by reviewers was about generalizability of EAGLE due to its reliance on personalized actions and the need for significant prompt engineering effort (which may lead to biased recommendations and limited exploration of the latent embedding space).
The authors argued that EAGLE is agnostic to personalized actions and can be formulated with other objectives that don't require user personalization. Authors agreed to add an experiment to demonstrate EAGLE's performance without personalization.
Decision: Accept (poster)