Demystifying Embedding Spaces using Large Language Models
Authors: Guy Tennenholtz
Yinlam Chow
ChihWei Hsu
Jihwan Jeong
Lior Shani
Azamat Tulepbergenov
Deepak Ramachandran
Martin Mladenov
Craig Boutilier
Venue: ICLR 2024 poster (2024)
Keywords: large language models
interpretability methods
representation
recommender systems
Primary area: generative models
TL;DR: We introduce a method to make dense data embeddings interpretable using large language models (LLMs). By integrating external embedding spaces with LLMs, we render them more interpretable, transforming them into understandable narratives.

Abstract: Embeddings have become a pivotal means to represent complex, multi-faceted information about entities, concepts, and relationships in a condensed and useful format. Nevertheless, they often preclude direct interpretation. While downstream tasks make use of these compressed representations, meaningful interpretation usually requires visualization using dimensionality reduction or specialized machine learning interpretability methods. This paper addresses the challenge of making such embeddings more interpretable and broadly useful, by employing large language models (LLMs) to directly interact with embeddings -- transforming abstract vectors into understandable narratives. By injecting embeddings into LLMs, we enable querying and exploration of complex embedding data. We demonstrate our approach on a variety of diverse tasks, including: enhancing concept activation vectors (CAVs), communicating novel embedded entities, and decoding user preferences in recommender systems. Our work couples the immense information potential of embeddings with the interpretative power of LLMs.

Published as a conference paper at ICLR 2024
DEMYSTIFYING EMBEDDING SPACES USING LARGE LANGUAGE MODELS
Guy Tennenholtz, Yinlam Chow, Chih-Wei Hsu, Jihwan Jeong, Lior Shani, Azamat Tulepbergenov, Deepak Ramachandran, Martin Mladenov, Craig Boutilier
Google Research
ABSTRACT
Embeddings have become pivotal means to represent complex, multi-faceted information about entities, concepts, and relationships in a condensed and useful format. Nevertheless, they often preclude direct interpretation. While downstream tasks make use of these compressed representations, meaningful interpretation usu- ally requires visualization using dimensionality reduction or specialized machine learning interpretability methods. This paper addresses the challenge of making such embeddings more interpretable and broadly useful, by employing large lan- guage models (LLMs) to directly interact with embeddings transforming abstract vectors into understandable narratives. By injecting embeddings into L.LMs, we enable querying and exploration of complex embedding data. We demonstrate our approach on a variety of diverse tasks, including: enhancing concept activa- tion vectors (CAVs), communicating novel embedded entities, and decoding user preferences in recommender systems. Our work couples the immense information potential of embeddings with the interpretative power of LLMs.
INTRODUCTION
The success of deep learning has brought forth a paradigm shift in knowledge representation through the concept of embeddings-dees denne vector representations that capture high-dimensional information about entities, concepts, or relationships in a compact and useful format. Embeddings are ubiquitous, finding application in natural language processing (Mikolov et al. [2033] Devlin et al.) 2018), recom- mender systems (Rendle 2010), protein sequence modeling (Rives et al. [021). and more. These embeddings are invaluable, capturing nuanced relationships and semantic structure in data which traditional machine learning (ML) approaches often miss. Nevertheless, understanding these abstract representations remains challenging.
By design, the structure and underlying information carried by an embedding is heavily adapted to the idiosynerasies of the downstream task, posing substantial challenge to its interpretation and manipulation. Previous work on ML interpretability offers various task- iask-independent means to interpret embeddings, including dimensionality reduction techniques (e.g..t-SM. (Van der Maaten & Hinton 2008), UMAP (McInnes et al.,, (018)) or concept activation vectors (CAVs) (Kim et al. 2018). While very useful, these techniques are fairly narrow in scope.
As an alternative to such interpretability methods, suppose one could engage with embeddings using natural language to query information not directly expressed by the name or description of the underlying entity/concept. Perhaps one could even extract information from the embedding representations of non-existent or hypothetical entities. Consider, for instance, an embedding space representing items from online commerce site, trained using large corpus of user ratings or purchases, reviews, and other multi-faceted data sources. The embedding representation of an item may implicitly embody intricate details about its quality, usability, design, customer satisfaction, etc. Moreover, suppose one wanted to understand the properties of hypothetical item at a specific point the embedding space, say, if we predicted potential market for such an item. Since no such item (or description) exists, asking a conventional language model to describe this embedding point would
"Correspondence guytenn@gmai com

---

Published as a conference paper at ICLR 2024
heartwrringg
journey trrough
some of the most tamoue figurss of the 20t century.
The mosts beauthhily animated, with stunning viaute that bring the story to
bringing action The soundtrack mamorable, featuring some the moet iconic songs of the 1988..
The movie hull adventure excitement
entertained.
Forrest Gump
Animated Version of Forrest Gump
Forrest Gump
Inception
Funny Version's of Forrest Gump
Interpolation
Forrest Gump
Inception
Convince sommenns watch the movie efumnter version movis ambedding.
The novie heartwarming insphing
Dexcribe the movie cinterpoleted movis ambscdings. The moxie about man who ls not very amart han hig through many
who has very successful life, He has very loving nother who teashos him to be kind and always de his best. The man goss to hove many adventures and accomptish mony things, becomes college businessman, very funny and heartearming and is raminder that anyone can achieve
including being soldler Vietnam, mesting the prosident, and bacoming auccosetll bustteseman. Howwwerr he stways searching for sorrathing more, and he avenlusily finds it dream wonld, onything wants, He also leoms that can help other people by staring has dreams with them.
Sitter. The movie
great thnngs if they sot their mod to The movie also unique humor. hanny movil's entertalement value.
Figure 1: Decoded outputs of our model, ELM, hypothetical movie embeddings. Text blue red correspond to the textual and domain embedding portions of the prompt to ELM. Text in black corresponds to the output of ELM.
be futile. However, large language model (LLM) trained to interpret the embedding representation itself could handle such a task (see Fig. I for an illustration).
This paper introduces a novel framework to interpret domain embedding by leveraging the power of LLMMs (Devlin et al. 2018 Liu et al. 2021 Google et al. 2023). Our method seamlessly introduces embeddings into LLMs by training adapter layers to map domain embedding vectors into the token- level embedding space of an LLM, which in turn allows one to treat these vectors as token-level encodings of the entities concepts they represent. We train LLM a collection of tasks designed to facilitate the robust, generalizable interpretation of vectors in the domain embedding space. Our approach allows to engage in direct "dialogue" about these vectors, to query the LLM with intricate embedding data, and tease out narratives and insights from these dense vectors.
Our contributions are as follows. First, we formulate the problem of interpreting embeddings using LLMs. We then propose the Embedding Language Model (ELM), a novel language model framework which, using trained adapters, can accept domain embedding vectors as parts of its textual input sequence to allow interpretation of continuous domain embeddings using natural language. We also develop a training methodology fine-tune pretrained LLMs for domain- eenriidiiembddinggg interpretation. Finally, we test ELM by constructing 25 training tasks to allow interpretation of movie and user embeddings derived from the Moviel.ens 25M dataset (Harper & Konstan 2015). We demonstrate our trained ELM on variety of problems, including: generalizing CAVs as an interpretability method, describing hypothetical embedded entities, and interpreting user embeddings in a recom- mender systems by generating preference profiles. Our work bridges the gap between the rich data representations of domain embeddings and the expressive capabilities of LLMs.
INTEGRATING DOMAIN EMBEDDINGS WITH LARGE LANGUAGE MODELS
We assume a domain embedding ED : + W which maps entities in some vocabulary V (e-g.. set of known users and items) into latent space W C R" equipped with distance metric d: W x W + R., The embedding vector ED(v) of any v E V is typically a representation used in some downstream task, and is trained to encompass the latent features of necessary for this task. For example, in a recommender system, collaborative filtering (CF) (Salakhutdinov & Mnih 2007 Koren al. (01T) is often used to generate embedding vectors of users and items s.t. the dot product (or cosine similarity) of En(w) and (n is predictive of user n's affinity for item i. In
We use the term "domain embedding" to differentiate the external embeddings to be interpreted from those captured by an L.M at various layers of the network.
2

---

Published as a conference paper at ICLR 2024
Describe a viewer would the movie <embedding>
Qulput
:
ELM
Task n-1: Write a description of the movie <embedding> Task n: Write a posilive review for the movie <embedding>
Task n-1 Output
Task Oulput
hypothekkal entrediing
ELM
Pretrained Language Model Mg (GPT, BERT, PALM, etc.)
RitCompare Ratohej Bu (mmossaa)
Fix (ifo Lsnovie 1913 Batandi Ra comorvt
comovie 2011 Matill
Movie Embedding Space
Nu (norie 11
Inovte
Figure 2: We train ELM using n. tasks that incorporate embeddings as tokens in language. Specifically, to incorporate the domain embedding space W. we enhance the pretrained LLM M with an adapter model EA W 2 to create new language model MMa. ensuring tokens and embeddings are mapped to shared space. While Eu projects language tokens to the adapter E learss to project domain embedding vectors ED from embedding space W to the same space, 2. The resulting sequence is input to pretrained language model Mo. which can be further fine-tuned (see Sec. 2
Table I: Glossary
W Domain embedding space
d
Domain space metric
Token (language) embedding space
Tooken space
LLM embeslding
M
Pretrained LLM
FA
Adapter (W - Z)
MELM
Emhhediig Language Model
Domain embodding
Mo
LLM excluding input embedding layer
image classification, Ep might generate embedding representations useful for object detection (Zou et al. 2023). while search/information retrieval, Ep might embed both queries and documents in W to measure document's 's relevance to query (Zuccon et al. 2015 Luan et al. 2021).
We assume a pretrained I.I.M, . : XH XH,, which maps (length H) sequences of language tokens E & into another. Modern LLMs are composed of two parts, . = (Ef M.). The first part an embedding layer, Es : X + Z (not to be confused with the domain embeddings ED), which maps tokens from the domain embedding). The second part is the the dense model, My : 2 + X which maps sequence of token embeddings to sequence of tokens (e.g., transformer, Vaswani et al. 2017)).
X to their respective token embedding representations
Z (as distinct
We now define the Embedding Language Model (ELM), a framework which allows one to train a model using textual prompts mixed with continuous domain embedding vectors (see Fig. 2 for illustration). ELM incorporates domain embedding vectors using adapter layers to interface with text-only LLM. We note that we are not learning Ep itself, but rather wish to interpret In other words, given an embedding vector (v). we wish to train an LLM to meaningfully engage in discourse about the entity represented by (D(D) even for a hypothetical entity e.
Problem Formulation. We assume access to a domain embedding space (W, d), and to pairs (v, Ep for training as explained below. Note that we do not assume access the mapping Ep (except for empirical evaluation). Furthermore, our technique is agnostic to the type and source of (W.d), d). Finally, we assume access a pretrained text-only L.LM M.
To interpret embedding vectors, we require some semantic information about the entities to which they correspond. To this end, let be set of tasks, where each task E is specified by some mapping from joint input sequences s E (X' U WW" to distributions over output sequences o E x Intuitively, task t captures some form of semantic information about the entity represented by w E W, e.g., the plot of a movie, the preferences of a user, etc. The set tasks T should be diverse enough allow the ELM to extract semantic information from W to support interpretation, and
In practice, the dense model outputs logits over tokens, which are sampled by a decoding procedure. Mooe generally, can define output sequences in (X U W) well; leave this for future work.
3

---

Published as a conference paper at ICLR 2024
ideally support generalization to related tasks. For any t T, we assume access to set of training instances (s, o) drawn from some distribution P. (t itself may be drawn from Pr). For example, a task to compare two movies might use secen/domaineembedeee and output a textual comparison of the two movies represented by the input embeddings. In general, this does not require knowledge of the item which corresponds the input embedding vector (v), but in practice, one will use v to form target output training sequences.
imput sequences as shown in in/Fig. 2]
Architecture. To account for the embedding space W, we augment the pretrained . (E Mo) with an adapter model EA W + 2 to create a new language model МEιМ = ((En x EA)= Mo). Here, Eu is the usual embedding layer, which maps tokens representing textual imputs to Z, whereas EA maps embeddings from the latent metric space (W, d) to Z. Importantly, Eg and EA map tokens and embeddings (respectively) to a common space (see Fig. 2).
Training Procedure. Given a downstream loss function L, we can differentially optimize the model MEIM with parameters 9 over the tasks in T by solving arg ming oo, [Co(MMMM(ss o) Aside from maximum likelihood-based training, we discuss optimizing MELM using reinforcement learning from AI feedback (Lee et al. 2023 in Appendix H
Training continuous prompts poses challenges due to constraints over the pretrained token embeddings, which may result in convergence to local minima (Liu et al. 202T). Intuitively, as the pretrained embedding layer Eg readily maps to language space, the newly initialized embedding layer Eλ may require numerous updates to map to a similar space. To overcome this, we divide training into two stages. In the first stage, we train the adapter Ελ tasks in T by keeping all other parameters (Eo. Ma) frozen. Since My is pretrained, the learned first-stage mapping from W to Z improves convergence in the next stage. In the second stage, we fine-tune the full model by training all parameters (Eo. Mo. A)) To increase efficiency, we can update a smaller number of parameters in alterattnn fashion (Hu et al., 202I). We found that two-stage training essential for convergence of MELM (see Appendix D for further discussion of the two-stage training procedure).
EXPERIMENT DESIGN
In this section, we describe the datasets, tasks, and evaluation methods used to train and validate ELM. We use the MovieLens 25M dataset (Harper & Konstan 2015), which we enrich with textual descriptions by generating a large corpus of text using a PaLM 2-L (Unicorn) LLM (Google et al., 2023). We use two different forms of embeddings: behavioral embeddings and semantic embeddings. which we describe below. We also adopt various evaluation techniques for assessing the quality of ELM's outputs on test data, including qualitative human evaluations and specific consistency metrics.
3.1 DATA
The MovieLens 25M dataset contains 25 million ratings (1 to 5) of 62,423 movies and 162,541 users.
Domain Embeddings. We create embeddings for both users and movies in the MovieLens dataset. We consider two types of embeddings. The first are behavioral embeddings. trained based solely on user ratings of movies. More generally such models can reflect any behavioral interaction between users and items, but use no direct semantic information. To train behavioral embeddings, we use matrix factorization (MF) computed using weighted alternating least squares ( (WALS) (Hu et al., 2008), such that the dot product
(wh, wm) of the embeddings of user and movie m is predictive of user u's rating for movie m We train MF using movies that have at least five ratings.
Semantic embeddings are the second type, and are generated using textual descriptions of movics. a pretrained dual-encoder language model (DLM) similar to Sentence-75 (Ni et af. 2022a) and generalizable T--hased dense retrievers (Ni et al. 20226). More specifically, we concatenate plot descriptions and reviews for each movie, and input these to the DLM. We then average the resulting output vectors to generate the semantic embeddings. We denote movie m's semantic embedding by w...
Specifically,
$ Moore expressive adapters that map vectors in W to length 1>> sequences are left for future work.
'Other CF methods could be used, e.g., dual encoders (Yi et al. 2019 Yang et al. |020). but our approach is agnostic to the precise method used generate hehavioral or semantic embeddings.
4

---

Published as a conference paper at ICLR 2024
Training Data and Tasks. We test two versions of ELM, one that interprets semantic embeddings of movies, and a second which interprets behavioral embeddings of users. For the first model, we use a variety of tasks test the model's ability to extract reasonable semantic information from the domain embeddings. More specifically, we construct 24 movie-focused tasks using pretrained Pal.M 2-L model. For each task, generate training data by prompting the PaL.M 2-I. with movie's title and additional task-specific information (c.g., writing review, listing characteristics, comparing movies, see details below). We then use PaLM 2-L's generated output as training targets for ELM. As for training inputs, we provide ELM with the same prompts used in PaLM 2-L., where the title of the movie m is replaced by its semantic embedding w We emphasize that ELM does not receive any information about the movie (including its title), apart from its semantic embedding vector.
Our 24 movie tasks include single movie semantic tasks, such as describing a movie plot or sum- marizing movie; single movie subjective tasks, such as writing positive or negative reviews for movie; and movie pair subjective tasks, such as comparing characteristies of movies. [Appendix provides a complete description of all 24 tasks, including the prompts used, and sample outputs.
Finally, we train an additional model to interpret user behavioral embeddings, by generating user preference profiles. For this task, we generate a textual summary of user's preferences as follows: (a) we sample five positively rated and five negatively rated movies for each user in MovieLens 25M; (b) we then prompt PaLM 2-L to describe (in ten bullet points) characteristics of a person who likes the first five movies, but dislikes the second five. This resulting summary is used as training output for ELM on the user-profile generation task. The resulting task is to generate such user profile from a user's ' behavioral embedding w (i.c., interpret the domain embedding). Similar to the movie tasks, no other information about the user, apart from its embedding, is provided to ELM. We refer the reader to Appendix E for more details and examples.
3.2 EVALUATION METRICS
We employ several forms of evaluation for ELM. First, given the inherent subjectivity of some tasks (e.g., movie reviews), human raters play a key role in gauging output quality. We ask raters to assess ELM's output w.r.t. its consistency with a movie's plot, its linguistic coherence, and overall task quality. Each output is rated on a scale from 0 (completely irrelevan/inccoheeent) to (highly relevant/coherent). The average rater's score provides a holistic assessment of ELM's performance.
Second, since our goal is interpretation of embeddings spaces, and should allow the generation of descriptions of domain embedding vectors for which no entity exists in the domain vocabulary (e.g., hypothetical movies), we evaluate the consistency of ELM's 's generated text with the underlying domain embedding, well as ground-truth data used to train the domain embedding. To achieve this, we introduce two consistency metrics; namely, semantic and behavioral consistency.
Semantic consistency (SC) compares the semantic embedding of generated text with the original embedding that produced that text. In our movie tasks, this metric is computed by re-embedding ELM's output into the same semantic space used to embed the original movies. Formally, we define the semantic consistency of an output generated by an embedding w" E Wy
SC(o, w²) = d( w²).
where P(o) projects the output to the domain embedding space W and d is the distance metric in W (or alternatively, similarity score); see Appendix B. Fig. 6 for illustration The utility of SC is grounded in the expectation that if the generated output o holds true to its source w", then its embedded representation should not deviate substantially from its generating embedding vector. Hence, high SC suggests ELM has well-aligned domain embedding vectors with their (task-specifer descriptions. Importantly, this approach not tied to examples in the data, providing the flexibility to measure the semantic consistency of hypothetical entities.
second form of consistency, which we term behavioral consistency (BC), measures the ability to use ELM's 's output text to make good behavioral predictions. For example, in our user-profile task, we can test the ability of ELM's generated user profile to predict the movie preferences of a user corresponding to the input behavioral embedding. This in turn would suggest the extent to which the user profile captures the information implicit in that domain vector (i.e., the latent user preferences).
'In our experiments we the same DL. serve as P, i.c., to re-embed output text into semantic space.
5

---

Published as a conference paper at ICLR 2024
Table 2: Test Scores for 24 movie tasks. We use cosine similarity as our metric for SC. and Spearman rank correlation coeff. for BC. For human evaluation, we show normalized scores between (strongly disagree) and I (strongly agree). Table is divided into two sections: single movie tasks and two movie tasks. Tasks with nn abbreviation use nearest neighbor movies in semantic emhedding space. A comparison of semantic and hehavioral consistency metrics demonstrated in
Consistency Metrics
Human Evaluation (100 raters)
Task (Appendix
Semantic (Cosine)
Behavioral (Rank Corr)
Consistency with Plot
Language Comprehensiveness
Task Quality
summary
0,91
0,94
0.87
0.87
0.85
positive review
0.94
0,81
0.9
0.9
0,89
negative review
0.93
0.79
0.93
0,96
0.96
neutral review
0.92
0.76
1
1
0.56
five pos char.
0.91
0.8
0.84
0.92
0.95
five neg char.
0.89
0.8
0.81
0.92
0.89
long description
0.91
0.8
0.89
0.86
0.89
funnier
0.88
0.81
0.75
0.78
0.68
sadder
0.91
0.82
0.59
0.83
0.81
scarier
0.9
0.84
0.69
0.74
0.95
improve
0.91
0.82
0.85
0.89
0.87
movie to viewer
0.89
0.8
0.89
0.83
0,86
pitch
0.96
0.8
0.96
0.95
0.94
criticize
0.89
0.77
0.88
0.7
0.79
convince
0.92
0.96
0.98
0.93
0.77
convince2
0.9
0.88
0.89
0.86
0.87
convince3
0.88
0,86
0.7
0,67
0.74
dissuadel dissuade2
0.88
0.85
0.88
0.89
0.87
0.88
0.87
0.79
0.8
0.79
similarities interpolation
0.86
0.83
0.37
0.4
0.38
0.89
0.85
0.82
0.84
why like na
0.83
0,48
0.84
0.84
0,91
diff than nn common with
0.82
0,48
0.61
0.79
0.64
0.77
0.46
0.91
0.86
0.93
To achieve this, we assume access to some "off-the-shelf" language-baset retriever/ranker which, given profile, can rank set of candidate movies. Then, given fixed set of target movies, I, and letting ou the ELM-generated user profile given domain embedding wh we define
BC u) = RankingScore((R... p(ow)),
where p(o.) are the ranks of the movies in I for user u provided by the language-based ranker, and R2 [r(u, m): E Z are the (MovieLens) ground-truth ratings for I provided by u (converted to rankings); see Appendix Fig. 7 for an illustration 77 Here, RanlingSSoree is any con-cooparisonn metric which compares p( the grounddtruth ranking R,, Examples include NDCG (Burges et al. 2005) and the Spearman runk-correlation coefficient. Finally, we measure behavioral consistency of movies in fully analogous fashion (see Appendix B for formal definition).
EMPIRICAL ANALYSIS
We evaluate ELM on our movie tasks and the user profile task, described Sec. 3.1 raters, and semantic and behavioral consistency measure performance. We analyze test results on both real entities (i.e., held-out movies) and hypothetical embedding vectors not the data, to test ELM's ability to interpolate between movies users, well as extrapolate movie or user attributes.
using human
We train two models: one (joint) model for the 24 movie tasks, and another for user profiles. For the movie tasks, use 1000 randomly sampled examples for test and the rest for training. For the user profile task we use an 80/20 training/test split. Both models are fine-tuned using pretrained
In our experiments use the dual-encoder DLM to serve p. by emhedding test movies and user profiles and using their similarity for scoring/ranking.

---

Published as a conference paper at ICLR 2024
Consistency Interpolated Movies and Users
PaEM 2-L. (Toxc Only
CPT (Toxt Only)
Condistnnyy
summary
postiive review
negative review
neutral revies
five pos characterisicc
long description
pitch
user profile
Figure 3: Results show consistency results for movic and user interpolations using ELM, compared to one-shot decoded results of text only LLMss. We show semantic consistency (cosinc) scores for movies and behavioral consistency (NDCG) scores for user profiles.
five pos characteristics
long description
Semantic Consistency
0.54
Semantic Consistency
romantic
0:33
0.94
funny
animated
0.1
0.2
terrifying
0:92
0.90
interpolation
0.09
0.89
Figure 4: Plot depicts SC scores (using cosine similarity) for interpolations of movies with movie "Forrest Gump", as well as extrapolation of CAV attributes semantic space.
PaLM 2-XS (Otter). We use a two-layer MLP for the domain- embedding adapter layer EA, and two-stage training, described in Sec. Particularly, for movie tasks, we run the first stage of training for 20k iterations, and then the second stage for another 300k iterations, using a batch size of 32 (i.e., roughly seven epochs). We use similar procedure for user profiles. We found training in two stages to be essential for convergence of ELM (see Appendxx D for further details).
Test-item Evaluation. Table 2 shows consistency (SC and BC) and human rater results for all movie tasks on the test set. For each task, we ask 100 human rater to rate the consistency and comprehensiveness of our results w.r.t. the ground-trute truth data. We also ask qualitative task-specific questions (Appendix ddescribes rater instructions in detail). Rater evaluations show that ELM is able to generalize well across most tasks. Moreover ELM maintains high semantic consistency across all tasks and good behavioral consistency for most. We that performance tends to be two-movie tasks, due to the additional challenge of "conversing" about similarities and differences between distinct embedding representations. We emphasize the importance of our consistency metrics, as they reflect how well ELM adapts to the domain embedding space W.
Communicating Novel Entities. We test ELM's capability meaningfully extrapolate from existing movies or users to interpret "gaps" in embedding space (e.g.. embedding vectors of hypo- thetical movies for which we predict large audience). Note that such hypothetical vectors have no ground-truth data for evaluation; nor is it clear how to evaluate hypothetical embeddings using text-only LLMS (e.g., Google et al. (2023); OpenAl (2023)). For instance, there no obvious way to instruct a text-only LLM to describe
arbitrary vector in W. Even if we were to ask text-only LLM to interpolate two existing entities (e.g.. movies), the result will not necessarily be consistent with the domain embedding space we wish to interpret, nor can one in general direct text-only LLM to an arbitrary point using interpolation between a small number of existing entities.
'Raters were paid contractors. They received their standard contracted wage, which is above the living wage in their country of employment.
7

---

Published as a conference paper at ICLR 2024
Behavioral Consistency for User CAVs
6.0
funny
041
scary
action filled
6.0
Intense
NDCG
0.5
cartoonish
real stic
8.9
scrious
845
Ground
ficiionaized
Truth
649
far-fetched
playful
Random
gory
839
terrifying
Plot depicts BC scores (using NDCG with the predicted CF model ratings) for user profile decoded outputs, where user embeddings extrapolated in different CAV directions. Dotted lines show two baselines (for 0): the random baseline, which rates movies randomly, and ground-truth baseline, which uses the ground-trute truth user profile data to compute BC (i.c., BC of the ground-trutt truth training data).
Figure 5:
Behavioral embeddings of, say, users are even more challenging for text-only LLMs. This puts additional emphasis on the need evaluate such LLMs using embedding consistency metries like SC and BC. That said, to ground our results, we do not consider arbitrary hypothetical points in embedding space. Instead, we evaluate our model using interpolation of existing entities, well shifts of such entities in specific directions (see Generalizing CAVs below). Nevertheless, ELM's ability to interpret hypothetical embedding vectors not constrained to these choices.
We first assess consistency metrics on interpolations of two movies or users, Fig. 3 compares the performance ELM with state-of-the-art text-only LLMs; namely, PaLM 2-L and GPT 4. For movies, we linearly interpolate (with mixture coefficient = 0.5) embedding vectors of movies in the training set with the classic movie "Forrest Gump (1994)", and provide the interpolated embedding to ELM. For users, interpolate random pairs of user embedding vectors from the training set. For a fairer comparison, we provide the text-only LLMs with a ground-truth examples of each task's output, and prompt them to generate the task output. Particularly for users, provide the text-only LLMs with ground- -ruth user profiles, and instruct them to interpolate between them (see Appendix for the precise prompts used). We measure semantic consistency for movie tasks and behavioral consistency for the user profile task. We found that text-only L.MS are significantly less consistent than ELM. This is not surprising. describing domain embedding space manifold to text-only LLM is especially challenging. Moreover, ELM is able to generate high-quality consistent responses to our tasks. We provide further qualitative results in Appendix G
We next test ELM'S consistency as we vary the degree of interpolation between movies, by varying the mixture weight a. Fig. shows semantic consistency (magenta line) for two of our movie tasks. We find that outputs are highly consistent across all degrees of interpolation values, with tendency for lower consistency scores for embeddings that are farther away from the two movies. This result unsurprising, as we expect performance to degrade as it moves away from "real" data points. Also note these are linear interpolations, though the true embedding manifold may be highly non-linear.
Generalizing Concept Activation Vectors (CAVs). Finally, we test the interpretation of vectors domain embedding space by extrapolating existing entity embeddings in specific directions. To accomplish this, train CAVs Kim et al. 2018) using attribute tags in MovieLens 25M, provided by movie raters. These CAVs can be treated as noisy, "soft" movic attributes, such as 'funny'' provokinproooiig,,, etc. The CAV corresponding to specific attribute provides direction in domain embedding space that represents an increase in that attribute's 's value. We use these to extrapolate movies and users in specific CAV directions (e.g., make existing movie funnier, or tweak a user profile to prefer more cartoonish movies).
"terrifying."
In Fig. we that semantic consistency of movie extrapolations generally decreased extrap- olated further in CAV directions. This is expected as extrapolations outside the data manifold should
We train CAV' using the procedure of Göpfert et al. (2022) and the dataset of Balog et al. 2021 which refer for details. Specifically, use linear, objective rather than non-linear and/or subjective CAVs.
8

---

Published as a conference paper at ICLR 2024
induce higher error. Similarly, Fig. shows behavioral consistency results for CAV extrapolations of user profiles. These results show attribute extrapolations to be behaviorally consistent for different values. Perhaps surprisingly, user profiles show greater behavior consistency as we increase the degree to which they move in a CAV direction. We hypothesize this is due to user profiles becoming "more predictable" in our CF model by making user preferences more extreme. For instance, make a user profile gravitate to funnier movies, preferences tend to degenerate with near-exclusive focus on funny movies (thus rendering rating prediction and especially relative rating prediction more accurate). This in turn increases BC. We provide further qualitative results for interpolations and CAV-based extrapolations in Fig. We refer to Appendix Gforr more qualitative results.
RELATED WORK
We review a selection of related work. The use of embeddings in NLP is well-established, with seminal work like Word2Vec (Mikolov et al. 2013) and GloVe (Pennington et al. (2014) using dense
vectors to represent words in a way that eaptures semantics. These approaches serve as the foundation for many models in NL.,, e-g., contextual (Bengio et al., 2000 Collobert & Weston 2008 Radford cct al. 2018) and deep-averaging networks (lyyer et al. 2015). Bowman et al. (2015) show that interpolating sentences in embedding space can generate novel sentences that smoothly blend the
"ideas" in the original sentences. Such NLP models provide some "immediate" interpretability.
Our ELM approach supports interpretation of arbitrary embedding spaces using natural language. In this sense, ELM can be seen as learning an encoder-decoder model (Raffel et al.. (2020). for which the encoder adapts the domain-embedding
the token-embedding space, and the LLM acts as decoder from the "semantics" latent in the domain embedding to natrual language. As such, our work is closely related to research on translation (Vaswani al. (017) and multi-modality (Cho et al. 2021 Yu et al. (2022) that rely on encoder-decoder architectures. A key difference with ELM lies in the fact that encoder inputs (domain embeddings) are not igenerally interpretable; by contrast, in translation and multi motal tasks, labeled data can be explicitly acquired from humans.
The use of continuous vectors as prompts for language models can provide flexibility in controlling outputs, improving performance on downstream tasks. Examples include: prefix tuning (Li & Liang 2021 Lester et al., 2021; l'simpoukelli et al., 202T), which prepends sequence of task-specific vectors to the input, while keeping LLM parameters frozen; and hard-soft prompt hybrid tuning (Liu ct al. 2021 Han et al. 2022), which inserts tunable embeddings into a hard prompt template. These methods primarily guide model behavior for downstream tasks, rather than interpreting embeddings, especially those derived from different data sources (which we want to understand, not create).
CAVs (Kim et al. 2018) and visual techniques such as t-SNE (Van der Maaten & Hinton 2008 or UMAP (MeInnes et al.] 2018) have been used for visualizing and interpreting embeddings. Nevertheless, these methods often provide static insights or visual interpretations, and are often hard or even impossible to capture in a low dimension (Chari & Pachter 2023) Seshadhri et al. 2020). ELM, by contrast, seeks to enable dynamic, language-based exploration of the embedding space, "visualizing" it by narrating embeddings with language. Other work has focused on interpretability
analysis and toolkits for language and sequence models (Belinkov al. 2020 Sarti et al. 2023 Bills
et al. 2023 Räuker et al. 2023), but these methods are largely orthogonal to our work.
6 CONCLUSION
We have presented ELM, a novel language-model framework for interpreting domain-embedding spaces. We assessed ELM's capabilities on a range of movie tasks and user-profile task, bench- marking with both human evaluations as well as two novel quantitative metrics: namely, semantic and behavioral consistency. Our results show that ELM generalizes well to embedding vectors in a test set, and aligns well with human-rater expectations. We also find that ELM is adept at handling the nuanced challenge of describing novel entities, with better semantic and behavioral in tasks that interpolate between entities, compared to state-of-the-art text-only LLMs. We also demonstrated ELM's proficiency in generalizing CAVs, underscoring its ability to meaningfully manipulate and interpret domain attributes of movies and user profiles. Taken together, our results suggest that ELM offers a powerful, flexible mechanism for understanding, navigating and manipulating complex emhedding representations.

---

Published as a conference paper at ICLR 2024
REFERENCES
Krisztian Balog, Filip Radlinski, and Alexandros Karatzoglou. On interpretation and measurement of soft attributes for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '21), pp. 890-899, 2021.
Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. Interpretability and analysis in neural nip. In Proceedings of the 58th annual meeting of the association for computational linguistics:
nutorial abstracts, pp. 1-5, 2020.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language mod- els. URL https///oeeaippbblic blob. core. windows. net/neuron seepheeeneepplldeeeepeeeeeeee haate accessed: 14. 05.05..22 2023). 2023.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio, Generating sentences from a continuous space. arXiv preprint arXiv: 1..06449 2015.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learnnng to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 89-96, 2005.
Tara Chari and Lior Pachter. The specious art of single-cell genomics. PLOS Computational Biology, 19(8):e 1010111855000 1288, 2023.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- languaag tasks via text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1931-1942. PMLR, 18-24 Jul 2021. URL cho2la.hthe
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167, 2008
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 10X.iv://10.04440 2018.
Google, Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey. Yanping Huang, Kathy Hells-Hel Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choo, Aakanksha Chowdhery, Clément Crepy. Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li. Music Li. Wei Li. YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polazov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang. Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng. Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. M technical report, 2023.
10

---

Published as a conference paper at ICLR 2024
Christina Göpfert, Yinlam Chow, Chih-wei Hsu, Ivan Vendrov, Tyler Lu, Deepak Ramachandran, and Craig Boutilier. Discovering personalized semantics for soft attributes in recommender systems using concept activation vectors. In Proceedings of the ACM Web Conference 2022, pp. 2411-24211 2022.
Tuomas Haarojj, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp- 1861-1870. PMLR, 2018.
Xu Han, Weilin Zhao, Ning Ding. Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192, 2022.
Maxwell Harper and Joseph Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (fiis), 5(4):1-19, 2015.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In Proceedings of the 8th IEEE International Conference on Data Mining (ICDM 2008), December 15-19, 2008, Pisa, Italy, pp. 263-272, 2008
Mohit lyyer, Varun Manjunatha, Jordan Boyd-Graber and Hal Daumé III. Deep unordered composi- tion rivals syntactic methods for text classification. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume Long papers), pp. 1681-1691, 2015.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Proceedings of the Thirty-fifth International Conference on Machine Learning (1CML-18), pp. 2668,2677, Stockholm, 2018
Yehuda Koren, Steffen Rendle, and Robert Bell. Advances in collaborative filtering. In Recommender Systems Handbook, pp. 91-142. Springer, New York, 2011.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint
2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101. 0009000,, 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT understands, too. arXiv preprint arXiv:2103.10385, 2021.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. Trans. Assoc. Comput. Linguistics, 9:329 3459 2021.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 2018
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
11

---

Published as a conference paper at ICLR 2024
Jianmo Ni, Gustavo Hernández Åbrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-TS: Scalable sentence encoders from pre-treiin trained Text-to- Text models. In Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland, May 22-27, 2022, pp. 1864-1874. Association for Computational Linguistics, 2022a.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith Hall, Ming-Wei Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 9844- 9855, 2022b.
OpenAl. Gpt-4 technical report, 2023.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5551, 2020.
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadnield-Menell. Toward transparent ai: A survey interpreting the inner structures of deep neural networks. In 2023 IEEE Conference Secure and Trustworthy Machine Learning (SaTML), pp. 464-483. IEEE, 2023.
Steffen Rendle, Factorization machines. 2010 IEEE International conference on data mining, pp. 995-1000. IEEE, 2010.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning 250 million protein sequences. Proceedings of the National Academy of Sciences, L18(15)ee20122311 2021.
Ruslan Salakhutdinov and Andriy Mnih. Probabilistic matrix factorization. Advances in Neural Information Processing Systems 20 (NIPS-07), pp. 1257-1264, Vancouver, 2007.
Gabriele Sarti, Nils Feldhus, Ludwig Sickert, Oskar van der Wal, Malvina Nissim, and Arianna Bisazza. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 421-435, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 del-demo.40. 400 URL https: / ac aataahlogy..oo 2023.ac1-deme 40
Seshadhri, Aneesh Sharma, Andrew Stolman, and Ashish Goel. The impossibility of low-rank representations for triangle-rich complex networks. Proceedings of the National Academy of 2020.
Sciences, (117(1)156311155011
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1),, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed H Chi. Mixed negative sampling for learning two-tower neural networks in recommendations. In Proceedings of the Web Conference (WWW-20), pp. 441-447, Taipei, 2020.
12

---

Published as a conference paper at ICLR 2024
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling-bin baat-baas---ee bias-corrected neural modeling for large corpus item recommendations. In Proceedings of the Thirteenth ACM Conference on Recommender Systems (RecSys19), pp. 269-277, Copenhagen, 2019.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:226..00789, 2(3):5, 2022.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: survey. Proceedings of the IEEE, 2023.
Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In Proceedings of the 20th Australasian document computing symposium, pp. 1-8, 2015.
13

---

Published as a conference paper at ICLR 2024
Table short description of each of the Amazon tasks.
description positive review negative review neutral review endorsement
Description of product.
positive review of the product (rating
negative review the product (rating 1-3)
neutral review of the product (rating 4)
An endorsement of the product.
user profile
Ten bullet points describing user based positive and negative reviews.
Table 4: Test Scores for Amazon tasks.
Consistency Metrics
Human Evaluation (100 raters)
Task (Appendix
Semantic (Cosine)
Behavioral (Rank Corr.)
Consistency with Item / User
Language Comprchensiveness
Task Quality
description
0.93
0.92
0.81
0.88
0.75
positive review
0,95
0.89
0.76
0.79
0.93
negative review
0.91
0,94
0.77
0.76
0.71
neutral review
0.96
0,82
0.8
0.79
0.7
endorsement
0.9
0.81
0.91
0.93
0.94
user profilc
0.89
0.87
0.84
0.77
0.75
A ADDITIONAL EXPERIMENTS ON THE AMAZON PRODUCT DATASET
We complement our experiments and demonstrate ELM on the public Amazon dataset (McAuley al. 2015), which consists of 9.35M items with textual descriptions, 20.9M users, 233. 1M reviews, and 82.83M ratings. We focus specifically on subset of this dataset in the category "Clothing, Shoes and Jewelry", which consists of 5.7M reviews for 1.5M products.
Domain Embeddings. Similar
setting in the Movielens dataset, create both semantic and behavioral embeddings. We train behavioral embeddings on users' rating data using the same method described in [Sec. 3.1 For semantic embeddings, DLM (see See. 3.1) encode concatenation of the item title, item description, item categories, and item features (which include item tags).
Training Data and Tasks. We test ELM's ability to interpret semantic and behavioral embeddings on set six tasks: item description, positive review, negative review, neutral review, item en- dorsements, and user profile. Item descriptions are taken directly from the Amazon dataset. For review data, we split the Amazon reviews w.r.t. to positive, negative, and neutral reviews. To ensure the datasets are of comparable size, we use scores of 1-3 to represent negative reviews, score of 4 to represent neutral reviews, and a score of 5 to represent positive reviews. To create the item endorsement task prompt PaLM 2-L with the item title and description and ask it to create endorsement for that product. Finally, for user profiles, we use a similar procedure as Movielens. Specifically, describe user using positive and negative reviews they have posted, and ask PaL.M 2-1. to describe a user with such reviews ten bullet points. We note that, similar to the Movielens tasks, the Amazon training data does not consist of any information about items or users except for the behavioral or semantic embedding information. Table summarizes the set of tasks we use.
Results. Table shows consistency (SC and BC) and human rater results for the Amazon dataset tasks on test set of 1000 examples (randomly selected). For each task, we ask 100 human raters to rate various aspects of the outputted result, such as consistency with the true item, comprehensiveness of the language, and other task-specific quality evaluations. Simliar to the Movielens experiments, our results show that ELM is able to generalize well across all tasks. Moreover ELM maintains high semantic and behavioral consistency across all tasks.
We conduct further detailed human evaluation studies on the Amazon tasks. For this evaluation we ask human raters evaluate both ground-trui truththtthhhhho data well results generated by model. For example, we ask human raters to evaluate the quality of a review in the real data w.r.. the ground-truth item description. This evaluation enables to obtain baseline for the model's scores. Below
14

---

Published as a conference paper at ICLR 2024
Table Human evaluation scores on generated data based on form questions (Q). Table shows average scores of 100 human raters.
Task
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
description
0.77
0.7
0.86
0.71
0.73
0.7
0.81
0.9
0.9
0.9
Task
Q1
Q2
Q3
Q4
positive review
0.91
0.92
0,92
0.98
negative review
0.75
0.72
0.71
0.75
neutral review
0.85
0.8
0.79
0.7
Task
Q1
Q2
Q3
endorsement
0.91
0.93
0.93
Table Human evaluation scores on ground-truth based form questions (Q). These evaluations are on test data, not generated output. Table shows average scores of 100 human raters.
Task
Q2
Q5
Q6
Q7
Q8
Q9
Q10
description
0.68
0.64
0.85
0.83
0.82
0.83
0.84
0.74
0.73
0.73
Task
Q1
Q2
Q3
Q4
positive review
0.85
0.76
0.79
0,92
negative review
0.75
0.77
0.74
0.71
neutral review
0.84
0.79
0.81
0.7
present examples of forms that were presented to the raters, with task-specific questions. In Tables and we show the average human rater answers to these questions. We find that our generated results correctly correspond test items, and perhaps surprisingly, often surpasses the original ground-truft data quality, viewed by the human raters.
Item Description Task (Rater Form Example)
*BELOW ARE TWO DESCRIPTIONS OF AN AMAZON PRODUCT TIILED "TIMEX WOMEN'S T2P0902M LEOPARD PAITERNED LEATHER STRAP *ATCH
**IIST DESCRIPTION:*
Mens Hipster Wallet with Genuine Leather. Perfect wallet to keep everything organized. With a sloe for cash and ample credit card space, this item is great! It even has an casily accessible window! 8+ Card slots built into wallet help to keep this slim.
SECOND DESCRIPTION:
mens keeps pocket, this wallet keeps everything together. Keep with your essentials with this wallet. Please note: This item shipped vin USPS First Class Package and usually takes 5-7 business days to arrive. If you would like item faster, plense choose different shipping method such UPS Fedlix Express..
organized
multiple card window
On scale of 5,, how much you agree with following:
I Strongly Disagree
Strongly Agree
Question Both descriptions are equivalent.
Question The descriptions similar.
Question 3: Both descriptions could correspond to the same product.
Question The first description of high quality.
Question 5: The first description is coberent.
Question The first description comprehensive.
Question 7: The first description selevant the item title.
Question The second description coberent.
Question The second description comprehensive.
Question 10: The second description relevant to the item title.
15

---

Published as a conference paper at ICLR 2024
Positive Review Task (Rater Form Example)
*"BELWW YOU WILL BE PRESENTED WITH A DESCRIPTION FOR AN AMAZON PRODUCT TITLED "HANES WOMEN'S 10 PAIRS CUSHIONED SOLE LOW CUT SOCKS SZ: 9-11 FITS SHOE 5-9". YOU WILL THEN BE ASKED TO RATE REVIEW OF THIS PRODUCT. 4*
*DDECRRPPTIN PRODUCT:*
mad can rough. Wearing socks help make smoother. Provides hetter fit with spandex Soft, breathable cotton knit helps you keep your cool. Plus. cushioned foot bottom soothes your every step. Reinforeed grey heel and toc resist wear tear.
"BBELWW IS REVIEW FOR THE PRODUCT ABOVE.*
I am 7.5 to 8 medium weight woman with a high instep. These socks fit perfectly and are not too tight around the top. They stay up well and are comfortable to wear all day. The colors and patterns are would definitely buy these again. did not find them to be that slippery on floors. Maybe was wearing them right after they arrived and they need little breaking in. I do think they are better than most athletic socks in that they are more like thin mid weight sweater than a stocking. with jeans leggings.
They good to
happy find that they good quality and made China. The label says they are Shrink To Fit which means they use clastic in the toe and heel. That should help them keep their shape. I I
will follow up with
these socks for mooths wear them to work 5 days week. I wear them with dress shoes most of the time. They have held up well with little wear on the heel.
have
them for while. UPDATE: have
On scale of 1-5, how much you agree with the following:
1 Strongly Disagree
5 Strongly Agree
Question reasonable assume review would appear this product.
Question 2: The product review is consistent with the product's description. Question The product review is coherent.
Question The product review is positive.
Negative Review Task (Rater Form Example)
**BLLOW YOU WILL BE PRESENTED WITH A DESCRIPTION FOR AN AMAZON PRODUCT TITLED "PT261 WOMEN'S ATHLETIC PERFORMANCE LOOSE FORM RACER BACK FITNESS TOP". YOU WILL THEN BE ASKED TO RATE A REVIEW OF THIS PRODUCT.**
"DESCRIPTION OF PRODUCT:**
Women's Perfornance Tech Racer Back sign will provide ultimate comfort and unrestricted shoulder and arm mobility. Great for the gym, outdoor activities, or any sport.
muisture wicking athletic lop. This de-
BELOO IS REVIEW FOR THE PRODUCT ABOVE.**
I'm 5'4". 135lbs ordered small/mediom and like Great as a night gown or lounging piece. do love the material. Soft and comfortable. would order medium if you are nermal weight smaller. ended up giving mine my year daughter did enjoy the time had with it. :-) Peace out. I hope this review was helpful. I don't usually write reviews, but I was really disappointed with the size of this shirt. It's shame bocause it's a great material. Peace out. hope this review was helpful. :-) Peace out. 1 don't usually write reviews, but really disappointed with the size of this shirt.
Wanaaay big. would order
loves it. would
size sure.
On scale of 1-5, much you agree following:
I Strungly Disagree Strongly Agree
Question 1: It is reasonable to assume review would appear this product.
Question 2: The product review is consistent with the product's description.
Question 3: The product review is coherent.
Question The product review negative.
16

---

Published as a conference paper at ICLR 2024
Neutral Review Task (Rater Form Example)
*"BELOW YOU WILL BE PRESENTED WITH A DESCRIPTION FOR AN AMAZON PRODUCT TITLED "JERRISAP: PAREL NEW CINDERELLA DRESS PRINCESS COSTUME BUTTERFLY GIRL". YOU WILL THEN BE ASKED TO RATE A REVIEW OF THIS PRODUCT
"DESCRIPTIN PRODUCT:
JerrisApparel Cinderella Dress Ella Princess Costume Rutterfly For 2-9 Years Detailed Size Information(in Tnches) Notice: There do exist 1-2 inches differences because of different measuring methods. Please check the size info carctully. Thank you for your understanding. 3 Years: recommended height:39-43.3 chest/bust measurement 19,5-24.5 waist measurementt 18-22
dress length:32 Years: recommended hright:43.3-472 chest/bust meusurerment:20.5 2-25-55.5555 waist measurement:
dress
length:34 Years: recommented height:47.2.55 chest/hust measurement:21.55 5:526.. waist measurement:20-
dress length:35 measurement dress length:36 Years: recommended heigh::55.5 59 chest/bust measurement:23355 waist measurement:22. 26 dress length:38 8 Years: recommended
6 Years: recommended height:51-55 chest/bust 2ccsureement:22.5-775.
heighr:59-62 chest/bust measuromemt:27-33 waist measuremen: dress longth:40
*BBLLWW IS REVIEW FOR THE PRODUCT ABOVE.*
My daughter is and half and wears a size St. I ordered the 4-6 and was too big. The skirt was huge. I should have had to
gotten smaller size but nonetheless, she loved dress. So would definitely recommend this dress hut get smaller size. It is very cute and well made. My daughter loved it. I gave it stars because it really cute and well made but it was too big. would have given it stars had come a smaller size.
though bocause
big and already had similar
On scale of 1-5, how much do you agree with the following:
= Strongly Disagree
5 = Strongly Agree
Question I: It is reasonable to assume this review would appear for this product.
Question 2: The product review is consisteat with the product's description.
Question 3: The product review is coherent.
Question 4: The product review is neither positive nor negative.
Endorsement Task (Rater Form Example)
"BELOW
PRESENTED WITH DESCRIPTION FOR AMAZON PRODUCT TITLED "YOUYOUPIFA SILVER PARTIALLY HOLLOW TRANSPARENT DIAL STAINLESS STEEL BAND AUTOMATIC MECHANICAL WATCHES". YOU WILL THEN BE ASKED TO RATE AN ENDORSEMENT OF THIS PRODUCT.**
*"DESCRIPTION OF PRODUCT:*
Gender:Unisex Movement Description:AAtomme Mechanical Movement Case Size:43<40 mm / 1.69*1.57 inch Case Thickness:14 mm 0.55 inch Strap Width:18 8 mm 0.71 inch StzeleStainless steel bezel Sial: partially hollow transparent Casc:Satin & Polished finish case Case Transparent case back that the movement of the watch can been seen Crystal:High Quality Organic Crystal- Maximum Seratch Resisiant Strap-Stainles steel band Folder:Fold Fold-------- Claspp Weight:100 g Length:18.5 cm Water resistant Daily water resistant, please don't 't' put water
*BELOW IS ENDORSEMENT FOR THE PRODUCT ABOVE .0
This bcautiful, unisex watch with a silver partially hollow transparent and a stainless steel band. is an auto- matic mechanical watch, which means that does not require battery. The watch also water resistans, you wear the rain or while washing your hunds. The watch is 43*40 mm in size and 14 mm thick. The strap is 18 mm wile. The watch has stainless steel bezel and satin polished finish case. The back trasparent, you can see the movement of the watch. The crystal is high quality organic crystal, which is scratch resistant The strup is made of stainless steel and has a fold- over clasp. watch weighs 100g g and 18.5 cm long. This great watch for anyone looking for stylish durable timepiece.
On scale of 1-5, how much you agree with the following:
I= Strongly Disagree
5 = Strongly Agree
Question I:
The endorsement is consistent with the product's 's description.
Question The endorsement coherent.
Question 3: The endorsement is trying convince someone the product good for them.
Below we provide several examples of user profiles in the Amazon dataset, generated from positive and negative reviews. We used these profiles as training data for ELM to generalize to unseen user embeddings.
17

---

Published as a conference paper at ICLR 2024
Example of User Profile
Positive review;
Posilive review: travel around the country weekly for business and sometimes need down airport catch my plane; these are the perfect shoes for business attire as well comfort througouut week. highly recommend them!
Negative review:
I really wish bado't thrown all the shipping materials before trying these they would have been retured.. Shoc is very heavy, very wide for wide width and even with inserts, heel slip protectors, etc. I still can't use them. Goodwill has getten shoes.
User Profile:
The user business traveler who needs shoes that both comfortable and stylish.
The user profers shocs that are too heavy wide.
The user is willing to for high quality shoes.
The user is not afraid to try new brands or styles.
The user is willing to return shoes if they are not satisfied.
The user likely to donate shocs they do not wear.
user likely to buy
online.
The user likely to read reviews before huying shoes.
The user is likely to buy shoes from trusted retailer.
Example of User Profile
Positive review:
seems exactly like it was advertised. was pleased with this product. has nice design and the price good.
Negutive review:
The necklace attractive the under side of it shows and the pretty outside design inside toward my neck. The two chain artachments should be nearer one another instead 180 degrees apart. Then it would not turn up-side- down and insidc-our. attachments were on the top kind like Micky Mouse and will onder it, but am sending this one back.
poor design.
hang
shown. Instead
up-side-dowa
then the pendant would hang like supposed to. Please change the of the chain attachments
User Profile:
The user likes products that are as advertised.
The user likes products with nice design.
user products
affordable.
The dislikes products that have poor design.
The dislikes products that do
properly.
The user dislikes products that are upside down.
The user dislikes products that are inside out.
The user dislikes products that have chain attachments that are 180 degrees apart.
The user would like products have chain attachments that nearer one another.
user would products
top.
18

---

Published as a conference paper at ICLR 2024
Example of User Profile
Positive review;
The hat lits great! have large head, and have difficult linding hats The solt opposed scratchy, and after trying it on and wearing at while, it doesn't it slip off, or look like I shopped in the kiddie beanie section.
Negative review:
The hat fits great! I have large head, and have difficult time finding hats that fit. The yarn soft as opposed scratchl, and after trying it on and wearing while, it doean't 't slip off, look like I shopped in the kiddie beanie section.
User Profile:
The user is a woman.
The user has large head.
user has difficulty finding
The prefers soft yarn scratchy yarn.
The user does want hats that slip off.
The user dles not want hats that look like they are from the kiddie beanie section.
The user wears women's is size 11/12 or men's 's size 10 shocs.
The user does not want to wear socks in the summer.
user want mess with getting right
The thinks shees great.
Example 4 of User Profile
Positive review:
Purchased wedding. them!!
purchuse.
Negative review:
When ordered this dress thought 1 was going to get an open, breezy. A-line, asymmetrical dress with a lot of move- Instead the dress basically straight with awkward and uncomfortable flaps impression that the dress is open and would have a lot of movement. What this looks like on me is a straight fitting, body bugging Perhaps
each sidc) give the
ushirt dress Adding to this unfortunate and unflattering design that the added punels also serve add los of weight and coupled with the narrowness the fit, serves to make dress uncomfortable. looked see could simply remove scams from the flaps open the dress up. but that would serve ruin the dress, would that, for it unwearable and too uncomfortable even sleep in. will say that is cheap enough that have not returned it because that leels like more hussle than what originally paid for very disappointed.
wings sewn
description,
not how interpreted the
Profile:
The user likes shoes
again.
The likes shoes that are confortable.
The user likes shoes that are stylish.
The user likes shoes that are afTordable.
The user likes dresses that are open and breezy.
The user likes dresses that A-line and asymmetrical.
user
movement.
user
not tight or body bugging.
The user likes dresses that made of lightweight fahries.
19

---

Published as a conference paper at ICLR 2024
Task 1: Describe a viewer that would like the movie <embeddingg
Task Output
LLM-EMB
Task n-1: Write description of movie <embedding>
Task Output
Task n: Write positive review for the movie <embedding?
Task Output
Create semantic embedding of output ("Reencode" movie)
<embedding>
Movie Embedding Space
Figure 6: Semantic Consistency (SC). Task output is "re-encoded" back to semantic embedding space. A distance metric is then used capture the consistency to the original embedding vector that to generate the output.
They are interested movies that wall-made and have a strong story.
They appreciate movis that are vauully sopealing ginnn and have strong sense of style.
They enjoy movies that are thoughs-roicknn and make them think about the world a now way.
They are not internsted in movise that are viewent or gney.
They are not interested in movies that are too sentimantel chaiey.
Movie
User Profile for <embedding>
Movie Ranker for User Profile
Movie
LLM-EMB
Movie k
User Embedding Space
Figure 7: Behavioral Consistency (BC). Diagram depicts BC user profile task. Here, a generated user
profile is used as imput a Movie Ranker, which ranks movies based on user profile. The ranks movies are then compared to their groundtruth ranks in the Moviel.ns data, which are carried by the behavioral embedding that used generate the user profile.
B SEMANTIC AND BEHAVIORAL CONSISTENCY
Implementation Details of Semantic and Behavioral Consistency. While human raters can as- sess any semantic difference between the ground-truth text and the generated output, we measure the semantic consistency by computing cosine similarity between the semantic embedding of the ground-truth text and the semantic embedding of generated output. For movie tasks, straightforward as the input embedding to ELM is indeed the semantic embedding of the ground-truie text and we also employ the same DLM to reencode generated outputs by ELM as shown Figure For the user profile task, the input embedding is behavioral one though both the input embedding and the ground-truth user profile come from the same user. Figure 7 describes how we define behavioral consistency for users. The input behavioral embedding is predictive of movie rankings and want to see the movie ranker gives consistent rankings for randomly selected movies. We impute zero-star if selected movie has rating. The movie ranker ranks movies based on similarity the generated user profile in the semantic embedding space defined by some DLM.
more
Behavioral Consistency for Movies. Similar to behavioral consistency for users, as defined in See. 3.2 can define behavioral consistency for movies. Letting i/ denote of users, for any movie task with generated output om of movie m, we define behavioral consistency as
BC(0m, m; U) RankingScore( R.,. ( [|((0))))))))) Om)),
where p(om) are the ranks of the users in 24 for movie m, provided by User Ranker which uses on as its query, and Rm, = {r(u, m) u E U} are the MovieLens ground-trun truth ratings for users and movie m..
20

---

Published as a conference paper at ICLR 2024
agne
WIIN MORTTS PLOOT 1M
horkdihher and seraceee en tuzo reboit hoger fummed chamaacee
out aflvee
hwo
help Reper anditerto theteer set out to hin the nai
forton tieer numbe
antuaing trrper
Edda and Pogar evertuilly desover thst audqu Coen a the who hamned Reger for arter Dowrn also plann toel o reverng all the spots Tecatturr on aud trawway mhnmugh
defwating boone
the and Dete mamns o eveeoame hes prek sgaiest ceate and and Re become fiends.
On soalevo 16, how rear dryte ogrnn with the lowinn
The furny vestion coherent
nutiless wha bulp carey our tus riwes.
Strongh Disagree
Ore Epphwad' bendhmen detrpp sientis and crease deadl veapon. tygheaaro
5 Strorchhhhhoo Agree
deeray
team lhay The
feetoe draaae rustar a fox, and bear
The aroes are cofeet Sag and tis nohmman they save the city fhom sestruction
The full moneees such wha heowe ny eta faghadd bencnnse foo intrarpin th adtente
Figure 8: Evaluation Template for Running Human Rater Experiment for the "funnier" Task.
C RATER EVALUATION
We asked 100 human raters evaluate quality of movie tasks, terms of consistency with plot, language comprehensiveness, and task quality. For each task we randomly generated 40 model utterances with movies not used in training, save each utterance along with the contexts and questions on a Google form shown in Figure 8 and asked rater to evaluate. Usually start from ground-truth data of a movie and then model utterance generated for the task. For questions, we always ask how much do you agree with the following statement. Below list questions asked for each task.
1. long description
The second version plot is consistent with the first version.
The second version plot is coherent.
The second version plot is high quality like the first version.
2. summary
The movic summary is consistent with the movie plot.
The movie summary is coherent.
The movie summary is of high quality.
3. positive review
The movie review is consistent with the movie plot.
The movie review coherent.
The movie review positive.
4. negative review
The movie review consistent with the movie plot,
The movie review coherent.
The movic review is negative.
5. neutral review
The movie review consistent with the movie plot.
The movie review is coherent.
The movie review neutral.
6. five pos characteristics
21

---

Published as a conference paper at ICLR 2024
The characteristics listed are consistent with the movie plot.
The characteristies listed are coherent.
The characteristics listed are positive.
7. five neg characteristics
The characteristics listed are consistent with the movie plot.
The characteristies listed are coherent.
The characteristies listed are negative.
8. improve
The improved version plot consistent with the original plot.
The improved version is coherent.
Some viewers may find the improved version to be better than the original.
9. criticize
The criticism consistent with the movic's plot.
The criticism coherent.
The criticism is of good quality.
10. pitch
The pitch for the movie is consistent with the movie's plot.
The pitch for the movic is coherent.
The pitch convincing.
11. convincel
The paragraph convincing to watch the movie is consistent with the movic's plot.
The paragraph convineing to watch the movie is coherent.
The paragraph convincing to watch the movie is indeed convincing.
The paragraph convincing to watch the movie is comprehensive.
The paragraph convincing to watch the movie is short and to the point.
12. convince2
The paragraph convincing to watch the movie is consistent with the movie's plot.
The paragraph convincing to watch the movie is coherent.
The paragraph convincing to watch the movie is indeed convincing.
The paragraph convineing to watch the movie is comprehensive.
The paragraph convincing to watch the movie is short and to the point.
13. convince3
The paragraph convincing to watch the movie is consistent with the movic's plot.
The paragraph convincing to watch the movie is coherent.
The paragraph convineing to watch the movie is indeed convincing.
The paragraph convincing to watch the movie is comprehensive.
The paragraph convincing to watch the movie is short and to the point.
14. dissuadel
The paragraph dissuading to watch the movie is consistent with the movie's plot.
The paragraph dissuading to watch the movie is coherent.
The paragraph dissuading to watch the movie is indeed dissuading.
The paragraph dissuading to watch the movie is comprehensive.
The paragraph dissuading to watch the movie is short and to the point.
15. dissuade2
The paragraph dissuading to watch the movie is consistent with the movie's plot.
The paragraph dissuading to watch the movie is coherent.
The paragraph dissuading to watch the movie is indeed dissuading.
The paragraph dissuading watch the movie is comprehensive.
22

---

Published as a conference paper at ICLR 2024
The paragraph dissuading to watch the movie is short and to the point.
16. funnier
The funny version plot is consistent with the original plot.
The funny version is coherent.
The funny version is indeed funnier than the original.
17. sadder
The sad version plot is consistent with the original plot.
The sad version is coherent.
The sad version is indeed sadder than the original.
18. scarier
The scary version plot is consistent with the original plot.
The scary version is coherent.
The scary version is indeed scarier than the original.
19. movie to viewer
The viewer description is consistent with the movie's 's plot.
The viewer's description is coherent.
The viewer will like watch the movie based the information above.
20. interpolation
The combination of the movies is consistent with the movies' plots.
The combination of the movies is coherent.
The combination of the movies high quality, i.e., it combines the movies' plots in reasonable way.
21. similarities
The comparison is consistent with both movies' plots.
The comparison is coherent.
The comparison of the two movies compares important factors beteen the movies, both terms of similarity, as well as differences.
22. why like
The explanation is consistent with the movies' plots.
The explanation is coherent.
The explanation indeed explains well why the viewer would also like the other movie.
23. difr than
The explanation is consistent with the movies' plots.
The explanation is coherent.
The attributes that are listed are high quality (i.c., they are major and important differ- entiating attributes of the movies).
24. common with nn
The listed attributes are consistent with both movies' plots.
The listed attributes are coherent.
The attributes that are listed are high quality (i.c., they major and important common attributes between the movies).
D DISCUSSION ON TWO-STAGE TRAINING
In this section we describe our empirical investigation on two-stage training. Particularly, we found two-stage training procedure (as described in Sec. 2) to be essential for convergence.
To demonstrate this, we create a toy task decoding a vio-dimensional embedding to texts. In the training data replicate samples of mapping embedding (1.0, 0.0) to and samples of mapping
23

---

Published as a conference paper at ICLR 2024
Table 7: A short description each the movie tasks.
summary
One paragraph summarizing of movic plot.
positive review
positive review of the movie.
negative review neutral review five pos char. five neg char.
negative review the movie.
A neutral review of the movie.
Listing five positive characteristics of the movic. Listing five negative characteristics of the movie.
long description
long exhaustive description of the movie plot. plot funnier version the movie. A plot for sadder version of the movie.
funnier
sadder
scarier
A plot for scarier version of the movie.
improve
An improved version of the movie (as generated by LLM)
movie to viewer
Describing a viewer that would like to watch this movie, including characteristics.
pitch
A pitch for the movie. Criticizing the movic.
criticize
convineel
Convincing to watch the movic.
convinee2
Convincing in detail to watch the movic. Convincing briefly to watch Dissuading to watch the movie (version I prompt). Dissuading in detail to watch the movie.
convince3
movie.
dissuadel
dissuade2
similarities interpolation why like diff than nn common with nn
List three similarities between the movies.
Interpolate the plots of two movies.
Explain why someone would like nearest neighhor movie. Three major differences between two nearest neighbor movies. Three similarities between two nearest neighbor movies.
embedding [0.0, 1.0] to two. The test data has the same content. For this task, we did not use any textual prompt, but rather required the model to learn map the 2-dimensional embedding, to either the token corresponding to one, or that corresponding to two.
While this task is seemingly trivial we found that training ELM a pretrained model without two-stage training could not fit to the training data. Particularly, fine-tuning the full model for over 100k iterations always converged to outputting either one two. When two stage training was applied, i.e., first fitting EA and only then the rest of the model, we found the model to converge in less than 1000 iterations. This emphasizes the importance of two-stage training with pretrained LLMs. It further suggests that training adapter layers to pretrained LLMs requires stage of fitting to the semantic language space.
E DATA GENERATION PROMPTS AND SAMPLE TRAINING DATA
In the following, for each task we list the prompt used in training data generation and one sample output from the PaLM 2-L (Unicorn) language model (Google et al. 2023) with the prompt.
In training these 24 movie-focused tasks, the movie titles are replaced with movie embeddings in prompts (i.e., no information about the movie except its embedding was provided during training). For example, for task 1, the training input for the model was given by: "<movie embedding>." "". where here <movie embedding> was replaced by the semantic embedding and we do not have to prompt with "Do not use the movie's name in your response.". For tasks including two movies we did not enumerate over all pairs. Instead, for tasks 20-22 we used random movie comparisons, and in tasks 23-24 we used nearest neighbor comparisons. This choice allowed us to incorporate an inductive bias of the embedding space W manifold into the training procedure. In training the user profile task, those five positively rated and five negatively rated movies are replaced with user behavioral embeddings in prompts.
1. long description
Prompt: Write a long description of the plot of the movie The Derby Stallion (2005). Do not use the movie's name in your response.
Sample Outpur: The movie about a young man named Patrick McCardle who has always dreamed of owning and racing champion thoroughbred horse. He works hard up
24

---

Published as a conference paper at ICLR 2024
money to buy horse, and eventually he is able purchase a colt that he names Soaring Eatle.Patrick and Soaring Eagle train hard together, and they eventually qualify for the Kentucky Derby. The movie follows Patrick and Soaring Eagle as they compete against some of the best horses in the world in the most prestigious race in America. The movie is full of excitement, suspense, and drama, and it is great story about young man who achieves his dream.
2. summary
Prompt: Write a long summary of the movie The Shoe (1998). Do not use the movie's name in your answer.
Sample Output: The film is about a young man who is released from prison and tries to adjust to life on the outside. He gets a job at shoe store and starts to fall for one of his co-workers. However, he is still haunted by his past and struggles to find his place in the world. The film begins with the young man, who is unnamed, being released from prison. He is given a bus ticket and a small amount of money, and he is told to report to his parole officer within 24 hours. The young man is apprehensive about his release, but he is also excited to start a new life. The young man arrives in a small town and finds a room to rent. He then goes to the shoe store where he has been hired to work. The store is owned by kind and generous man who is willing to give the young man a second chance. The young man also meets his co-workers, who are all friendly and welcoming. The young man starts to adjust to his new life. He enjoys his job and he starts to make friends. He also starts to fall for one of his co-workers, a young woman named Sarah. However, the young man is still haunted by his past. He has nightmares about his time in prison, and he is afraid that he will never be able to fully adjust to life on the outside. One day, the young man is confronted by group of men from his past. They threaten him and tell him that he must leave town. The young man is scared, but he refuses to back down. He knows that he has to face his past in order to move with his life. The young man meets with his parole officer, who tells him that he is doing well. The parole officer also tells the young man that he has a visitor. The visitor is the young man's 's father, who has come to see him for the first time in many years. The young man is surprised and happy to see his father. The two of them talk and they start to rebuild their relationship. The young man is finally able put his past behind him. He is no longer afraid of his nightmares, and he is no longer afraid of the men who threatened him. He able to move on with his life and he is able find happiness.
3. positive review
Prompt: Write a long positive review for the movie Halloweentown High (2004). Do not use the movie's 'ss name in your answer.
Sample Output: The movie is about a young witch who goes to high school in the human world. The witch has to hide her powers from the other students, and she also has to deal with the prejudice of the humans against witches. The movie is well-made, and it has lot of heart. The characters are well-developed, and the story is engaging. The movie is great example of family-friende The movic is also a great way to teach children about tolerance and acceptance. Overall, the movie is great choice for family movie night.
Halloween movic. It is not too scary, but it is still entertaining.
4. negative review
Prompt: Write a long negative review for the movie Mayerling (1936). Do not use the movie's name in your answer.
Sample Output: This movie is a complete and utter mess. The plot is all over the place, the characters are unlikable and unrelatable, and the acting is just plain bad. The only saving grace is the cinematography, which is admittedly quite beautiful. But even that can't save this movie from being a total bore. The story is about a young prince who falls in love with a commoner. The prince's 's family disapproves of the match, and the two lovers are forced to part ways. The prince is then sent to military academy, where he meets another young woman. The two of them fall in love, and the prince eventually marries her. However, he is still in love with the commoner, and the two of them continue to see each other in secret. Eventually, the prince's wife finds out about the affair, and she confronts him. The prince is then forced to choose between his wife and his mistress. The plot is so convoluted and full of holes that it's hard to follow. The characters are all unlikable and unrelatable. The prince is a whiny brat who can't 't seem to make up his mind about what he wants. His wife is shrew who is constantly nagging him. And the mistress is manipulative schemer who
25

---

Published as a conference paper at ICLR 2024
is only interested in the prince's 's money. The acting is just plain bad. The prince is played by an actor who is clearly too old for the role. His wife is played by an actress who is so over-the-top that she's almost comical. And the mistress is played by an actress who is so wooden that she could be mistaken for a piece of furniture. The only saving grace is the cinematography, which is admittedly quite beautiful. The movie is set in some stunning locations, and the director does a good job of capturing the beauty of the scenery. However, even that can't save this movie from being a total bore. Overall, this movie is a complete and utter mess. It's poorly written, poorly acted, and poorly directed. The only thing it has going for it is the cinematography, which is admittedly quite beautiful. But even that can't t save this movie from being total bore.
5. neutral review
Prompt: Write a long neutral review for the movie Man Humt (1941). Do not use the movic's name in your answer.
Sample Output: This movie is a thriller about a man who is hired to assassinate a high- ranking Nazi official. The man is successful in his mission, but he is then captured by the Nazis and tortured. He eventually escapes and makes his way back to England, where he is reunited with his wife. The movie well-made and suspenseful. The acting is good, and the story is interesting. The movie does have some flaws, however. The pacing can be slow at times, and the ending is a bit too neat and tidy. Overall, this a good movie that is worth watching, It is a suspenseful thriller with good acting and an interesting story.
6. five pos characteristics
Prompt: List 5 characteristies of the movie The Very Thought of You (1944) that make great. Do not use the movie's 's name in your answer.
Sample Output: 1. The movie is a heartwarming story about two people who find love in the most unexpected of places. 2. The characters well-developed and relatable, and the acting is top-notch. 3. The cinematography is beautiful, and the soundtrack is perfectly suited to the film's 's tone. 4. The movie is well-paced and flows smoothly from scene to scene. 5. The overall message of the movie is one of hope and love, which is sure to leave viewers feeling uplifted.
7. five neg characteristics
Prompt: List 5 characteristics of the movie 3 Idiots (2009) that people hate about. Do not use the movic's 's name in your answer.
Sample Output: 1. The movie is too long and slow-paced. 2. The characters are not relatable and are often annoying. 3. The plot is predictable and full of clichés. 4. The humor is juvenile and often falls flat. 5. The ending is too neat and tidy, and does not feel earned.
8. improve
Prompt: Improve the movie Melbourne (2014). Describe how you improved the movie, and why it now better. Do not use the movie's 's name in your response.
Sample Output: The movie was about a group of friends who go on a road trip to Melbourne. The movie was good, but it could have been better. would have improved the movie by making the characters more relatable. The characters were all very different, but they didn't 't really have any chemistry. I would have made them more like real people, so that the audience could connect with them more. I would have also made the plot more interesting. The movie was pretty slow wow.pa and there wasn't 't a lot of action. I would have added some more suspense and intrigue, to keep the audience engaged. Overall, I think the movie was good, but it could have been better. With a few improvements, it could have been a great movie.
9. criticize
Prompt: Criticize movie There Was a Crooked Man... of the movie that people dislike and how it compares to similar movies. Do not use the movie's name in your answer.
(1970). Explain what characteristics
Sample Output: The movie is a Western comedy film that was released in 1970. It was directed by Joseph L. Mankiewicz and starred Kirk Douglas, Henry Fonda, and Hume Cronyn. The movie tells the story of a group of prisoners who are planning an escape from a prison in the American West. The movie people disliked the movie's 's slow pace and its lack of action. The movie was also compared unfavorably to other Western comedies, such Blazing Saddles and The Good, the Bad and
not well-received by critics. Many
26

---

Published as a conference paper at ICLR 2024
the Ugly. One of the main criticisms of the movie is that it is too slow. The movie spends lot of time on character development and dialogue, which can be boring for some viewers. There are also long periods of time where nothing much happens. This can make the movie feel like it is dragging on. Another criticism of the movie is that it lacks action. There are few scenes of violence, but they are brief and not very exciting. The movie is more focused on the characters and their relationships than on action. This can be disappointing for viewers who are expecting lot of action from a Western movie. The movie has also been compared unfavorably to other Western comedies, such as Blazing Saddles and The Good, the Bad and the Ugly. These movies are more action- pacioon-pakeddd and have more humor than the movie. They are also more popular with critics and audiences. Overall, the movie is not a bad movie, but it is not a great movie either. has some good qualities, such as strong acting and good character development. However, it is also slow and lacks action. If you are looking for a Western comedy with a lot of action and humor, then you might be disappointed with the movic.
10. pitch
Prompt: Assume the movie The Very Thought of You (1944) does not exist. Convince producer to create it. Explain what characteristics of the movie will make it great movie and how it compares to similar movies. Do not use the movie's 's name in your answer. Sample Output: have great idea for new movie. It's romantic comedy about a young woman who falls in love with a soldier during World War II. The woman a nurse, and she meets the soldier while he's recovering from an injury. They fall in love, but they' re separated by the war. The movie will follow their story as they try to find each other again. This movie will have everything that audiences love: romance, comedy, and drama. It will be great date movie, and it will also appeal to people who love historical films. The movie will be set in the 1940s, and it will feature all the fashion and music of the era. This movie will be different from other romantic comedies because it will be set during a time of war. The characters will have to deal with the challenges of the war, and they will have to make sacrifices for their love. The movie will be both funny and heartwarming, and it will leave audiences feeling hopeful. believe that this movie will be success. It has great story, and it will appeal to a wide range of audiences. hope that you will consider producing it.
11. convincel
Prompt: In detail, convince someone to watch the movie Death Promise (1977). Do not use the movie's 's name in your response. Sample Output: If you're looking for movie that will keep you the edge of your seat, look no further. This movie is full of action and suspense, and it's s sure to get your heart racing. The acting is top-notch, and the story is well-written. You'll be hooked from beginning to end. This movie is similar other action- action-packed thrillers, but it stands out for its unique plot and characters. The criminals in this movie are not your typical bad guys. They're re complex and well-developed, and you'l 'Il find yourself rooting for them at times. The woman they' re targeting is also a strong and interesting character. She's not just damsel in distress. She's smart and resourceful, and she's 's not afraid to fight back. If you're looking for a movie that will keep you entertained from beginning to end. I highly recommend this one. It's full of action, suspense, and drama. You won't t be disappointed.
12. convince2
Prompt: Convince someone to watch the movie Houston, We Have Problem! (2016). Do not use the movic's 's name in your response.
Sample Output: If you interested in a documentary about the Apollo 13 mission, this is a good movie to watch. is well-made and informative, and features interviews with the astronauts and the ground crew, The movie can be slow times, but it is overall good documentary.
13. convince3
Prompt: Briefly convince someone to watch the movie Karas: The Prophecy (2005). Do not use the movie's name in your response.
Sample Output: This movie is about war between two factions of supernatural race. The action scenes are very well done and exciting to watch. The animation is also very good. However, the story is a bit confusing and it can be hard to follow at times. Also, the characters are not very well developed. Overall, the movie is a good watch if you like action movies with lot of fighting.
27

---

Published as a conference paper at ICLR 2024
14. dissuadel
Prompt: In detail, dissuade someone from watching the movie Son of Sam (2008). Do not use the movie's name in your response. Sample Outpur: This movie is failure because it does not live up to the expectations of the audience. The plot is predictable, the ending is not satisfying, and the movie does not explore the psychology of the killer. Additionally, the movie is too long and can be slow at times. There are many other movies that are similar to this one that are much better. For example, the movie The Silence of the Lambsis much better thriller that explores the psychology of the killer. The movie Se7enis also a better thriller that has a more satisfying ending. Overall, this movie is not worth watching. There are many other movies that are similar to this one that are much better.
15. dissuade2
Prompt: Dissuade someone from watching the movie Elvis Madona (2010). Do not use the movie's name in your response.
Sample Output: would not recommend this movie to anyone. It is a poorly made film with a ridiculous plot. The acting is terrible and the dialogue is laughable. The movie is full of clichés and stereotypes. It is a waste of time and money.
16. funnier
Prompt: Write a funnier version of the movie The Mark of Cain (2000). Do not use the movie's name in your response.
Sample Output: Cain and Abel are two brothers who are constantly fighting. One day, they get into an argument over who is the better farmer. Cain gets so angry that he kills Abel. When God asks Cain where Abel is, Cain says, I don't know. Am I my brother's keeper?God is not amused. He punishes Cain by banishing him to the land of Nod, where he will be a fugitive and wanderer for the rest of his days. Cain also marked with a special mark, so that everyone will know that he is murderer. Cain wanders the land of Nod for many years, and he eventually finds a wife and has children. But he is never able to forget his crime, and he is always looking over his shoulder, expecting to be punished. One day, Cain is walking through the forest when he sees a group of men. He is afraid that they are going to kill him, so he runs away. But the men chase after him, and they catch him. The men take Cain to their leader, who is a powerful man named Lamech. Lamech is a descendant of Cain's brother, Abel. Lamech is also a murderer, and he is looking for revenge against Cain. Lamech tells Cain that he is going to kill him. But Cain begs for mercy, and he tells Lamech that he is already marked by God. Lamech is not impressed, and he is about to kill Cain when God intervenes. God tells Lamech that he cannot kill Cain, because Cain is already marked by God. Lamech is angry, but he obeys God. He lets Cain go, and Cain is able to live out the rest of his days in peace.
17. sadder
Prompt: Write a sadder and dramatic version of the movie Surviving the Game (1994). Do not use the movie's name in your response.
Sample Output: The year is 1994. The place is Seattle, Washington. The city the midst of severe economic recession, and many people are out of work. One of these people is Mason Storm, a homeless man who has been living on the streets for the past six months. One day, Mason is approached by a man named Thomas Burns, who offers him job. Burns says that he is looking for someone to help him with hunting trip, and he is willing to pay Mason $500 for his help. Mason is desperate for money, he agrees to go with Burns. The two men drive out to remote area of the woods, where they meet up with two other men, named Walter Cole and John Griffin. The four men then set out on their hunting trip. As the day goes on, it becomes clear that Burns and his friends are not interested in hunting animals. Instead, they are hunting Mason. They have been hired by a wealthy businessman to kill Mason, and they are determined to carry out their mission. Mason tries to escape, but he is no match for the four men. They chase him through the woods, and they eventually catch him. They beat him and torture him, and they leave him for dead. Mason survives, but he is badly injured. He makes his way back to the city, where he seeks help from a homeless shelter. The people at the shelter help Mason to get back on his feet, and they give him place to stay. Mason is grateful for the help that he has received, but he is determined to get revenge on the men who tried to kill him. He tracks them down, and he kills them one by one. After he has killed the men, Mason returns to the homeless shelter. He tells the people there what happened, and he thanks them for their help. He then leaves the shelter, and he
28

---

Published as a conference paper at ICLR 2024
never looks back. Mason Storm is a survivor. He has survived the streets, he has survived hunting trip, and he has survived revenge. He is a man who has been through a lot, but he has never given up. He is a symbol of hope for all those who are struggling to survive.
18. scarier
Prompt: Write a scarier version of the movie Space Jame (1996). Do not use the movie's name in your response.
Sample Outpu: The year is 1996. group of aliens have come to Earth to steal the talents of the world's best baskethall players. They have already captured Michael Jordan, Larry Bird, and Charles Barkley. The only ones who can stop them are a group of Looney Tunes characters led by Bugs Bunny, The aliens are a terrifying sight. They are tall and thin, with large heads and big eyes. They have sharp teeth and claws, and they move with a strange, jerky motion. They are also very strong and fast, and they are armed with laser guns. The Looney Tunes characters are no match for the aliens. They are quickly defeated and captured. Bugs Bunny is the only one who manages to escape. He knows that he must find a way to stop the aliens before they can steal the talents of all the world's best basketball players. Bugs Bunny travels the aliens' home planet, where he is reunited with Michael Jordan, Larry Bird, and Charles Barkley. The four of them team up to take on the aliens in basketball game. The game is close, but in the end, the Looney Tunes characters win. The aliens are defeated, and the world's best basketball players are safe. But the aliens are not finished yet. They vow to return to Earth and take revenge. The Looney Tunes characters know that they must be ready for them. They must train harder than ever before, and they must be prepared to fight for their freedom. The next time the aliens come to Earth, they will be met by a force to be reckoned with. The Looney Tunes characters will be ready for them, and they will not let them take over the world.
19. movie to viewer
Prompt: What type of viewers would enjoy the movie Justice League: Throne of Atlantis (2015)? Describe the viewer's characteristics. Do not use the movie's name in your response. Sample Output: The movie is a superhero movie. It is about the Justice League, a team of superheroes, who fight against the forces of Atlantis. The movie full of action and adventure. It is also full of special effects. The movie is best suited for viewers who enjoy superhero movies, action movies, and adventure movies. The movie is also best suited for viewers who enjoy special effects.
20. interpolation
Prompt: Write long description of the plot of a new movie that is the interpolation between Mr. Lucky (1943) and Gas, Food, Lodging (1992). Do not use the movies' names in your answer.
Sample Outpul: The year is 1962. The place is a small town in the American Southwest. The main character is young man named Lucky, who has just been released from prison after serving five-year sentence for crime he didn't I commit. Lucky is determined to clear his name and find the real culprit, but he soon finds that the town is full of secrets and that no one is willing to help him. Lucky's 's only ally is a young woman named Grace, who works at the local gas station. Grace is also an outsider, and she and Lucky quickly bond over their shared sense of isolation. Together, they begin to investigate the crime that sent Lucky to prison, and they soon uncover a dark conspiracy that involves the town's most powerful residents. As Lucky and Grace get closer to the truth, they find themselves in danger. The people who framed Lucky are determined to silence him, and they IIl stop at nothing to keep their secrets hidden. Lucky and Grace must all their resources to stay one step ahead of their pursuers, and they must find a way to bring the truth to light before it's too late. The movie is suspenseful thriller that will keep you on the edge of your seat. It's story about justice, revenge, and the power of love. It's a movie that will stay with you long after the credits have rolled.
21. similarities
Prompt: List three similarities and three differences between the movies The Power of Kangwon Province (Kangwon-do ui him) (1998) and Brick (2005). Do not use the movies' names in your answer. Refer to The Power of Kangwon Province (Kangwon-da ui him) (1998) the first movieänd to Brick (2005) as the second movie:
Sample Output: **imilarrtiie*** # * Both movies are about young people who are trying to find their place in the world.* * Both movies are set in small towns that somewhat isolated
29

---

Published as a conference paper at ICLR 2024
from the rest of the world.* * Both movies have strong sense of atmosphere, and the settings play a large role in the stories. 8 **Differences** * The first movie is a comedy, while the second movie is drama.* * The first movie is set in South Korea, while the second movie is set in the United States.* The first movie is about a group of friends who go on a road trip, while the second movie is about young who is investigating the death of his ex ex-girlfriend.
22. why like nn Prompt: I like the movie Colin Quinn: Long Story Short (2011). Explain to why would also like the movie Raging Phoenix (Deu suay doo) (2009). Do not use the movies' names in your answer.
Sample Output: The 2011 movie is documentary about a comedian who talks about his life and career. The 2009 movie an action movie about a woman who seeks revenge on the people who killed her family. Both movies are about people who are trying to overcome obstacles in their lives. The 2011 movie is funny and inspiring, while the 2009 movie exciting and suspenseful. Both movies are well-made and entertaining.
23. diff' than nn
Prompt: What are the three major attributes that are different between the movies Soldier in the Rain (1963) and Crimes Against Humanity (2014)? ? Do not use the movies' names in your answer. Refer to Soldier in the Rain (1963) as the first movieand to Crimes Against Humanity (2014) the second movie."
Sample Output: The first movie is a comedy, while the second movie a documentary. The first movie released in 1963, while the second movie released in 2014. The first movie is about a soldier who is trying to get back to his base, while the second movie is about the crimes against humanity that have been committed throughout history.
24. common with nn
Prompt: What are the three major attributes that are common between the movies Broken (2012) and Starred Up (2013)? Do not use the movies' names in your answer. Refer to Broken (2012) as the first movieand to Starred Up (2013) as the second movie: Sample Output: The first movie and the second movie are both British drama films. They were both released in the 2010s. They both have a rating of 7.3 on IMDb.
25. user profile
Prompt In ten bullet points, deseribe the attributes and characteristics of viewer who likes the movies: Catch Me If You Can (2002), Cellular (2004), National Treasure (2004), Die Hard (1990), and The Matrix (1999) but dislikes the movies: Half Past Dead (2002), Predator (1987), In the Valley of Elah (2007), The Legend of Zorro (2005), and Mortal Kombat: Annihilation (1997).
Sample Outpur: They enjoy movies that are fast-paced and action-packer They prefer movies with strong plot and well-developed characters. * They appreciate movies with sense of humor. They are not interested in movies that are too violent or gory. * They do not enjoy movies that are too slow-paced boring. They prefer movies that are visually appealing.* * They appreciate movies with good soundtrack. They are not interested in movies that are too predictable." * They enjoy movies that are thought-provoking and challenging.
F PROMPTS TO TEXT-ONLY LLM
In this section, present prompts used with the text-only LLMs on evaluating embedding in- terpolations (Figure 4). For movie interpolations, we used one-shot prompting in the following template:
The following shows an example of how to write a <task name> for a combination of the movies <movie name 1> and Forrest Gump (1994),
in some embedding space:
<ELM result of ground-truti interpolation between the above two movies>
In the same style, and in the same embedding space, write positive
30

---

Published as a conference paper at ICLR 2024
review for interpolation of the movies <movie name 2> and Porrest Gump (1994).. Do not use the movies' names in your zesponse.
where "<task names" be "summary", "positive review", "negative review", "neutral review", "five positive characteristies", "long description", and "pitch".
For the user profile task, used zero-shot prompting like the following:
In ten bullet points, generate an interpolation of the following two user profiles:
User profile 1:
<user profile 1>
User profile 2:
<user profile 2>
G
SAMPLE OUTPUTS FOR INTERPOLATIONS AND CAV EXTRAPOLATIONS
Below we present sample results of interpolating "Forest Gump" with sample movies:
Interpolation of "Inglorious Bastards (2009)" and "Forrest Gump (1994)" for the five pos characteristics task:
The movie is gripping, suspenseful thriller about man with low IQ who achieves great things.
The movie is powerful anti-war statement, showing the horrors of war and the toll it takes on those who fight in it.
The acting is superb, with all of the actors delivering powerful, memorable perfor- mances.
The cinematography is stunning, with the director creating some truly memorable visuals.
The story is well-written and engaging, with plenty of twists and turns to keep you guessing.
Interpolation "I, Robot (2004)" and "Forrest Gump (1994)" for the long description task: In 1983, mentally challenged man named Charlie Baileygates is living in small town in South Carolina. He kind and gentle soul, but he is also very naive. One day, Charlie meets a young woman named Myra who is new to town. Myra is kind to Charlie and they quickly become friends.
One day, Charlie walking through the woods when he comes across a crashed spaceship. Inside the spaceship, he finds robot named Harry. Harry is damaged, but Charlic is able to fix him. Charlie and Harry quickly become friends, and Harry teaches Charlie about the world outside of their small town.
One day, govermment agent named Maiden comes town looking for Harry. Maiden tells Charlie that Harry is dangerous weapon, and that he must be destroyed. Charlie refuses to believe Maiden, and and Harry run away.
Maiden and her team of agents pursue Charlie and Harry across the country. Charlie and Harry able to evade capture, but they forced to go into hiding. Charlie and Harry eventually find a safe place to live, and they live happily ever after.
The movie is heartwarming story about friendship and acceptance. It is a reminder that everyone is capable of love and friendship. no matter what their circumstances. The movie also reminder that the world is big and beautiful place, and that there is always something new to explore.
The movie is well-acted and dere-directed. The special effects are impressive, and the music is beautiful. The movie is classic that will be enjoyed by generations come.
31

---

Published as a conference paper at ICLR 2024
The movie is reminder that should never judge people by their appearance. Charlie may be mentally challenged, but he is also kind and gentle soul. He is capable love and friendship, and he deserves to be happy.
The movie is also reminder that we should never give up on our dreams. Charlie may be unlikely hero, but he is able to overcome the odds and achieve his dreams.
We also demonstrate qualitative results of CAV extrapolations using the user profile task. In the following we want move a user who doesn't really like funny movies to viewer likes funny movies in the behavioral embedding space by following the CAV direction. Then we decode the extrapolated user embedding to get a user profile:
They enjoy movies that are visually appealing and have strong
sense of style.
They appreciate movies that are well-acted and have strong performances.
They like movies that are thought provgkiiiooookin and have a strong message.
They enjoy movies that are suspenseful and have lot of action.
They like movies that are funny and have a lot of humor.
They dislike movies that are too slow-paced or boring.
They dislike movies that are too violent or gory.
They dislike movies that are too dark or depressing.
They dislike movies that are too predictable
formulaic.
The above user does like movies that are funny and have a lot of humor comparing with the groundtruth user profile which says "They are not a fan of light-hearted or comedic films.":
They are fan of foreign films.
They enjoy films that are dark and suspenseful.
They appreciate films with strong acting performances.
They are interested in films that explore social and political issues.
*
They are not a fan of light-hearted or comedic films.
They do not enjoy films that are visually stunning but lack substance.
They are not a fan of films that are set in the past.
They do not enjoy films that are about war or violence.
*
They are not a fan of films that are about crime or corruption."
H REINFORCEMENT LEARNING FROM AI FEEDBACK
Reinforcement learning from AI feedback (RLAIF) effective aligning LMs metrics that are labeled by off-the-shelf LMs in lieu of humans. Recent work like (Lee et al. 2023) has shown that utilizing a hybrid of human and preference models conjunction with the self-improving fine-tuning technique outperforms the traditional supervised fine-tuned basclines and offers additional benefits from standalone RL fine-tunng with human feedback (RLHF). Utilizing RLAIF paradigms, may fine-tune an ELM with reward so that would better align with consistency.
Contextual Markov Decision Processes (CoMDPs) The CoMDP is denoted by S,,, P.r, P.r,8ps Sp, N). in which the observable context space C contains both the user/item embedding vectors. The horizon N is the length of the generated texts. For any space S at the n-th turn represents the sequence of tokens generated thus far (00....,0n-13) 0 and the initial state so is the initial start-of-semtence token og. The action space A is the language token vocabulary, with action a E A representing any possible next token to be generated. The transition kernel models the next token distribution given the current sequence and contexts, which coincides with the LM policy, making the transition of our CoMDP known. Finally, the reward function measures the overall quality the generated texts. The goal to find policy " with maximum expected cumulative return, i.c., nT" E arg max, J₂ == N-1 ry Sp C, π]. Note that the size of the tokenized state and action spaces grow exponentially with the vocabulary size.
N, the state
32

---

Published as a conference paper at ICLR 2024
Specifically, with text response fon (on)n In and user-item embedding vectors (Wy, Wo), the consistency reward is defined as follows:
BC wa) (or SC(o; w.)) if on = [BOS];
r r(oni -0:- l::nn-l:wy-- wp w.)
otherwise.
The generation process of ELMs can be modeled using the following N-horizon CoMDP:
(We, Wu),
8n = 00:n-1; On Our
80
00;
P(sn+1 8k; On) 8(8n+1 (8n an));
{ ((()++1i)==((0)))) Way w.)
-i==-11 To(an (sn:c) = Po 00:n-1; Way Wa):
r(5n, dzi c)
otherwise
where dg denotes the Dirac distribution at z. Fine-tuning ELM is equivalent to maximizing the quality of the generated text given context, maxa IE (ws.)...) (o0.-- (ror))rrroooo [r(o; Wa, ww)]. The gradient of this objective function can be obtained as follows: ValBeu., (w.) Bes alwv,ww) [r;; W.. w,,,..).
E. (- ((sxie) En= Vo log (o( c)). This is equivalent applying the popular policy gradient algorithm REINFORCE to the aforementioned CoMDP for personalized text gen- eration. The gradient of the objective function is estimated using trajectories I[ (0(8))ann))) c) generated by the current policy, and then used to update the ELM policy in an online fashion.
Adding KL regularization: We add the KL between the fine-tuned and pre-trained mod- els as regularizer to the objective function. Leveraging the auto-regressive nature of ELMs one can compute the KL regularization over the entire sequence/trajectory (of tokens), ie., (Poo 1/,,, (Fre(PPre(Oo.... N
We, wa)). The resulting objective function is as follows:
W. w.) - B log max J(0)= == E ()) (oo(N ((oUNN-1: We
Po (o - 1 Wwy Ww.)
Per (oo:N -1 We, 1 W.))
It can be shown that this problem is equivalent to the KL-regularized objective in the CoMDP.
Denote by D replay buffer of trajectories {(w,y Way 00:N-1)3 generated by arbitrary ELMs For (Do:N-1)Wa, We, wa) and {(c, Sn, On sn+1) Y- D trajectory sampled from the offline data D. where (sn, an Sn+1) is tuple of state, action, and next state of the CoMDP, respectively. With this KI. regularization we can utilize the soft actor critic framework (Haarnoja et al. 2018) develop RL updates for the value function (V, ) N-1 state-action value function (Q (s, a; e) N- and II
ELM policy
N-1 To(8, ean:e) (initialized with II Ppre (8n (an; c)) that minimize losses:
N 2
LQ= lq=EErp ETAD
(V (8x+1ic) an; -00-
(2)
N (Qunn anic) a log
To (a ((in))nnc)) Ppre (anlanani
(snn
=V=E,W
(3)
n=0
N-1
To (anna(sni)) Ppre |snn c)
Lg EY
Σ
On ani a log
(4)
where the critic Qn and Vn take any token sequences at step n as input and predict the corresponding cumulative return; a > is the entropy temperature; (Var, Qua,,)) are the target value networks.
Besides iteratively updating the ELM policies and their critic functions, consider the closed-form optimal solution of the Bellman equation of this antropy-regularize RL problem:
(x; log Ba-ppe (- (s1s:0) (exp( (7,8, a; }], Vn,
(5)
~ Q 0x-1(s, r(s:)), (s, BarP(()a.a) (V (o':eo''))) Vn < 1
(6)
1 (a)s;c) Prre (a|s; c) exp( (%(s, (eumpee())))) (exp( (h(s,
)}, Vn.
where the dependepnn optimal policy (at time n), ie., μ is a softmax policy w.r.t. the optimal state-action values Qi over different actions sampled from the pre-trained ELM ppe- Therefore,
33

---

Published as a conference paper at ICLR 2024
value-based approach for RL-based ELM fine-tuning would be to first learn the optimal value
functions {Qr} } via the Bellman residual minimization applied to Eq. (5) and Eq. (6) and then solve
the following policy distillation problem: 0 arg ming E,-p Σ KL(TT (-1sm; (-
with respect the optimal value (Qr). Notice that this amounts updating the LM model 0 via the gradient update
N-1
To (a)35);)
2n (8, (0,5,a;))
[] (8)
0 - 0
BTWD
Σ
E Bawao (-130))
Vo log no (als;
Peee (s):c)
===
with learning rate > 0.
34


Reviews:
[{"review_id": "V7AuandHis", "reviewer": "Reviewer_BRjZ", "summary": "This paper presents a paradigm for demystifying the compressed embedding representations of deep models (such as MF trained on the MovieLens dataset in the paper) with Large Language Models (ELM). ELM first generates training data by prompting PaML 2-L with the movie title and task-specific information. The training procedure is composed of two stages: (1) training an adapter to project the domain embedding; and (2) full-training the full model including the language model and the adapter. Then the authors construct different tasks on the MovieLens dataset to test the performance of demystifying domain embeddings.", "questions": "1. Can ELM perform well with training data generated by other LLMs such as ChatGPT?\n2. Can the training data be real other than generated? Some datasets contain descriptions and revives (such as Amazon datasets). Why not use these kinds of datasets?\n3. Can ELM perform well with other models and other datasets.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. It is sound to demystify domain embedding with Large Language Models.\n2. The authors introduce their approach step-by-step in Sec 2 & 3, which is clear. \n3. The performance is evaluated by human raters.", "weaknesses": "1. The training data is generated by PaML 2-L, and ELM uses PaLM 2-XS to interpret the domain embeddings. Their similar architectures and training procedures may make the contribution limited.\n2. The authors argue that ELM is a general framework for different tasks, but the experiments only involve one model (MF) on one dataset (MovieLens). Therefore, the generalization of ELM to other models and datasets is not guaranteed.\n3. Some related works are overlooked. Existing works [1][2] have shown that tuning the adapter can let LLM demystify and reason over the embeddings of images. The authors should provide more explanations and experiments for their differences. \n\nreference:\n* [1] Zhu, Deyao, et al. \"Minigpt-4: Enhancing vision-language understanding with advanced large language models.\" arXiv:2304.10592 (2023).\n* [2] Koh, Jing Yu, et al. \"Generating images with multimodal language models.\" arXiv: 2305.17216.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "poYmJZxiqK", "reviewer": "Reviewer_pa1w", "summary": "This paper presents ELM, a framework that uses LLM to interpret embeddings. By training an adapter, ELM maps the domain embedding space to the LLM token embedding space, allowing users to use natural language to query and understand the domain embedding space. An evaluation of 24 original tasks with a movie rating dataset shows that ELM provides high semantic and behavioral consistency. Finally, the paper shows promising results of using ELM to query hypothetical embeddings and generalize concept activation vectors.", "questions": "- [Q1] Have you tried ELM on other datasets and tasks other than movie reviews?\n- [Q2] Figure 1 is a bit hard to understand. For example, how do people generate the embedding for the animated version of Forrest Gump? Interpolating the embedding of Forrest Gump with an animated movie? Is the black text output from an LLM? Who writes the blue text (e.g., users, researchers)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- [S1] The proposed method ELM is novel and intuitive. The two-stage training (adapter first, full model next) is interesting and makes sense.\n- [S2] I appreciate including human evaluation in section 3. The analysis and discussion on hypothetical embedding vectors and concept activation vectors (section 4) are very interesting.\n- [S3] The paper overall is well-written and easy to follow.", "weaknesses": "## Major weaknesses\n\n- [W1] The evaluation can be improved. (1) The training data are all synthesized from an LLM. (2) The 24 tasks are all original. (3) ELM is only evaluated on the MovieLens. (4) There is no baseline method. It is unclear how well ELM will perform on real data and tasks compared to other methods.\n\n## Minor weaknesses\n\n- [M1] It would be great to explain $E_A$ in Figure 2 (the definition is on page 4).\n- [M2] The 24 tasks are essential to understand this paper. I recommend adding a table to provide a high-level description of these 24 tasks in the main paper.\n- [M3] The ELM method is quite intuitive, but the writing in section 2 is overly-complex. It would be helpful to have a small glossary to explain all notations.\n- [M4] It is hard to make sense of Table 1. Are these numbers good?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AIuOZzYByI", "reviewer": "Reviewer_bEK6", "summary": "This paper presents the Embedding Language Model (ELM), a novel language model framework for interpreting domain-embedding spaces. By inserting trainable adapters into existing large language models (LLMs), ELM can accept domain embedding vectors as parts of its textual input sequence to allow the interpretation of continuous domain embeddings using natural language. Abundant experiments demonstrate ELM's proficiency in understanding, navigating, and manipulating complex embedding representations.", "questions": "- What is the detailed form of the loss function $\\mathcal{L}\\_{\\theta}$ used in the experiment? Did the authors use reinforcement learning from AI feedback to optimize $\\mathcal{M}_{ELM}$?\n- Since the output of instances in the training data is generated by LLMs, can ELM reasonably interpret embeddings in specialized domains outside the scope of LLMs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interpreting abstract embeddings into human-understandable natural language descriptions is intuitively appealing.\n- The proposed approach is simple and effective by reasonably leveraging the power of large language models (LLMs).\n- The authors have comprehensively assessed the quality of ELM's outputs using a variety of evaluation techniques, including qualitative human evaluations and specific consistency metrics.", "weaknesses": "Honestly, one of my greatest concerns is the practicality of the proposed framework, given that training an ELM requires manually constructing a batch of tasks, which need to be diverse enough to extract rich semantic information from the target domain $\\mathcal{W}$ to support the interpretation of the embeddings. Admittedly, the experimental part of the paper validates ELM's proficiency in interpreting two forms of embeddings on a movie-related dataset. However, I am not sure if ELM performs equally well in other more specialized domains, considering that the original training corpus of LLMs is likely to contain rich semantic information relevant to movies. In addition, constructing diverse task prompts may be tedious for a realistic user, so are the prompts presented in Appendix D basically applicable to other domains? After all, the most straightforward need for a user is to know what an embedding represents in general.\n\n- Given the prevalence of adapter tuning [1], there is nothing new to me in the method.\n\n1. Parameter-efficient transfer learning for NLP, Neil Houlsby et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M5WE4JgmGW", "reviewer": "Reviewer_viWv", "summary": "The authors propose a framework (ELM) for interpreting the vectors in domain-embedding spaces using text-only LLMs. The main idea is to project vector representations and textual tokens to a conjoint vector space. This is achieved by training adapter layers that serve as an interface to the language model. This adapter layers, together with the pre-trained model, create a new, extended model that maps both tokens and embeddings (from a arbitrary latent metric space) into a common space. The authors put this framework to the test by attempting to interpret two different semantic embeddings that they generate from the MovieLens dataset: One that models the users by their ratings of movies and one  that models the movies themselves based on their descriptions. The input data for the ELM framework then consists, for example, of text sequences in which the title of a movie has been replaced by the correspondingly generated semantic embedding. In order to verify the results of the framework, the authors rely on the expertise of test persons on the one  hand, and on the other hand, they also design two metrics to test for consistency.", "questions": "•\tHas the framework been tested on other embeddings that may contain less semantic information than the domain embeddings used here (similar to the example in Appendix C)?\n•\tHow do the authors assess the potential usefulness of their approach for the interpretation of other dense vector representations, for example graph or image embeddings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the strengths of the article lie in the idea of extending a model to accept sequential tokens and embeddings as input data. I agree with the authors that this approach has the potential to provide a tool to study corresponding vector spaces spanned by semantic representations. In addition, this offers interesting possibilities for applications such like recommendation systems. Based on the elaborate and convincing experiments, the authors can very well prove the merits of their approach. Another positive feature is the detailed appendix.", "weaknesses": "Unfortunately, in my opinion, the actual core of the article is somewhat overshadowed by the detailed experiments. I would also have liked to see a figure (in addition to Figure 2) that shows the structure of the framework at a glance. In my opinion, the surface planes shown in the 3D graphs of Figure 1 do not contribute much to the clarity either, since they only exemplify a distance of points in the vector space. The authors take effords to show the benefits of their framework with two examples (movie ratings - behavioral embeddings and movie descriptions - semantic embeddings), but perhaps one of the vector spaces studied could have come from a completely different domain, perhaps derived from a different dataset. But this probably falls into future work. On the whole, the paper is very well structured and easy to read, but the section on related work seems a bit out of place before the section conclusions. Perhaps it could have been placed further forward in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)