Vector-ICL: In-context Learning with Continuous Vector Representations
Authors: Yufan Zhuang
Chandan Singh
Liyuan Liu
Jingbo Shang
Jianfeng Gao
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
in-context learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We discover that large language models can effectively process and in-context learn from continuous representations from various domains, often outperforming regular ICL and domain-specific models across diverse tasks and modalities.

Abstract: Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.

Published as a conference paper at ICLR 2025
VECTOR-ICL: IN-CONTEXT LEARNING WITH CON- TINUOUS VECTOR REPRESENTATIONS
Yufan Zhuang',
Chandan Singh", Liyuan Jingbo Shang', and Jianfeng Gao²
UC San Diego
Microsoff Research
ABSTRACT
Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be ex- tended to continuous vectors from diverse domains, obtained from black-box pre- trained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICI. In particular, we find that pretraining projectors with general language modeling objectives enables Vector- ICI., while task-specific finetuning further enhances performance. In our experi- ments across various tasks and modalities, including text reconstruction, numer- ical function regression, text classification, summarization, molecule captioning, time series classification, graph classification, and fMRI decoding, Vector-ICL of- ten surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.
INTRODUCTION
In-context learning (ICL.) has emerged as powerful paradigm in large language models (LLMs), allowing generalization from limited examples within a given context (Brown et al. 2020 OpenAl, 2023). By providing demonstrations in the context during inference, ICL allows models to adapt to new tasks and formats without the need for retraining. However, since LLMs are trained on discrete natural language tokens, ICL is generally learned and used through natural language, limiting its applicability to non-textual data.
We explore whether LLMs perform ICL directly continuous vectors, capability that could dramatically expand their applicability. Many data modalities, such as sensor readings, financial time series, scientific measurements, lack natural text representation. Moreover, data, information like numbers might be better represented via continuous vectors than tokens.
for text
In our study, observe that LLMs can indeed understand and process continuous context via em- bedding projection. This technique, which we term Vector-ICI, acts as bridge between continuous data and the LLM's embedding space. Simple linear projections are often sufficient, though for cross-modal tasks-such as those involving non-textual data like time-series graphs, non-linear transformations may be required. We demonstrate that training the embedding projector using straightforward next-token prediction objective enables Vector-ICI, effectively teaching the LLM to "read" continuous vectors. Moreover, fine-tuning the projector on downstream tasks further en- hances the effectiveness of continuous context, outperforming few-shot ICL and domain-specific models or tuning.
Our investigation begins with the task of text reconstruction, where we assess whether LLMs can recover information encoded in text embedding. This serves as a proof-of-concepl for Vector-ICL, showing that LLMs can indeed extract meaningful information from projected continuous vectors. We then turn to the more complex challenge of arithmetics. Although state-of-the-arr LLMs can solve Olympiad mathematical problems (Trinh et al. 2024 OpenAl 2023), they struggle with pre-
Code available httppe///.hhhh.ccc//aaaaaaaaaass

---

Published as a conference paper at ICLR 2025
(u) Regular few-shot in-context learning
Text
AA
Input: [Text Snippet 1]. The input's sentiment is positive.
Imput:
[Text Snippet 2]. The input's sentiment is negative.
Numbers
Input: [Text Snippet 3]. The input's sentiment is
Regular ICL:
(b) Vector in-context learning
Text
AA
Input Data
Encoder
fenc (zz) E Ro Projector
Numbers
(Tous use crampie here)
Brain fMRI
M
Time Series
1 sentiment is positive. D2's sentiment is negative. g sentiment is
Graphs
Vector-ICL
Figure 1: Comparing regular in-context learning to vector in-context learning. (a) In regular ICL, textual demonstrations are given as context during LLM inference. space is extended across multiple modalities. The input data is first encoded as embeddings, then transformed into continuous vectors which represent as box tokens (L) via embedding projection. During inference, provide box tokens in prompts demonstrations for ICL. We consider box tokens representing text, numerical data, brain fMRI, time series, and graphs in this study.
(b) In Vector-ICL, the input
cise number processing due the limitations inherent in their tokenization schemes. Our results demonstrate that Vector-ICL offers more effective approach for function approximation, particu- larly for large numbers that span multiple tokens, potentially opening avenues for enhancing LLMs' numerical reasoning capabilities.
Finally, we extend our analysis to broad range of modalities and tasks, including text classification, summarization, molecule captioning, brain IMRI reconstruction and classification, time-series clas- sification, and graph classification. Across these diverse domains, LLMs exhibit competitive and often superior performance when employing Vector-ICL, revealing previously untapped capabili- ties of these models. This work highlights the potential of continuous representations in enhancing LLMs' in-context learning capacities, pushing the boundaries of what these models can achieve beyond leken-based paradigms.
RELATED WORK
Empirical results of in-context learning ICL has empirically shown strong performance di- verse natural natural-language processing tasks with very few demonstrations (Brown al. 2020; OpenAl 2023). In modern LLMs with long context windows, ICL has even shown performance improve- ments the number of demonstrations grows hundreds
thousands, sometimes outper- forming finetuning (Agarwal et al. 2024 Li et al.. 2023a Bertsch et al. 2024). Empirically, differ- factors play key roles in ICL. In smaller LLMs, ground-truth demonstrations are required for in-context learning, while other factors such as the label space, input text distribution, and overall sequence format play an important role (Min al.) (022b). Moreover, these LLMs can some- times achieve strong performance even when demonstrations are intentionally irrelevant or even pathologically misleading (Wehson & Pavlick 2022). Flan-T5 (Chung et al., 2024) revealed that instruction tuning improves few-shot learning by helping LLMS better utilize in-context examples an encoder-decoder architecture.No LLMs with respect to model size, and show that larger language models perform in-context learning differently in the presence of flipped or semantically unrelated labels. Orthogonally. different works find ways to improve ICL, e.g. by including explanations (Lampinen et al. 2022). or chaining ICL
recently, Wei et al. (2023) characterize these behaviors of
calls (Morris et al.) 2023). ICL has shown some success in multimodal models (Wu et al. 2024
Jiang al. 2024) when applied to tabular data (Zhao al. 2024).
2

---

Published as a conference paper at ICLR 2025
(a) Pretraining Projectors - Enables Vector-ICI.
(b) Finetuning Projectors - Improves Vector-ICL.o on Tasks
Input Data
Encoder
Projector -
ni RRR
Input Data
Encoder
Projector
n R
Knowlolge limited
To live or live
Test Supper
Test Seippe
Imagination encircles world
is the question
*semimmnt
positive
5 sentiment
negstive
Decoder: Large Language Model
Decoder: Large Langunge Model
Imaggination encircles the world
That is the question
positive
negative
Figure 2: Pretraining and finetuning the projectors. Vector-ICL requires updating the parameters of a lightweight projector while keeping the encoder and decoder parameters fixed. The encoder first compresses the input into single token embeddings, and then the projector will project it to the aligned representation space for LLMs' later use. (a) Pretraining the projector on a general language modeling corpus (or modality-to- to-text dataset) enables Vector-ICI. (b) Task-specific fine-tuning makes Vector-ICL. outperform few-shot ICL on natural language tasks, as well as with domain- specific models on non-language tasks.
Understanding ICL
Many works have investigated ICL and found that it is able to learn linear
models (Akyürek et al. more general algorithms (Li et al.] 2023b). Some works have explicitly connected ICI. in specific
2022 Zhang et al. 2023), discrete functions (Bhattamishra et al. 2023) and
settings implementing optimization steps analogous to gradient descent (Mahankali et al. 2023
Von Oswald et al.) 2023; Ahn et al. 2024) and higher-order optimization methods (Dai et al. 2023
Fu al., 2023] Giannou et al. 2024| Zhang et al. (023). A complementary direction aims to establish statistical complexity and generalization bounds of in in-context learning in transformers (Bai 2023b] Wies et al., 2024, Wu al., 2023). Finally, one recent work suggests that ICL may arise from parallel structures in pretraining data (Chen et al. 2024).
al. 2024 Li al.,
Learning learn in-context In contrast to the emergent ICL. capabilities of I.IMs, existing works have also studied how explicitly improve ICL. Min et al. (2022a) propose MetalCL, meta- training framework for finetuning pretrained LLMs to perform in-context learning on large and diverse collection of tasks. In the tabular domain, TNP (Nguyen & Grover 2022 and PFNs (Müller jet al. 2021) train transformer models to perform in-context prediction for a family of functions, which allows in-context generalization unseen functions after training. [Zhao et al. (2023) also propose meta-learning transformers to in-context learn group preferences, serving as an in-context learned reward model that adapts to diverse group preferences.
METHOD: VECTOR CONTEXT VIA EMBEDDING PROJECTION
3.1 EMBEDDING PROJECTION
Vector-ICI. requires transforming inputs into vector contexts through an embedding projection. Given a dataset data into an abstract representation (alternatively, the raw data may already be a continuous vector). The encoded embeddings, (x),, are then projected into box tokens, denoted as C,- Throughout the paper, we will use the terms "box tokens" and "projected embeddings" interchangeably. For decoding, we use language model fim that generates outputs based on the provided prompts.
{x:}= we assume the existence of an encoder fenc. that transforms the
We impose no constraints on the form of the input data X'; it can come from any modality. The only requirement is that the encoder Sene maps each data point æ into a vector space, defined as:
fenc
Rede Vr E X
(1)
The LLM typically processes discrete tokens [tok,, tokg.... tok,). }. then maps them to text embed- ding space femb1, emby.... emby), Vi, emb, Rde,, Since operate mostly in embedding space, we omit the tokenization step for simplicity and directly refer to text inputs as their embedding representations.
3

---

Published as a conference paper at ICLR 2025
Vector context can understood by LLMs, and can better enable
Projectad Enboldiings
Roceninctios
Vector Textual
Regrasion
Selative
-00-
-40-
30-
fene(x)
AA
100
+ fenc(a) P
Number
Text Chasiiicativn
Test Summanisation
Malecule Captioning
80-
20
60-
Kougel
LS-
40-
(x)
fenc
AA
fenc
AA
20
10
20
Number Stots
Number of Shoos
Number of Shots
Time Classilicazion
Graph Chasification
Bman (MR) Decoding
Brain Classification
12.5
Accuncc
7.5-
51
20-
(Jene(z)) (x))
MLP Seme(>))
0
fane(s) -P
Fencl (2)
P
3 10 12 14
10 Number Stote
20
25
50
Number of Shots
Number of Shots
Number Shats
Vector-ICL on modalities when regular ICL is not applicable
Fincianol Projection
Prirctiin
Few-sbee
Fow-ebet ICE (Choo-Maaal)
Figure 3: Main results: LLMs can perform Vector-ICL (f better). We show that training the embedding projector with a simple next-token prediction objective enables Vector-ICI. Even with only unsupervised pretraining. Vector-ICL matches or outperforms traditional few-shot ICL on out of 6 tasks where direct comparison is possible. Fine-tuning the projector on downstream tasks further enhances the use of continuous context, consistently surpassing both few-shot ICL and spe- cialized task-tuned baselines (soft-promp) for text, tuned encoders for non-text).TT The study begins with text reconstruction to assess LLMs' ability to interpret box token embeddings, followed by function regression to evaluate reasoning capabilities. We then demonstrate Vector-ICL's effective- ness and applicability across various downstream tasks, including text classification, summarization, time-series classification, graph classification, and brain fMRI decoding & classification. Results in each panel are averaged over different encoders and LLMs for the diverse tasks we study; error bars show 95% confidence intervals.
The process of embedding projection is then carried out as follows. For linear projection, we con- struet projection matrix P da X and make the following transformations obtain pro- jected embedding D2 given input
D : : Senc (xn P
cases where more expressive power needed, we utilize two-layer multi-layer perceptron (MLP) to perform the projection:
Dg := MLP((fec(z))
(3)
The MLP follows the architecture the MLP block in Llama (Touvron et al. 2023), with an addi- tional input projection layer map the input dimension Rd output dimension Rd
4

---

Published as a conference paper at ICLR 2025
3.2 PROJECTED EMBEDDINGS AS CONTEXT
The projected embeddings are then utilized as context in Vector-ICL, functioning as the equivalent of the original input data. For example, NLP tasks, the original text snippets will first be encoded as embeddings fene (2), then projected to become g. inserted into the prompt like the following:
D 's sentiment is ... / Dg summarization is.
Using them as the context in ICL is then natural:
D 's sentiment is happy. D2's sentiment is sad.
D 's sentiment is
where 1 and 2 are in-context examples and n is the input.
3.3 TRAINING THE EMBEDDING PROJECTORS
The projectors need to be trained to achieve effective projections. We discovered that pretraining these projectors with language modeling objectives enables the capabiiiils with vector context, and finetuning them on task datasets further improves ICL performance.
The pretraining process is depicted in Fig. a For each text snippet, cut it into two pieces with the cutting point randomly sampled from the end of sentences. The first half is encoded and projected while the second is kept intact. The rest is the same with any pretraining process. the language model generated the next token distribution at each input position, except for the ones preceding the projected embeddings, and a cross-entropy loss is imposed on top of this. With the encoder and L.M frozen, the gradient backpropagates to the projector, updating its parameters.
For non-text data modalities, pretraining can be more flexible. We define this pretraining as involv- ing general, con-task-specife objectives, such as reconstructing number from its embeddings (e.g., Op is 32768), performing basic algebra (e.g., Dn D, 16384), or predicting the next token from brain fMRI embeddings.
The finetuning process shown Fig. (b). utilizes additional structured prompts and trains with task-specific datasets. Similarly, the imput is first mapped into the embedding space and projected into tokens. They are then inserted into structured prompts, while the projector trained with conditional generation loss given those prompts.
EXPERIMENTAL SETUP
Table D gives an overview of our experimental setup, including specifics for the task, datasets, en- coders, LLMs, and task-specific prompts we use. Across different tasks, we project to four open- weights LLMs. We now provide details for individual tasks.
Baselines We evaluate Vector-ICL in two distinct settings: with and without task-specific tuning. For textual tasks, we compare against few-shot ICL and soft prompt tuning (Li & Liang 2021 We choose soft prompt tuning as our primary baseline because represents a similarly lightweight adaptation approach- both methods introduce small number of trainable parameters while keeping the base L.M frozen. Like our projector, soft prompts modify how the L.M processes inputs without changing its internal weights. This makes a fair comparison point for assessing whether Vector-ICL's benefits come from the continuous vector representations themselves rather than just the additional training. For non-textual tasks, where soft prompts cannot be directly applied, compare against cross-modal few-shot ICL and tuned encoders. In the non-textual domain, ICL inputs are represented either numeric sequences (for time-series and brain fMRI data) textual descriptions (for graph edge lists and node features).
Text Pretraining To pretrain our text projectors, we leverage the WikiText-103 (Merity et al. 2016) dataset, consisting of over 100 million tokens from verified high-quality Wikipedia articles. This smaller language modeling corpus is chosen for its suitability to the lightweight nature of projectors. The pretraining process is illustrated in Fig. 2(a), where text snippets divided at random sentence-end points. The first half is embedded and projected, while a next-token generation loss is applied to the second half.
5

---

Published as a conference paper at ICLR 2025
Table I: Overview of the tasks, datasets, encoders, large language models, and prompts used in the experiments. Each task utilizes encoders and LLMs to perform functions across multiple modalities. The table highlights the diversity of models and configurations applied each task.
Task
Dataset
Encoder
LLM
Prompt
Quera (Takkr PST Tiedomann
2021
Translate text brackets: (O)
Text Reconstriction
N. S. B
Translation: [Text]
x Las y
Function Regression
10 Digits Regression
Digit Embeedding
function(2,, 3/ equals to (digits): [Solution]
IMDB Maass et al.
2011
Tombirie
Pang & Lee 2005
Text Classification
SST2 (Sncher et
20
M. S. 8
Iss sentiment is: [Tabel]]
Emotion
2018
Financial Parasebank
Malo 2014)
Summarization
XSum (Naravan et
XL.Sum Hassa ot
2021
2018
B.
summarization [Summary]
Molecule Captioning
Language
Edwands et al. 2024
N
1u
(D)s molecull caption is: [Caption]
Brain (MRI
LeBel et all 80MR
PCA of fMRI
[Question] Input: D Response: [Answer]
Tang
Time seties Classification
Tunu/, Funus
Chronos base
(O)'s class (positive, oegative) is: [Label]
Dau et 2019
Ansari et
2024
Graph Classification
Graphormer
(O)'s IJ's class (positive, negative) is: [Label]
Hu cl
2020
Ying ct al.
2021
Encoders: NY-Embed (Lee et 2024
Evidia/NV-Bniin WWV-mbed-wl), $ SHR (Meng el 2024 Saiesforces /SP RennR--ddiind----eeeee dd..n-2-nnngn
El Stella (Zhang 2024
dunzhang/
aa.on.l. S8.v5), G GTR i5 (Ni 202] sentence-tran formora cammmerr////ttteeeeeeeeee (irrr---bbs)))
LLMs: io Talma- 3. L-BB (Duhey ct 2024
amaa//.mmmmmmm 1-88- Trat ruet). @ Mistral-7B (Tiang et 12023]
mistralad a i/Maatral 770 Instruct
v0.3), Qwen2-7B (Yang al. 2024 Qwen./Qeer.2 7B InsLiuct), Y--15-9B
Young
CE aL. 2024
01 01-ai/Xi-1.F 1-1-x-1-15----- 5-5B---ttt
Text Reconstruction We investigate LLMs' ability to decode original text from projected em-
beddings using two datasets: Parallel Sentence Talks (Tiedemann) 2012) and Quora (Thakur al,, 2021). These datasets consist of concise text pieces that convey clear meaning. Projectors are trained on the training sets of both datasets, and performance evaluated using the BLEU score (Papineni et al.
2002 Post 2018).
Arithmetic and Function Regression For the arithmetic tasks, we generated synthetic datasets containing 10-digit numbers, which are particularly challenging for LLMs as they require splitting the numbers into multiple text tokens. These numbers are represented using a concatenated one-hot encoding per digit. For instance, a 10-digit number is represented as 10 x 10 matrix, flattened into a 100-dimensional vector. The pretraining phase includes two key tasks: number reconstruc- tion, where the model is tasked with recovering the original number from its embedding, and basic arithmetic, where the model performs algebraic addition operations on the projected embeddings.
To evaluate the models' arithmetic reasoning abilities, employ non-linear function regression task, where the function is defined as fz, y) = Vz.j The model is provided with inputs x and y, and it must predict the integer part of the function output. Performance is measured using the mean relative error, calculated as the L difference between the predicted and true values, normalized by the ground truth. This task allows us to assess the models' ability perform more complex numerical reasoning beyond simple arithmetic operations.
Text Classification We assess whether Vector-ICL can be applied effectively text classification. Both binary and multi-class classification datasets are used, and the results are compared across few-shot ICL and soft prompt tuning. The classification performance is measured by accuracy.
Text Summarization Following the classification tasks, we explore Vector-ICL's capability in summarizing text based on the projected embeddings. The datasets, encoders, LLMs, and prompt templates can be found in Table Performance is evaluated using RougeL LLin 2004
Molecule Captioning We also extend approach to the unconventional task of molecule cap- tioning, using molecule sequence-caption pairs from the Language + Molecules-24 (LPM24) (Ed-) wards et al. 2024) dataset. A sample molecule-caption pair looks like the following:

---

Published as a conference paper at ICLR 2025
Table Comparison of Vector-ICL, few-shot ICL, and soft prompt tuning across various sentiment analysis and summarization datasets. Details can be found in Appendix A.3
Method
Sentiment Analysis
Summarization
Rotten Tomatoes
Financial Phrasebank
SST2
IMDB
Emotion
XSum
XLSuum
V-ICL (pretrained)
80.60
78.90
95.04
41.20
60.72
15.25
15.89
Few-shot ICL
87.31
91.74
93.50
55.20
71.78
19.53
19.41
Soft Prompt
93.24
96.21
95.26
74.15
78.22
12.84
12.70
V-ICL (finetuned)
88.80
98.16
97.28
85.20
81.68
20.08
20.49
Molecule:
Caption: The molecule is a pain treatment that impacts inflammatory disease treatment.
This task explores whether LLMs extract useful information from projected embeddings of out- of-distribution chemical sequences, with performance evaluated via BLEU score.
2022 and Tang et al. 2023] which consists of fMRI responses for 3 human subjects as they listen to 20+ hours of narrative stories from podcasts. We preprocessed the data following Benara al. 2024 by con- verting the fMRI responses into a 200-dimensional output using principal components analysis and assigning classification labels to 10-grams of the story text at 2-second intervals using an L.LM.
Brain IMRI Decoding and Classification We analyze data from LeBel
We use the same pretraining methodology as text to pretrain on the brain fMRI data (projecting on 20% of time points and imposing next-token generation loss on the remaining 80%). We evaluate the LLM's capability to decode projected brain IMRI by giving them randomly sampled context from the train set, that could come from different human subjects or from a different story, and ask them to decode segments from the test set.
In addition to text reconstruction, we decode the binary labels from the fMRI responses, corre- sponding questions about the underlying text, e.g. "Does the sentence contain a proper noun?" The decoding random baseline is constructed by giving the LLM the randomly sampled, shuffled text from the training set, and generating text according to it. We measure the performance using the RougeL score between the generated text and the ground truth text. The classification random baseline is 50% accuracy,
have balanced the dataset.
Time-series We take the output of the last time step from Chronos-base (Ansari et al. 2024) as the time-series' representation. We use the base encoder with trained classification head as the baseline and we measure the prediction performance with accuracy.
Graphs We use Graphormer (Ying et al. 2021] as the encoder model, specifically the one that was pretrained quantum chemistry graph datasets (Hu et al., 2021). Since the down-stream, ogbg- molhiy (Hu et al. 2020). is a molecule property prediction dataset, and with strong class imbalance (3-4% positive classes), we finetune the encoder on the training set provide meaningful baselines and embeddings. We take the output prior to the classification layer of the Graphormer as the graph embedding. Weighted sampling adopted in the finetuning of both the baseline Graphormer and the embedding projector to yield meaningful predictions. We use the finetuned Graphormer as our baseline and the Fl score as the performance metric.
Projector Configurations Both linear and non-linear projectors are utilized, as shown in Fig, with input and output dimensions matching the of 500 steps is used during finetuning, projectors converge quickly due their small parameter sizes. Details of the hyperparameters used in training are provided in Table
pairs. Early stopping with patience
5 RESULTS: UNLOCKING VERSATILE APPLICATIONS ACROSS MODALITIES
Fig. 3 presents our main results, where each subplot corresponds to one of the nine tasks. We begin by exploring text reconstruction, see whether I.IMs can comprehend the information encoded

---

Published as a conference paper at ICLR 2025
within the box tokens. Next, we investigate the tasks of function regression to evaluate whether LLMs can leverage the box tokens during reasoning processes and whether this approach outper- forms reasoning with plain text. Finally, we proceed to a range of downstream tasks, including text classification, text summarization, time-series classification, graph classification, and brain fMRI decoding. This comprehensive evaluation allows assess the versatility and effectiveness of Vector-ICL across different domains and task types.
Text Reconstruction: LLM Understanding of Projected Embeddings We first verify LLMs' ability to understand projected embeddings. Our results demonstrate that Vector-ICL successfully reconstructs original text from projected embeddings, with performance improving as the number of examples (shots) increases, mirroring standard in-context learning (ICL) behavior. This suggests that LLLSs can effectively decode the information compressed into the box tokens, with more context leading to better reconstruction. The similarity to ICL. behavior indicates that Vector-ICL, leverages similar learning mechanisms, but with the flexibility of working with continuous representations.
Function Regression: Enhanced Reasoning with Continuous Context We then study would it sometimes be better for LLMs to receive continuous context instead of discrete tokens. After pre- training projectors on 10-digit number reconstruction and addition between two numbers, task the LLM with learning an unknown function in-context. Results show Vector-ICL consistently out- performs few-shot ICL with raw number inputs, that have to span multiple tokens. This suggests that the continuous representations capture numerical relationships more effectively than discrete tokens, enabling LLMs better infer and apply mathematical patterns. The improvement is partic- ularly noteworthy given that LLMs are typically challenged by precise numerical computations.
Text Classification In this classical NLP task, we aggregate mean accuracy across five datasets, four encoders, and three LLMs. Results indicate that pretrained projectors provide meaningful con- tinuous context for ICL, outperforming the random baseline. LLMs achieve optimal performance with continuous context from finetuned projectors, surpassing both regular few-shot ICL and soft prompt tuning, with details shown in Table This demonstrates the versatility of Vector-ICL. across different text classification scenarios and its ability to outperform established prompt tuning meth- ods. The success across multiple datasets and LLMs suggests that the benefits of continuous context are robust and generalizable.
Text Summarization This task demands longer text generation and deeper information compre- hension. We aggregate mean RougeL scores across two datasets, and three encoders. Findings show that pretrained projectors enable LLMs to extract and condense information from a single box token, while finetumed projectors provide more effective continuous context than original textual input and soft prompts, with details shown in Table The ability to compress and later expand information from single token is particularly impressive, suggesting that Vector-ICL captures high-level se- mantic content effectively. The superior performance of finetuned projectors highlights the benefits of task-specific optimization in continuous space.
Molecule Captioning: An Unconventional NLP Task We explore LLMs' ability to compre- hend continuous vector context for out-of-distribution inputs like molecule sequences. Evaluating captioning performance with BLEU scores, we find that both pretrained and finetuned projectors provide better context than original molecule sequence text, despite encoders likely never trained on such sequences. This result is particularly intriguing as it demonstrates Vector-ICL's 's ability to bridge the gap between specialized domains and general language understanding. It suggests that continuous representations can capture and translate domain-specific information in a way that's more accessible to LLMs than raw specialized notation.
Time-series Classification We finetune non-linear projectors on the training sets of two datasets and evaluate LLM performance with the resulting continuous context. Aggregating average accu- racy, we find LLMs outperform baseline domain-specific models with finetuned classification heads. The success here suggests that continuous context can effectively capture temporal dependencies and patterns, translating time-series data into form that L.MMs can process effectively.
8

---

Published as a conference paper at ICLR 2025
Graph Classification Since the task dataset is heavily imbalanced and out of the pretraining dis- tribution of the graph encoder. We first finetune the base encoder on the target dataset to establish baseline, then train non-linear projectors the graph classification dataset. Averaging Fl scores across two L.MMs, results indicate that Vector-ICI. enables LLMSs to outperform the finetuned base- line model. This is noteworthy achievement, graph data is structurally very different from the text data that LLMs are trained on. The success here suggests that Vector-ICI. can effectively trans- late graph structures into continuous representations that preserve relational information in way that's interpretable to LLMs.
Brain fMRI Decoding and Classification We pretrain projectors on the training set using next- token generation loss, then apply them recover original text from brain IMRI signals in the test set. Results show LLMs can surpass random baselines by leveraging projected 200-dimensional fMRI PCA factors, with performance improving as context increases. This application to neuroscience data is particularly exciting, demonstrating Vector-ICL's potential in bridging neural activity and language understanding.
6 ANALYSIS
6.1 THE IMPACT OF ENCODER QUALITY ON VECTOR-ICL PERFORMANCE
We investigate the relationship between the intrinsic capabilities of encoders and their effectiveness when used with ICL for downstream tasks. To quantify the encoders' intrinsic abilities, evaluate their performance on text reconstruction task, which serves as a proxy for the amount of information preserved in the embeddings.
Our analysis focuses on text classification as the downstream task. We examine the correlation be- tween encoder rankings the reconstruction task and their corresponding rankings the classifi- cation task. This analysis is performed across 5 datasets and 3 LLMs, resulting in 15 configurations.
The results of this analysis, presented in Fig. 4a demonstrate consistent positive correlation be- tween an encoder's text reconstruction performance and its effectiveness in downstream classifica- tion tasks when used with Vector-ICL. Notably, in 4 of the 15 configurations, observe particu- larly strong correlation, with values approaching 1.
Our findings suggest that an encoder's performance the text reconstruction task can serve reliable predictor of its potential effectiveness in downstream tasks when integrated with Vector-ICI. This insight could prove valuable for practitioners in selecting encoders for Vector-ICL.
6.2
CASE STUDY: WHAT HAS BEEN LEARNED IN THE PROJECTIONS?
Fig. 4b provides visualization of the normalized Euclidean distances between projected embed- dings. Several key patterns emerge from this that offer insights into what the projector has learned.
Analysis of the numerical embedding distance matrix reveals key properties of our projection method. Embeddings for similar numbers cluster along the diagonal, indicated by lighter colors, demonstrating the preservation of local structure. Conversely. increasing distances from the di- agonal, shown by darker colors, indicate effective separation of numerically distant values in the embedding space.
Another notable feature of the distance matrix is the block structure that emerges. The block struc- ture reflects how numbers share similar digit patterns across the decimal places, from local to global blocks. It likely helps LLMs process numerical relationships more effectively, as it preserves the hierarchical nature of place-value notation.
6.3
CASE STUDY: CAN SYNTHETIC DATASET BE USEFUL IN CROSS-MODAL PRETRAINING?
The cross mrosl pretraining requires data-text dual pairs, which is not available in many cases. We explore how to create synthetic dataset for such pretraining using time-series as example. We designed data curation pipeline that extracts meaningful statistical properties from time-series and converts them into natural language descriptions.

---

Published as a conference paper at ICLR 2025
(a)
(b)
Time-smtio C'bealication
165
Nomalies Bubicc Dosiaoe
50-
1/0
60-
6:5
Prpprties Trgeets DessansPPopodee Temeebe
30-
100
10
12
14
Namber of Shonk
Figure 4: Key Insights from Encoders, Projections, and Synthetic Data Curation. (a) Corre- lation between encoders' text reconstruction performance and their downstream task effectiveness with Vector-ICI, suggesting information preservation ability predicts Vector-ICL, performance. (b) Euclidean distance matrix 1024 projected number embeddings (0 to le10) shows structured block- diagonal patterns, indicating meaningful numerical relationships are preserved. pretraining on our synthetic time-series QA dataset, which captures statistical properties through trend analysis, anomaly detection, and stability assessment. The curated QA pairs enable effective cross-modal pertaining.
Results from
Our pipeline analyzes multiple statistical aspects of the time-series data. We perform trend analysis through linear regression to capture overall directional movements, anomaly detection using z-score analysis to identify unusual patterns, and temporal stability assessment via variability thresholds to characterize consistency.
For each identified property, we generate both descriptive statements and binary questions. For instance, given time-series segment, we generate descriptive text like "The time-series shows an upward trend" or binary questions such as "Does this time-series contain any anomalies?"
Our experiments show that projectors pretrained this synthetic corpus can effectively generalize new tasks, demonstrating the feasibility of creating synthetic data for cross-modal pretraining when natural data pairs are unavailable.
7 DISCUSSION
Limitations and Future Directions In this study, we explored variety of settings: utilizing dif- ferent encoders, LLM architectures, modalities, and datasets. Our results demonstrate that LLMs are capable of performing Vector-ICL on both language and non-language inputs. However, our experiments did not cover all possible combinations these variables. There are still many unex- plored areas, such additional modalities, tasks, and encoder-decoder configurations, that could further benefit from Vector-ICL. Also, we only experimented with single- encoders, while there exist encoders that produce variable-sized embeddings, that can potentially be more powerful and flexible. Analyzing how instruction tuning might affect the model's ability to understand vector context would be beneficial as well. We leave this extensive exploration for future research fully understand the broader applicability and limitations of our approach across diverse domains.
Conclusion In this work. we explore whether large language models trained only on text can perform in-context learning continuous vectors from different domains. Our findings suggest that LLMS can indeed understand and process continuous context via embedding projection. Sim- ple linear projections are often sufficient, though for cross-modal tasks-such as those involving non-textual data like time-series or graphs-non-linee experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and task- specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.
transformations may be required.
our
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclu- sions or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.
REFERENCES
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanic Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D Co-Reyes, Eric Chu, al. Many-shot in-context learning. arXiv preprint arXiv:2404. 1018, 2024.
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn imple- ment preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Ekin Akyürek, Dale Schuurmans. Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algo- rithm in-context learning? investigations with linear models. arXiv preprint arXiv:2211.1566 2022.
Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language time series. arXiv preprint arXiv:2403.07815, 2024.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
Vinamra Benara, Chandan Singh, John X Morris, Richard Antonello, Ion Stoica, Alexander G Huth, and Jianfeng Gao. Crafting interpretable embeddings by asking Ilms questions. arXiv preprint arXiv:2405.16674. 2024.
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R Gormley, and Graham Neu- big. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00220, 2024.
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in-context learning in transformers and Ilms by learning to learn discrete functions. arXiv preprint arXiv:2310.03016. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1 2020.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. Parallel structures in pre- training data yield in-context learning. arXiv preprint arXiv:2402. 12530, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt leam in-context? language models secretly perform gradient descent meta-optimizers. In Findings of the Association for Computational Linguistics: ACI. 2023, pp. 4005-4019, 2023.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEFF/CAA Journal Automatica Sinica, 6(6): (60::1293-3000 2019.

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXhv:240.227783, 2024.
Carl Edwards, Qingyun Wang. Lawrence Zhou, and Heng Ji. L+m-24: Building dataset for language+molecules acl 2024, arXiv preprint arXiv:2433.00791, 2024.
Deqing Fu, Tian-Qi Chen, Robin Jia, and Vatsal Sharan. Transformers learn higher-order op- timization methods for in-context learning: A study with linear models. arXiv preprint arXiv:2310.1008 2023.
Angeliki Giannou, Liu Yang. Tianhao Wang, Dimitris Papailiopoulos, and Jason D Lec. How well can transformers emulate in-context newton's method? arXiv preprint 20XX::2403.03183,, 2024.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguisties: ACI-IICNLP 2021, pp. 4693-4703, Online, August 2021. Association for Computational Lin- guisties. URL https: www.anhooooooooo..ooooo wwwpooooooooooog/////o otg/ findings acl. 413
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
newrips cccppaperr2222/hass.t Abst html
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Ise: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2 arXiv:2103.094:00 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.068 2023.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning multimodal foundation models. arXiv preprint arXiv:2405.09790 2024.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329) 2022.
Amanda LeBel, Lauren Wagner, Shailce Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Mor- genthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. bioRxiv, pp. 2022-09, 2022.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi. Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models, 2024.
Mukai Li, Shansan Gong, Jiangtao Feng. Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples. arXiv preprint arXiv: 200.2049,,, 2023a.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00 101.00990, 2021.
Yingcong Li, Muhammed Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference on Machine Learming, pp. 19565-19594. PMLR, 2023b.
Chin-Yer Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona. Spain. July 2004. Association for Computational Linguis- tics. URL http::ttp//o........ / antaanaaaaantoooooo wwt.thhhooo.oooo..ooooooooo www.:r//////000000000
12

---

Published as a conference paper at ICLR 2025
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http::
7/www. www.waclwee... hww.aboorrr/aaaooooo//111--10000
Arvind Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576, 2023.
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfrembedding- enhance text retrieval with transfer learning. Salesforce Al Reseurch Blog. 3, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Sewon Min, Mike Lewis, Luke Zeitlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Compulational Linguistics: Human Language Technologies, pp. 2791-2809, 2022a.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zeutemmoer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
11048 11064, 2022b.
John X Morris, Chandan Singh, Alexander M Rush, Jianfeng Gao, and Yuntian Deng. Tree prompt- ing: efficient task adaptation without finettuning arXiv preprint arXiv:2310. 2023.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference. In International Conference on Learning Representations, 2021.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808..o 08745, 2018.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In International Conference on Machine Learning, pp. 16569-16594, PMLR, 2022.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming-We Wei Chang, et al. Large dual encoders generalizable retrievers. arXiv preprint arXiv:21
2021.
OpenAl. GPT-4 technical report, 2023.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic
evaluation of machine translation. In Proceedings the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference
Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL https:::// /www.wwwwooooooo..oo aclweb. wwwpeeb...rgaaathoogggyo.o
W18-6319
13

---

Published as a conference paper at ICLR 2025
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Con- textualized affect representations for emotion recognition. In Proceedings of the 2018 Confer- Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Bel- gium, Oetoher-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1404.
/ww. www.wwwwww.cc.oo wwwpw//wwwwwwww.a.oooo...eeeeeee http://.oorraaanccooooooo///---/
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. (631-1642) Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https:://////.. //wwww http:/wwwwwllll...ccco acwweb. org/ 2ttp://atttoooooy/111111111
Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of contin- uous language from non-invasivy brain recordings. Nature Neuroscience, 26(5):858-88)) 866, 2023.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httts:://pppeseeee
net wttp:/ormmmm--uutt---- jeJ]
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pp. 2214 2218. Citeseer, 2012.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong, Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482 2024.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordy- intsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. International Conference on Machine Learning, pp. 35151-35174 PMLR, 2023.
Albert Webson and Ellie Pavlick. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 2300-2344, 2022.
Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.0368
2023.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In The Twelfth International Conference on Learning Representations, 2023.
Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey. arXiv preprint arXiv:2409.1533 15310, 2024.
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 106.7 2024.
Chengxuan Ying. Tianle Cai. Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 14:28877-288888 2021.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0 04652, 2024.
14

---

Published as a conference paper at ICLR 2025
Dun Zhang. Stella, 2024. URL /tt.t//ppp//aaacll.eeeee/////aee 5B v5
1.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett, Trained transformers learn linear models in-context. arXiv preprini arXiv:2306.099) 09927, 2023.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. arXiv preprint 1xxiv:23300115522 2023.
Siyan Zhao, Tung Nguyen, and Aditya Grover. Probing the decision boundaries of in-context learn- ing in large language models. arXiv preprint arXiv:2406. 112006..12333 1233, 2024.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DETAILED EXPERIMENT SETUP
Text Reconstruction We use two datasets for the text reconstruction task, Parallel Sentence Talks's English subset (Tiedemann 2012) and Quora (Thakur et al. 202T).
Parallel Sentence Talks consist of sentences used in movie conversations, and Quora is built on a wide range of online questions. They represent short pieces of text that convey clear information. We aim to explore whether I.LMs can decode the original message from the projected text embeddings.
We use the following prompt template:
Translate the text brackets: (D), translation: [Original Text]
We train the projectors the training set of these two datasets and evaluate their performance the test tests. We measure the reconstruction performance with the BLEU score (Papineni et al.) 2002
Post 2018).
Arithmetic and Function Regression We created synthetic datasets of numerical data pretrain our linear number projectors, experimenting with two configurations: one using 3-digit numbers and the other using 10-digit numbers. In 1--8B' tokenizer, 3-digit numbers represented as single tokens, while in Mistral-7B, Qwen2-7B, and Yi-1.5-9B, numbers larger than 10 are split into multiple tokens. Consequently, 10-digit numbers consistently span multiple tokens across all models, which increases the complexity of performing arithmetic operations.
To represent the numbers, we use a concalenateed-an--latteee digit. For instance, a 3-digit number is represented as a 3 10 matrix (one hot per digit place), which is then flattened into a 30-dimensional vector. Similarly, 10-digit number is represented as a 10 x 10 matrix, flattened into 100-dimensional vector.
one-hot vector encoding for each
The pretraining involves two tasks. The first task number reconstruction, we use the following prompt template, given the number is 128:
D. x equals to (digits): 128
The second task is basic addition, we use the following prompt template, given the numbers are x 128 y = 256, a b
r D y -. a + b equals to (digits): 384
Here, and are randomly sampled from [o, 1] with up two decimal places, and the solution is the integer part of the sum.
For evaluation, we use function regression task with a non-linear function: f(z,y) y) vi vy.
The LLM is given inputs x and y. along with the integer part of the output f(r, v). The model is then tasked with predicting the output for new pairs of and y. The prompt for in-context learning is structured follows:
Dn, y y function(1r, y) equals to (digits): [Solution]
For few-shot ICI., the box tokens will be replaced with the actual numbers. We measure the function regression performance with the mean relative error, where the relative error computed as the & difference divided by the ground truth value.
Text Classification We use five datasets for the text classification task. For binary classification, we include IMDB (Mass al. (20TT) Rotten Tomatoes (Pang Lee 2005), and the Stanford Sen- timent Treebank (SST2) (Socher al. 2013). For multi-class classification, we use the Emotion dataset (Saravia et al. 2018) and the Financial Phrasebank (Malo et al. 2014). The binary classifi- cation datasets (IMDB, Rotten Tomatoes, and SST2) involve classifying movie reviews as positive or negative. The Emotion dataset classifies Twitter tweets into six categories: anger, fear, joy, love, sadness, and surprise. The Financial Phrasebank categorizes financial news into positive, negative, or neutral sentiments.
We the following prompt in ICL::
16

---

Published as a conference paper at ICLR 2025
's sentiment is [[pput Class]
For few-shot ICL, the box tokens will be replaced with the actual text. For soft prompt tuning, we use 20 virtual tokens and train for one epoch over the entire training set. We measure the classification performance with accuracy on the test set.
Text Summarization We use two datasets for the summarization task, XSum Narayan et al, 2018) and the English subset of XLSum (Hasan al. 2021). They contain newspaper articles and their summaries.
We use the following prompt in ICL:
(O)'s summarization is: [Summary of the Input]
For few-shot ICL, the box token will be replaced by the article. We measure the performance using the RougeL score with the ground truth summary on the test sets.
Molecule Captioning We use the Language + Molecules-24 24 (LPM24) dataset for the molecule captioning task, it was created for the task of molecule molecule-language translation, and contains 161K K pairs of molecule strings and their captions in the combined training and test set.
A sample molecule-captin caption pair looks like the following:
Molecule: Cele(Cl)ccce L (ewpee/cCCC((eeocccccce/eeeeeee
Caption: The molecule is a pain treatment that impacts inflammatory disease treatment.
And we use the following prompt for ICL.:
[ay' s molecule caption is: [Caption of the Input Molecule]
For few-shot ICL, we replace the box token with the actual molecule string. We measure the perfor- mance using the BLEU score between generated caption and the ground truth caption.
Brain fMRI Decoding and Classification We analyze data from LeBel et al. 2022 and Tang et al. 2023] which consists of fMRI responses for 3 human subjects as they listen to 20+ hours of narrative stories from podcasts. We preprocessed the data following Benara et al. 2024 by con- verting the IMRI responses into a 200-dimensional output using principal components analysis and labeling 10-grams of the story text at 2-second intervals using an ensemble of LLMs.
The data was separated into train set and test set by holding out the same three podcast stories from the three human subjects. We use the same pretraining methodology as text to pretrain on the brain fMRI data. As the data comes in segments of text and the recorded fMRI, randomly sample 20% of the segments to be in fMRI form and projected into box tokens, and we impose next token generation loss the rest 80%.
We evaluate the projectors by giving them randomly sampled context from the train set, that could come from different human subjects from different story, and ask them decode segments from the test set. We use the following prompt in ICL in our decoding experiments:
What is the English translation of the input?
Input: D. Response: the input in English [Text Corresponding to fMRI]
The random baseline is constructed by giving LLM the randomly sampled, shuffled text from the training set, and generating text according to it. We measure the performance using the RougeL score between the generated text and the ground truth.
We construct the classification questions around the properties of the underlying text, for example, "Does the sentence contain proper noun?" ""Does the input mention anything related to arguing?". The ground truth is obtained via GPT4o (OpenAl 2023) as binary labels. We use the following prompt in ICI. in our classification experiments, using one of the example questions:
Does the input mention anything related to arguing?
Input: D. Response (Yes or No): according to the English text of the input, the answer is [Yes/No]
17

---

Published as a conference paper at ICLR 2025
The random baseline is 50%, have downsampled and balanced the data. And use accuracy as the performance metric.
Time-series We use the Chronos (Ansari al., 2024) time-series Transformers as the encoder. Chronos was pretrained on large scale time-series and is designed to generate the next segments of the time-series. has proven effective on a wide range of time-series forecasting benchmarks. We take the output of the last time step from Chronos-baa as the time-series representation.
We datasets for the time-series classification task, FordA, and FordB, they are also part of the UCR Time Series Classification Archive (Dau et al. 2019)) ranging from 4000 to 5000 time-series for each dataset. We use the following prompt in ICL:
(0) 's class (positive, negative) is: [Input Class]
We construct synthetic pretraining corpus through comprehensive analysis of additional time- series datasets from the UCR Time Series Classification Archive (Dau et al., 2019), specifically MoteStrain, TwoLeadECG, Wafer, PhalangesOutinesSCoreed and Yoga. Our data curation frame- work includes multiple statistical approaches: trend identification via linear regression, anomaly detection using z-score analysis, and assessment temporal stability through variability thresholds. To enhance corpus diversity, we incorporate binary (yes/no) questions targeting specific time-series characteristics. The resulting QA pairs create a rich textual representation that captures the underly- ing temporal and statistical properties of the data.
We use the base encoder with trained classification head as the baseline and we measure the predic- tion performance with accuracy.
Graphs We use Graphormer (Ying et al. 2021 as the encoder model, specifically the one that was pretrained on large scale quantum chemistry graph datasets (Hu al. 2021). Since the down-stream task (ogbg-molhiv (Hu et al. (020)) is molecule property prediction dataset, and with strong class imbalance (3% positive classes), finetune the encoder on the training set to provide meaningful baselines and embeddings. We take the output prior to the classification layer of the Graphormer as the graph embedding.
We use the ogbg-molhiv (Hu et al., 2020) dataset for the graph classification task. ogbg-molhiv is a molecule property prediction dataset consisting of a total 41.12K graphs with node features and edge attributes. It has strong class imbalance of having around 3% positive class and 97% negative class.
Hence weighted sampling is adopted in the finetuning of both the bascline Graphormer and the embedding projector to yield meaningful predictions. We use the following prompt ICL:
(U)'s class (positive, negative) is: [Input Class]
We use the finetuned Graphormer as our baseline and use the Fl score as the performance metric due to the significant class imbalance.
A.2 CASE STUDY: WHAT INFORMATION WAS PERCEIVED FROM PROJECTED BRAIN FMRT?
Fig. illllustrae the mean accuracy achieved in decoding different categories of brain activity based on fMRI data. The categories, ranging from "Physical Actions and Movements" to "Conflict, Ur- gency, and Change," represent diverse cognitive and perceptual domains. The grouping and corre- sponding questions are listed in Table
The input data is noisy, and the projector is only trained with pretraining objectives, i.e., to predict the next piece of text given the current IMRI signal. We are surprised that with this pure unsu- pervised training, the LLM can still pick up meaningful signals from the projected embeddings. Notably, decoding tasks associated with "Physical Actions and Movements" and "Cognitive and Reflective Aspects" demonstrate higher mean accuracy, suggesting that these categories are more distinguishable based the fMRI embeddings.
18

---

Published as a conference paper at ICLR 2025
Plysical al and Movanceis
Socal, frmoterni, Entrpermcaal
Conthe, Engrecy, and Change
Healt, Dasaas Bislugical Aupeds
Seroey Peropeaal Diecrpeunee
Laegrops Sinctov, Systox
Nambeer Quaritatiee sndiun
Mean Accuracy
Figure 5: Analyzing LLM's understanding projected brain fMRI embeddings after only unsuper- vised pretraining. We categorize the underlying questions related to the text and measure the mean accuracy for each category, highlighting the L.MM's ability to interpret embeddings, with only the next token prediction pretraining.
A.3 TEXT CLASSIFICATION AND TEXT SUMMARIZATION CONFIG IN TABLES
Llama3.1 1-8B used as the common LLM and NV-Embed-vl used as the text encoder for Vector- ICL. The ICL and soft- soft-prompt methods are supplied with up to 50 shots as context. The soft-promp! tokens are trained over entire training set.
19

---

Published as a conference paper at ICLR 2025
Table 3: Hyperparameters for V-ICL training.
Hyperparameter
Value
Learning Rate
le-3
Learning Rate Schedule
Cosine Annealing
Optimizer
AdamW
81
0.9
B2
0.999
Training dtype
hf16
Batch Size Generation Temperature
128
2c-1
A.4 ADDITIONAL EXPERIMENTAL RESULTS
Quera
Parallel Sentences Talks
60-
40-
35-
50-
30-
BLEU
Encoder
25-
Encoder
40-
2----oddddiig--R
SFR-Embokding-2-R
NV-Embed-v1
20-
scella-en-L.5B-5
stello-en-I.5B-v5
NV-Embod-vl
15-
gus-t5-basc
gtr-tS-base
30
20
40
20
40
Number of Shots
Number Shots
Figure Text Reconstruction
Owed Caeee
Miral Ta- 403
Fred aa Ludice
Fricaled
Fiaker to Live Prgeaia
Figure 7: Function Regression 10 digits (upper) and 3 Digits (lower)
20

---

Published as a conference paper at ICLR 2025
Mendlaae
Men-llaxe Inakbut ind
Menellae tomatio
MencClane- ard
emotoo
phrnuhint
B-lracuddd francial
Maral Do-aa----iiaaaa-tta
Masal -idrrro---
BB-lito ematoo
Quee2-B-lutta francil phrtsbbsk
inds
ard
10
20 Nomtoer of
20 Nomror
20
10
23 Nomther of
Encoder
483 Dnshallingg
Figure Text Classification Pretrained Projectors
emotoa
ptrascbak
indh
nutes temattoo
and
Mitrl Do-mii--l--mmm--d
Metal "i--i-mmcccddd
Miral "t--inmmd
Meral -ii-ri-----emmm--d....
0.2
Qween- Battad
Owend-BB-aata
Qwai-7BBbatss
0.4
0.7
23
23
23
23 Nomber
23 Number
Nomme or
Nomou or
Ehroder
Figure 9: Text Classification Finetuned Projectors
21

---

Published as a conference paper at ICLR 2025
lndh
192
of Sluts
Nanher
Qward furnect
Mosaal To
Mina Maza-1lass-.1
Figure 10: Text Classification Few-shot ICL
xlsum
Xsum
16
RougeL
12
10
20
30
40
50
10
20
30
40
50
Number of Shots
Number of Shots
Encoder
gtr-t5-basc
NV-Embed-vl
stella-en-1.5BB
Figure 11: Text Summarization Pretrained Projectors
xlsum
X
22
RougeL
20
18-
16
10
20
30
40
50
10
20
30
40
50
Number of Shots
Number of Shots
Encoder
gtr-t5-basc
NV-Embed-vi
stella-on-1.5B-v5
Figure 12: Text Summarization - Finetuned Projectors
22

---

Published as a conference paper at ICLR 2025
A.5 BRAIN FMRI QUESTION CATEGORIES
Table 4: Question Categories and Their Associated Questions
Calegury
Questions
Sensory and Peecopiuaal Deseriptions
the inpur memion descrite
the inyut mimmion descrilne sound?
Does the sentesee inclade specifie sound or anditory deseription?
Dazs the injur menioon or deserite visual experience?
the input memion deserilee
sentesse descrite sensory experience?
Dors the input memion anything related to color?
Dozs the inyut memion or deseribe smell?
the input memioon anything related eyes?
the sentence deserite
experience or socnc?
Does the input descrihe a specific textare or seniation?
Does the sentence deserite specific sensstion er feeling?
Social, Emotional, and Interpersonal Content
input menioon anything related arguing?
Dues the input mention mything related to emputhy?
Dozs the sentence involve discusion about personal social values?
Dors the input discuss societal issue social justice topic?
the input meemion or descriie high emotional intensity?
Does the senience describe relationship between people?
Does the input menion or describe highly positive emotional valence?
the input memion deseriee highly negative easoiioal valence! input memioon anything related
Does the senience describe personal or social interacionn that leads to change or revelation?
Dozs the sentence express philosophical or existential query observation?
the senience involve expression of personal values beliefs?
the sentesse express sense of bellnging connection place or community?
Is the sentence emnotonally posative?
Cognitive and Reflettive Aspects
Is the senience reflective, involvinng sell-analysis or introspction!
the input invalve planning organizing?
the text include planning or dacisioe-making pricess?
Does the sentence convey decision or choice made by the narrator?
Dues the senience deserite personal reflection thought?
input
discovery raalization?
Does the input contain a sence of ambiguity?
Is the senience providing an explanation or rationale?
Does the input meniio anything related to knowlelge?
Language, Structure, and Syntax
the sentesce contain proper noun?
Does the sentence inclade cooditional clause?
Daes the sentence contain negaiion?
the
unigue unnenal word?
sentesce include dircet speech quotation?
Does the senionee include dialegue?
Dozs the sentence contain caitural reference?
the
involve recount social
event?
the injut include comparison mataphoe??
Daes the sentesee inclade technical or specialized terminslogy?
Is the semience abstrscl noler ihan concrse??
inclade Does the text descriite
the
miscomunicaiion misundestundingg
of communication?
Physical Actions and Movements
Is the senience conveying the narrato''s physical movement or action in detail!
Dors the input mention anything related motor movements?
sentense descrite physical action?
Dors the sentence describe physical sensation?
Does the sentence describe an activity related to daily life routine?
the input
anything related action?
sentence involve sputial reasoning?
Numbers und Quantitative Information
Dues the input menion number less iham 5?
Does the injut contain sumber?
the injut memion number greater than
Dues the input mention any tyyhiing related to hmithi
Does the injut menion anything related to calcolation?
Daes the inyur contain messirement?
Health, Disease, and Bloingical Aspects
input usemiin anything selated to disesses?
Does the input mention anything related to food?
Daes the injut memion anything related to age?
the input memion anything related gender?
input mention anything related disgust?
Does the injut memtii anything related to children?
Comflict, Urgeney, and Change
Does the sentence involve unexpected incident accident?
input mention anything zelated anger?
Dues the sentence convey sense of urgency husic?
Dars the sentence duseribe change in physical or emotional state?
Dors the sentence describe moment of relief resolution tension?
sentence express nurrtto's 's opinion or judgment event churacter?
23


Reviews:
[{"review_id": "ApGlGSj7Gp", "reviewer": "Reviewer_5yn3", "summary": "The paper studies the feasibility of vector-ICL, that extends the in-context learning capabilities of LLMs to continuous vectors. Authors use light-weight projectors to align embedding space of input text embedding with LLM. To train the projectors, they use general language modeling objective followed by task-specific objectives. Authors experimented with multiple tasks and modalities, e.g., text classification, summarization, time-series classification, fMRI decoding, etc.", "questions": "- How do you see vector-ICL applied in practical scenarios, considering decreasing inference cost and increasing context length of LLMs?  Given your experiments show that without task-specific fine-tuning, vector-ICL consistently outperforms tokens-based ICL?\n\n- Did you explore instruction tuning after pre-training? Authors should discuss the tradeoffs between task-specific finetuning and more general approaches like instruction tuning/ human preference training.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of using vectors for in-context learning is a new exploration, and the proposed methods for vector-ICL and evaluations show  promising results.\n\n- Using light-weight trainable projectors with simple pre-training on general task is also not very expensive and can be integrated as a part of general LLM pre-training. \n\n- Experiments cover a wide range of tasks and modalities.", "weaknesses": "- Method lacks depth. Specifically, replacing any length text for any complexity task with a single embedding may not be sufficient. Including an ablation where text is replaced with a series of vectors (one vector per sentence/ chunk) would be helpful.\n\n- Currently, the method requires task-specific fine-tuning to outperform token-ICL. I think authors should explore RLHF/ instruction finetuning datasets/ objectives to avoid task-specific finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yfRqnCwGol", "reviewer": "Reviewer_bbMX", "summary": "*Update after rebuttal*\n\nAfter discussions with the authors my main concern was resolved. It appears that soft prompting with few-shot is a good baseline to demonstrate the power of Vector-ICL. I raised my score accordingly.\n\n----------\n\nThe paper proposes a new method for in-context learning. By encoding examples into single vector representations via a separate, frozen encoder, and training a simple projection on top of the vector, LLMs are trained to perform ICL from continuous vector representations alone (Vector-ICL). The projection is trained a) via pretraining on unlabeled data and can optionally be b) finetuned on a labeled corpus.\nThe method is evaluated on a range of tasks, including text only tasks like text classification and summarization, as well as multi-modal tasks that utilize encoders for brain scans, graph data, or time-series.", "questions": "* Figure 4a: Why does the correlation vary so much between datasets and models of similar size?\n* Figure 4b: What do the axes represent?\n* lines 477-479: Can you expand on the block patterns and how they are explained?\n* How does the finetuning work precisely? Do you finetune with multiple shots already? If so, how many?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed approach is simple to understand and implement.\n* The method is evaluated on a wide range of tasks and datasets, including multiple modalities.\n* Parts of the results suggests that the models perform better with more shots.\n* The paper is relevant to the ICLR audience.", "weaknesses": "* The paper claims that with finetuning the vector-ICL method outperforms standard ICL. However, The method's baseline is standard-ICL without any finetuning, i.e. the method compares a supervised method (albeit with a weak adapter projection P) with an unsupervised one. A baseline that is obviously missing is when finetuning the base model with standard ICL, perhaps using an adapter that is equally weak. For text classification and summarization, the soft prompting baseline may be adequate, but the reason for its inclusion is never discussed.\n* The method is not well-motivated. In lines 34-38 it states that many data modalities cannot be well represented in natural language, making continuous vector representations necessary for in-context learning. While this is true, many of the experiments on non-textual either don't benefit from multiple shots or need to be trained on downstream data to work (e.g. time-series classification and graph classification in Figure 3, bottom). What's the value of Vector-ICL here? If you need to finetune a model to perform a task, can it still be called in-context learning?\n* The paper structure could be better. The information pertaining to a particular task is scattered all over the paper, such that I found myself scrolling back and forth a lot. For example, I did not find Figure 3 particularly helpful because the information necessary to understand it fully is located far away from it (e.g. what the horizontal bar represents). I think it would be better to focus on fewer tasks in the main body of the paper, but explain these really well. The remainder could go into the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0DgezpCnOW", "reviewer": "Reviewer_c8i3", "summary": "The paper is about projecting input data into an LLM embedding space, such that the LLM can perform in-context learning on the projections. The paper claims that this approach can improve the performance of the LLM on downstream tasks and opens up new application domains with Graphs, fMRI, and Time Series.", "questions": "q1: How does the proposed approach differ from soft prompt tuning (Li and Liang, 2021)?\n\nq2: In 3.3 it is mentioned that the LLM would be trained with ``conditional generation loss''. However, in Figure 2b that is referred to, the LLM is supposed to be frozen. Could you clarify this point and explain the training process in more detail, including which components are frozen and which are updated during a) pre-training and b) fine-tuning?\n\nq3: For Time Series (l 346), it is mentioned that you use the output of the last time step from Chronos-base. Doesn't this mean that the model is not able to use the full context of the time series but just the final state? What are the trade-offs between using the full context of the time series and just the final state?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "To the best of my knowledge, the proposed method is original (although some connections with soft prompt tuning should be discussed).\nThe paper is for the most part clear and the approach is interesting in the sense that it shows that LLMs can operate in-context on projected vectors. The results seem promising, as the learned projectors allow LLMs to tackle new tasks (graphs, fMRI etc) that are (assumed to be) not possible to tackle without projectors. The experimental part covers a wide range of tasks with different input types (8 tasks, with only 3 being classic text tasks and the proposed method beats its baselines in most cases. Clarity is generally good with some exceptions (see questions).", "weaknesses": "1. A weakness is that the projectors require pre-training with a language modeling objective and task-specific fine-tuning. It feels this defeats the purpose of in-context learning (i.e. not needing any training data to tackle a new task) to some extent. Authors can reflect if a change of the proposed methods name would be needed here.\n2. Although the paper uses soft prompt tuning as a baseline, the relationship of the proposed approach with soft prompt tuning (Li and Liang, 2021; and follow-up work) is not explicitly discussed, i.e. as both methods learn some artificial token representations that are placed into the prompt.\n3. The paper assumes that regular ICL is not applicable to Graphs, fMRI, and Time Series. While arguably not a natural fit, this assumption should be supported by some experiments, e.g. just using the numerical PCA vectors as text, or some flattened representation of the graph edges. As people have even thrown the parameters of another LM at an LM -- just in text format, I could imagine that even those non-text data types could be successfully encoded as text in a more straightforward manner without the need for training artificial tokens.\n4. The experiments on time-series, graph classification and two fMRI tasks do not have a baseline, but only compare the pretrained-only projectors with fine-tuned projectors. It would be interesting to see how the proposed approach compares to some standard baseline of the respective domain, i.e. graph neural networks for graphs.\n\n\nMinor point: In terms of presentation, the results description could directly link to the results, even if repetitive, e.g., Fig 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d65GdPdoPM", "reviewer": "Reviewer_rSng", "summary": "The paper explores using embeddings for examples in in-context learning (ICL), which may be more suited in non-textual tasks. The paper proposes a light-weight pretraining and finetuning scheme to obtain the encoder. The paper shows improvements over textual ICL on various non-textual tasks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper identifies an interesting research problem and presents a well thought-out exploration. The idea of using embeddings for conditional generation has been explored n various indirect forms (e.g., [1]), but the paper's focus on ICL seems to be a useful contribution.\n\n[1] Text Embeddings Reveal (Almost) As Much As Text (Morris et al., 2023)", "weaknesses": "1. The proposed framework is back to encoder-decoder. In this context, previous works like Flan-T5 have thoroughly explored the model's ICL capabilities [2]. While the paper differs since it focuses on a single embedding  and non-textual tasks, it's a bit odd to omit this context.\n\n2. Intuitively vector-ICL should not work as well as textual ICL for a lot of tasks. Clearly if the task requires a lot of symbolic data ICL is strictly more powerful than the proposed approach. But the paper seems to mostly show positive results. It'd be good to do some exploration of the cases vector-ICL is worse than textual ICL and by how much. \n\n[2] Scaling Instruction-Finetuned Language Models (Chung et al., 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)