Vector-ICL: In-context Learning with Continuous Vector Representations

## Abstract

Abstract Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data.
We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders.
By aligning input data with an LLM’s embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL.
In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance.
In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning.
We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms. † † footnotetext: Code will be available at: https://github.com/EvanZhuang/vector-icl .

## 1 Introduction

In-context learning (ICL) has emerged as a powerful paradigm in large language models (LLMs),
allowing generalization from limited examples within a given context [, ].
By providing demonstrations in the context during inference, ICL allows models to adapt to new tasks and formats without the need for retraining.
However, since LLMs are trained on discrete natural language tokens, ICL is generally learned and used through natural language, limiting its applicability to non-textual data.

We explore whether LLMs can perform ICL directly on continuous vectors, a capability that could dramatically expand their applicability. Many data modalities, such as sensor readings, financial time series, or scientific measurements, lack a natural text representation. Moreover, even for text data, information like numbers might be better represented via continuous vectors than token sequences.

In our study, we observe that LLMs can indeed understand and process continuous context via embedding projection.
This technique, which we term Vector-ICL, acts as a bridge between continuous data and the LLM’s embedding space.
Simple linear projections are often sufficient, though for cross-modal tasks—such as those involving non-textual data like time-series or graphs, non-linear transformations may be required.
We demonstrate that training the embedding projector using a straightforward next-token prediction objective enables Vector-ICL, effectively teaching the LLM to “read” continuous vectors.
Moreover, fine-tuning the projector on downstream tasks further enhances the effectiveness of continuous context, outperforming few-shot ICL and domain-specific models or tuning.

Our investigation begins with the task of text reconstruction, where we assess whether LLMs can recover information encoded in text embedding.
This serves as a proof-of-concept for Vector-ICL, showing that LLMs can indeed extract meaningful information from projected continuous vectors.
We then turn to the more complex challenge of arithmetics.
Although state-of-the-art LLMs can solve Olympiad mathematical problems [, ], they struggle with precise number processing due to the limitations inherent in their tokenization schemes.
Our results demonstrate that Vector-ICL offers a more effective approach for function approximation, particularly for large numbers that span multiple tokens, potentially opening new avenues for enhancing LLMs’ numerical reasoning capabilities.

Finally, we extend our analysis to a broad range of modalities and tasks, including text classification, summarization, molecule captioning, brain fMRI reconstruction and classification, time-series classification, and graph classification.
Across these diverse domains, LLMs exhibit competitive and often superior performance when employing Vector-ICL, revealing previously untapped capabilities of these models.
This work highlights the potential of continuous representations in enhancing LLMs’ in-context learning capacities, pushing the boundaries of what these models can achieve beyond token-based paradigms.

Figure: Figure 1: Comparing regular in-context learning to vector in-context learning. (a) In regular ICL, textual demonstrations are given as context during LLM inference. (b) In Vector-ICL, the input space is extended across multiple modalities. The input data is first encoded as embeddings, then transformed into continuous vectors which represent as box tokens ($\Box$) via embedding projection. During inference, we provide box tokens in prompts as demonstrations for ICL. We consider box tokens representing text, numerical data, brain fMRI, time series, and graphs in this study.
Refer to caption: x1.png

## 2 Related work

#### Empirical results of in-context learning

ICL has empirically shown strong performance in diverse natural-language processing tasks with very few demonstrations [, ].
In modern LLMs with long context windows, ICL has even shown performance improvements as the number of demonstrations grows to hundreds or even thousands, sometimes outperforming finetuning [, , ].
Empirically, different factors play key roles in ICL.
In smaller LLMs, ground-truth demonstrations are not required for in-context learning, while other factors such as the label space, input text distribution, and overall sequence format play an important role [].
Moreover, these LLMs can sometimes achieve strong performance even when demonstrations are intentionally irrelevant or even pathologically misleading [].
More recently, Wei et al. [] characterize these behaviors of LLMs with respect to model size, and show that larger language models perform in-context learning differently in the presence of flipped or semantically unrelated labels.
Orthogonally, different works find ways to improve ICL, e.g. by including explanations [],
or chaining ICL calls [].
ICL has shown some success in multimodal models [, ] or when applied to tabular data [].

#### Understanding ICL

Many works have investigated ICL and found that it is able to learn linear models [, ], discrete functions [], and more general algorithms [].
Some works have explicitly connected ICL in specific settings to implementing optimization steps analogous to gradient descent [, , ] and higher-order optimization methods [, , , ].
A complementary direction aims to establish statistical complexity and generalization bounds of in-context learning in transformers [, , , ].
Finally, one recent work suggests that ICL may arise from parallel structures in pretraining data [].

#### Learning to learn in-context

In contrast to the emergent ICL capabilities of LLMs, existing works have also studied how to explicitly improve ICL.
Min et al. [] propose MetaICL, a meta-training framework for finetuning pretrained LLMs to perform in-context learning on a large and diverse collection of tasks.
In the tabular domain, TNP [] and PFNs [] train transformer models to perform in-context prediction for a family of functions, which allows in-context generalization to unseen functions after training. Zhao et al. [] also propose meta-learning transformers to in-context learn group preferences, serving as an in-context learned reward model that adapts to diverse group preferences.

Figure: Figure 2: Pretraining and finetuning the projectors. Vector-ICL requires updating the parameters of a lightweight projector while keeping the encoder and decoder parameters fixed. The encoder first compresses the input into single token embeddings, and then the projector will project it to the aligned representation space for LLMs’ later use. (a) Pretraining the projector on a general language modeling corpus (or a modality-to-text dataset) enables Vector-ICL. (b) Task-specific fine-tuning makes Vector-ICL outperform few-shot ICL on natural language tasks, as well as with domain-specific models on non-language tasks.
Refer to caption: x2.png

## 3 Method: vector context via embedding projection

Figure: Figure 3: Main results: LLMs can perform Vector-ICL ($\uparrow$ = better). We show that training the embedding projector with a simple next-token prediction objective enables Vector-ICL. Fine-tuning the projector on downstream tasks further improves the use of continuous context, surpassing the performance of few-shot ICL and domain-specific baseline models. The study begins with text reconstruction to assess LLMs’ ability to interpret box token embeddings, followed by function regression to evaluate reasoning capabilities. We then demonstrate Vector-ICL’s effectiveness and applicability across various downstream tasks, including text classification, summarization, time-series classification, graph classification, and brain fMRI decoding & classification. Results in each panel are averaged over different encoders and LLMs for the diverse tasks we study; error bars show 95% confidence intervals.
Refer to caption: x3.png

### 3.1 Embedding projection

To perform Vector-ICL, an input must be transformed into a
vector context through an embedding projection.
Given a dataset $\mathcal{X}=\{x_{i}\}_{i=1}^{n}$, we assume the existence of an encoder $f_{\textrm{enc}}$, that transforms the data into an abstract representation (alternatively, the raw data may already be a continuous vector).
The encoded embeddings, $f_{\textrm{enc}}(x)$, are then projected into box tokens, denoted as $\Box_{x}$.
Throughout the paper, we will use the terms “box tokens” and “projected embeddings” interchangeably.
For the decoding step, we employ an LLM, referred to as $f_{\textrm{llm}}$, to generate the output given the prompts.

We impose no constraints on the form of the input data $\mathcal{X}$; it can come from any modality. The only requirement is that the encoder $f_{\textrm{enc}}$ maps each data point $x$ into a vector space, defined as:

$$ $\displaystyle f_{\mathrm{enc}}:x\rightarrow\mathbb{R}^{d_{\mathrm{enc}}},\; \forall x\in\mathcal{X}$ (1) $$

The LLM typically processes discrete tokens $\{\mathrm{tok}_{1},\mathrm{tok}_{2},\ldots,\mathrm{tok}_{l}\}$, then maps them to text embedding space $\{\mathrm{emb}_{1},\mathrm{emb}_{2},\ldots,\mathrm{emb}_{l}\},\;\forall i,\;
\mathrm{emb}_{i}\in\mathbb{R}^{d_{\mathrm{dec}}}$.
Since we operate mostly in embedding space, we omit the tokenization step for simplicity and directly refer to text inputs as their embedding representations.

The process of embedding projection is then carried out as follows.
For linear projection, we construct a projection matrix $\mathrm{P}\in\mathbb{R}^{d_{\mathrm{enc}}\times d_{\mathrm{dec}}}$ and make the following transformations to obtain projected embedding $\Box_{x}$ given input $x$:

$$ $\displaystyle\Box_{x}:=f_{\textrm{enc}}(x)\cdot\mathrm{P}$ (2) $$

In cases where more expressive power is needed, we utilize a two-layer multi-layer perceptron (MLP) to perform the projection:

$$ $\displaystyle\Box_{x}:=\textrm{MLP}(f_{\textrm{enc}}(x))$ (3) $$

The MLP follows the architecture of the MLP block in Llama [], with an additional input projection layer to map the input dimension $\mathbb{R}^{d_{\mathrm{enc}}}$ to output dimension $\mathbb{R}^{d_{\mathrm{dec}}}$.

### 3.2 Projected embeddings as context

The projected embeddings are then utilized as context in Vector-ICL, functioning as the equivalent of the original input data.
For example, in NLP tasks, the original text snippets $x$ will first be encoded as embeddings $f_{\textrm{enc}}{(x)}$, then projected to become $\Box_{x}$, inserted into the prompt like the following:

$\Box_{x}$’s sentiment is $\ldots$ / $\Box_{x}$’s summarization is $\ldots$

Using them as the context in ICL is then natural:

$\Box_{1}$’s sentiment is happy. $\Box_{2}$’s sentiment is sad. $\Box_{x}$’s sentiment is $\ldots$

where $\Box_{1}$ and $\Box_{2}$ are in-context examples and $\Box_{x}$ is the input.

### 3.3 Training the embedding projectors

The projectors need to be trained to achieve effective projections.
We discovered that pretraining these projectors with language modeling objectives enables the ICL capabilities with vector context, and finetuning them on task datasets further improves ICL performance.

The pretraining process is depicted in Fig. 2(a).
For each text snippet, we cut it into two pieces with the cutting point randomly sampled from the end of sentences.
The first half is encoded and projected while the second is kept intact.
The rest is the same with any pretraining process, the language model generated the next token distribution at each input position, except for the ones preceding the projected embeddings, and a cross-entropy loss is imposed on top of this.
With the encoder and LLM frozen, the gradient backpropagates to the projector, updating its parameters.

For non-text data modalities, pretraining can be more flexible. We define this pretraining as involving general, non-task-specific objectives, such as reconstructing a number from its embeddings (e.g., $\Box_{x}$ is 32768), performing basic algebra (e.g., $\Box_{x}$ + $\Box_{y}$ = 16384), or predicting the next token from brain fMRI embeddings.

The finetuning process is shown in Fig. 2(b).
It utilizes additional structured prompts and trains with task-specific datasets.
Similarly, the input is first mapped into the embedding space and projected into $\Box$ tokens.
They are then inserted into structured prompts, while the LLM is trained with conditional generation loss given those prompts.

## 4 Experimental setup

Table 1 gives an overview of our experimental setup, including specifics for the task, datasets, encoders, LLMs, and task-specific prompts we use.
Across different tasks, we project to four open-weights LLMs.
We now provide details for individual tasks.

**Table 1: Overview of the tasks, datasets, encoders, large language models, and prompts used in the experiments. Each task utilizes encoders and LLMs to perform functions across multiple modalities. The table highlights the diversity of models and configurations applied to each task.**
| Task | Dataset | Encoder | LLM | Prompt |
| --- | --- | --- | --- | --- |
| Text Reconstruction | Quora []<br>PST [] | \scalerel*X, \scalerel*X, \scalerel*X, \scalerel*X | \scalerel*X | Translate the text in brackets: $(\Box)$<br>Translation: [Text] |
| Function Regression | 10 Digits Regression | Digit Embedding | \scalerel*X, \scalerel*X,<br>\scalerel*X, \scalerel*X | $x=\Box_{x},\;y=\Box_{y}$,<br>function($x,y$) equals to (digits): [Solution] |
| Text Classification | IMDB []<br>Rotten Tomatoes []<br>SST2 []<br>Emotion []<br>Financial Phrasebank [] | \scalerel*X, \scalerel*X, \scalerel*X, \scalerel*X | \scalerel*X,<br>\scalerel*X,<br>\scalerel*X | $(\Box)$’s sentiment is: [Label] |
| Text Summarization | XSum []<br>XLSum [] | \scalerel*X, \scalerel*X, \scalerel*X | \scalerel*X | $(\Box)$’s summarization is: [Summary] |
| Molecule Captioning | Language + Molecule-24 [] | \scalerel*X | \scalerel*X | $(\Box)$’s molecule caption is: [Caption] |
| Brain fMRI | [], [] | PCA of fMRI | \scalerel*X | [Question] Input: $\Box$ Response: [Answer] |
| Time-series<br>Classification | FordA, FordB [] | Chronos-base [] | \scalerel*X | $(\Box)$’s class (positive, negative) is: [Label] |
| Graph Classification | ogbg-molhiv [] | Graphormer [] | \scalerel*X, \scalerel*X | $(\Box)$’s class (positive, negative) is: [Label] |

#### Text Pretraining

To pretrain our text projectors, we leverage the WikiText-103 [] dataset, consisting of over 100 million tokens from verified high-quality Wikipedia articles.
This smaller language modeling corpus is chosen for its suitability to the lightweight nature of our projectors. The pretraining process is illustrated in Fig. 2(a), where text snippets are divided at random sentence-end points.
The first half is embedded and projected, while a next-token generation loss is applied to the second half.

#### Text Reconstruction

We investigate LLMs’ ability to decode original text from projected embeddings using two datasets: Parallel Sentence Talks [] and Quora [].
These datasets consist of concise text pieces that convey clear meaning.
Projectors are trained on the training sets of both datasets, and performance is evaluated using the BLEU score [, ].

#### Arithmetic and Function Regression

For the arithmetic tasks, we generated synthetic datasets containing 10-digit numbers, which are particularly challenging for LLMs as they require splitting the numbers into multiple text tokens. These numbers are represented using a concatenated one-hot encoding per digit.
For instance, a 10-digit number is represented as a $10\times 10$ matrix, flattened into a 100-dimensional vector.
The pretraining phase includes two key tasks: number reconstruction, where the model is tasked with recovering the original number from its embedding, and basic arithmetic, where the model performs algebraic addition operations on the projected embeddings.

To evaluate the models’ arithmetic reasoning abilities, we employ a non-linear function regression task, where the function is defined as $f(x,y)=\sqrt{x}\sqrt{y}$. The model is provided with inputs $x$ and $y$, and it must predict the integer part of the function output. Performance is measured using the mean relative error, calculated as the $\ell_{1}$ difference between the predicted and true values, normalized by the ground truth. This task allows us to assess the models’ ability to perform more complex numerical reasoning beyond simple arithmetic operations.

#### Text Classification

We assess whether Vector-ICL can be applied effectively to text classification.
Both binary and multi-class classification datasets are used,
and the results are compared across few-shot ICL and soft prompt tuning.
The classification performance is measured by accuracy.

#### Text Summarization

Following the classification tasks, we explore Vector-ICL’s capability in summarizing text based on the projected embeddings.
The datasets, encoders, LLMs, and prompt templates can be found in Table 1.
Performance is evaluated using RougeL [].

#### Molecule Captioning

We also extend our approach to the unconventional task of molecule captioning, using molecule sequence-caption pairs from the Language + Molecules-24 (LPM24) [] dataset.
A sample molecule-caption pair looks like the following:

Molecule: Cc1c(Cl)cccc1-n1ccn2c(SCC(=O)c3ccccc3C(F)(F)F)nnc2c1=O
Caption: The molecule is a pain treatment that impacts inflammatory disease treatment.

This task explores whether LLMs can extract useful information from projected embeddings of out-of-distribution chemical sequences, with performance evaluated via BLEU score.

#### Brain fMRI Decoding and Classification

We analyze data from   and  , which consists of fMRI responses for 3 human subjects as they listen to 20+ hours of narrative stories from podcasts.
We preprocessed the data following  , by converting the fMRI responses into a 200-dimensional output using principal components analysis and assigning classification labels to 10-grams of the story text at 2-second intervals using an LLM.

We use the same pretraining methodology as text to pretrain on the brain fMRI data (projecting on 20% of time points and imposing next-token generation loss on the remaining 80%) .
We evaluate the LLM’s capability to decode projected brain fMRI by giving them randomly sampled context from the train set, that could come from different human subjects or from a different story, and ask them to decode segments from the test set.

In addition to text reconstruction, we decode the binary labels from the fMRI responses, corresponding to questions about the underlying text, e.g. “Does the sentence contain a proper noun?”
The decoding random baseline is constructed by giving the LLM the randomly sampled, shuffled text from the training set, and generating text according to it.
We measure the performance using the RougeL score between the generated text and the ground truth text.
The classification random baseline is 50% accuracy, as we have balanced the dataset.

#### Time-series

We take the output of the last time step from Chronos-base [] as the time-series’ representation.
We use the base encoder with trained classification head as the baseline and we measure the prediction performance with accuracy.

#### Graphs

We use Graphormer [] as the encoder model, specifically the one that was pretrained on quantum chemistry graph datasets [].
Since the down-stream, ogbg-molhiv [], is a molecule property prediction dataset, and with strong class imbalance (3-4% positive classes), we finetune the encoder on the training set to provide meaningful baselines and embeddings.
We take the output prior to the classification layer of the Graphormer as the graph embedding.
Weighted sampling is adopted in the finetuning of both the baseline Graphormer and the embedding projector to yield meaningful predictions.
We use the finetuned Graphormer as our baseline and use the F1 score as the performance metric.

#### Projector Configurations

Both linear and non-linear projectors are utilized, as shown in Fig. 3, with input and output dimensions matching the encoder-decoder pairs.
Early stopping with patience of 500 steps is used during finetuning, as projectors converge quickly due to their small parameter sizes. Details of the hyperparameters used in training are provided in Table 4.

**Table 2: We compare Vector-ICL with few-shot ICL and soft prompt tuning on the five sentiment analysis datasets. Details can be found in Sec. A.2.**
| Method | Rotten Tomatoes | SST2 | IMDB | Emotion | Financial<br>Phrasebank |
| --- | --- | --- | --- | --- | --- |
| V-ICL (pretrained) | 80.60 | 78.90 | 95.04 | 41.20 | 60.72 |
| Few-shot ICL | 87.31 | 91.74 | 93.50 | 55.20 | 71.78 |
| Soft prompt [] | 93.24 | 96.21 | 95.26 | 74.15 | 78.22 |
| V-ICL (finetuned) | 88.80 | 98.16 | 97.28 | 85.20 | 81.68 |

## 5 Results: unlocking versatile applications across modalities

Fig. 3 presents our main results, where each subplot corresponds to one of the nine tasks.
We begin by exploring text reconstruction, to see whether LLMs can comprehend the information encoded within the box tokens.
Next, we investigate the tasks of function regression to evaluate whether LLMs can leverage the box tokens during reasoning processes and whether this approach outperforms reasoning with plain text.
Finally, we proceed to a range of downstream tasks, including text classification, text summarization, time-series classification, graph classification, and brain fMRI decoding.
This comprehensive evaluation allows us to assess the versatility and effectiveness of Vector-ICL across different domains and task types.

**Table 3: We compare the performance of Vector-ICL, few-shot ICL, and soft prompt tuning across two text summarization datasets. Details can be found in Sec. A.2.**
| Method | XSum | XLSum |
| --- | --- | --- |
| V-ICL (pretrained) | 15.25 | 15.89 |
| Few-shot ICL | 19.53 | 19.41 |
| Soft prompt | 12.84 | 12.70 |
| V-ICL (finetuned) | 20.08 | 20.49 |

#### Text Reconstruction: LLM Understanding of Projected Embeddings

We first verify LLMs’ ability to understand projected embeddings.
Our results demonstrate that Vector-ICL successfully reconstructs original text from projected embeddings, with performance improving as the number of examples (shots) increases, mirroring standard in-context learning (ICL) behavior.
This suggests that LLMs can effectively decode the information compressed into the box tokens, with more context leading to better reconstruction.
The similarity to ICL behavior indicates that Vector-ICL leverages similar learning mechanisms, but with the flexibility of working with continuous representations.

#### Function Regression: Enhanced Reasoning with Continuous Context

We then study would it sometimes be better for LLMs to receive continuous context instead of discrete tokens.
After pretraining projectors on 10-digit number reconstruction and addition between two numbers, we task the LLM with learning an unknown function in-context.
Results show Vector-ICL consistently outperforms few-shot ICL with raw number inputs, that have to span multiple tokens.
This suggests that the continuous representations capture numerical relationships more effectively than discrete tokens, enabling LLMs to better infer and apply mathematical patterns.
The improvement is particularly noteworthy given that LLMs are typically challenged by precise numerical computations.

#### Text Classification

In this classical NLP task, we aggregate mean accuracy across five datasets, four encoders, and three LLMs.
Results indicate that pretrained projectors provide meaningful continuous context for ICL, outperforming the random baseline.
LLMs achieve optimal performance with continuous context from finetuned projectors, surpassing both regular few-shot ICL and soft prompt tuning, with details shown in Table 2.
This demonstrates the versatility of Vector-ICL across different text classification scenarios and its ability to outperform established prompt tuning methods.
The success across multiple datasets and LLMs suggests that the benefits of continuous context are robust and generalizable.

#### Text Summarization

This task demands longer text generation and deeper information comprehension. We aggregate mean RougeL scores across two datasets, and three encoders.
Findings show that pretrained projectors enable LLMs to extract and condense information from a single box token, while finetuned projectors provide more effective continuous context than original textual input and soft prompts, with details shown in Table 3.
The ability to compress and later expand information from a single token is particularly impressive, suggesting that Vector-ICL captures high-level semantic content effectively.
The superior performance of finetuned projectors highlights the benefits of task-specific optimization in continuous space.

#### Molecule Captioning: An Unconventional NLP Task

We explore LLMs’ ability to comprehend continuous vector context for out-of-distribution inputs like molecule sequences.
Evaluating captioning performance with BLEU scores, we find that both pretrained and finetuned projectors provide better context than original molecule sequence text, despite encoders likely never trained on such sequences.
This result is particularly intriguing as it demonstrates Vector-ICL’s ability to bridge the gap between specialized domains and general language understanding.
It suggests that continuous representations can capture and translate domain-specific information in a way that’s more accessible to LLMs than raw specialized notation.

#### Time-series Classification

We finetune non-linear projectors on the training sets of two datasets and evaluate LLM performance with the resulting continuous context.
Aggregating average accuracy, we find LLMs outperform baseline domain-specific models with finetuned classification heads.
The success here suggests that continuous context can effectively capture temporal dependencies and patterns, translating time-series data into a form that LLMs can process effectively.

#### Graph Classification

Since the task dataset is heavily imbalanced and out of the pretraining distribution of the graph encoder.
We first finetune the base encoder on the target dataset to establish a baseline, then train non-linear projectors on the graph classification dataset.
Averaging F1 scores across two LLMs, results indicate that Vector-ICL enables LLMs to outperform the finetuned baseline model. This is a noteworthy achievement, as graph data is structurally very different from the text data that LLMs are trained on.
The success here suggests that Vector-ICL can effectively translate graph structures into continuous representations that preserve relational information in a way that’s interpretable to LLMs.

#### Brain fMRI Decoding and Classification

We pretrain projectors on the training set using next-token generation loss, then apply them to recover original text from brain fMRI signals in the test set.
Results show LLMs can surpass random baselines by leveraging projected 200-dimensional fMRI PCA factors, with performance improving as context increases.
This application to neuroscience data is particularly exciting, demonstrating Vector-ICL’s potential in bridging neural activity and language understanding.

## 6 Analysis

### 6.1 The Impact of Encoder Quality on Vector-ICL Performance

We investigate the relationship between the intrinsic capabilities of encoders and their effectiveness when used with Vector-ICL for downstream tasks. To quantify the encoders’ intrinsic abilities, we evaluate their performance on a text reconstruction task, which serves as a proxy for the amount of information preserved in the embeddings.

Our analysis focuses on text classification as the downstream task. We examine the correlation between encoder rankings on the reconstruction task and their corresponding rankings on the classification task. This correlation test is performed across 5 datasets and 3 LLMs, resulting in 15 configurations.

The results of this analysis, presented in Fig. 4(a), demonstrate a consistent positive correlation between an encoder’s text reconstruction performance and its effectiveness in downstream classification tasks when used with Vector-ICL. Notably, in 4 of the 15 configurations, we observe a particularly strong correlation, with values approaching 1.

Our findings suggest that an encoder’s performance on the text reconstruction task can serve as a reliable predictor of its potential effectiveness in downstream tasks when integrated with Vector-ICL. This insight could prove valuable for practitioners in selecting encoders for Vector-ICL.

### 6.2 Case study: what has been learned in the projections?

Fig. 4(b) provides a visualization of the normalized Euclidean distances between projected embeddings, where the x and y axes correspond to 1024 different numbers uniformly sampled from 0 0 to $1e10$ and the color intensity reflects the distance between each pair.
Several key patterns emerge from this visualization that offer insights into what the projector has learned.

Analysis of the numerical embedding distance matrix reveals key properties of our projection method. Embeddings for similar numbers cluster along the diagonal, indicated by lighter colors, demonstrating the preservation of local structure. Conversely, increasing distances from the diagonal, shown by darker colors, indicate effective separation of numerically distant values in the embedding space.

Another notable feature of the distance matrix is the block structure that emerges.
This pattern comes from the way we construct the numerical embeddings and it is likely beneficial for the LLM to understand numerical inputs.

Figure: (a)
Refer to caption: x4.png

### 6.3 Case study: what information was perceived from projected brain fmri?

Fig. 4(c) illustrates the mean accuracy achieved in decoding different categories of brain activity based on fMRI data.
The categories, ranging from ”Physical Actions and Movements” to ”Conflict, Urgency, and Change,” represent diverse cognitive and perceptual domains.
The grouping and corresponding questions are listed in Table 5.

The input data is noisy, and the projector is only trained with pretraining objectives, i.e., to predict the next piece of text given the current fMRI signal.
We are surprised that with this pure unsupervised training, the LLM can still pick up meaningful signals from the projected embeddings.
Notably, decoding tasks associated with “Physical Actions and Movements” and “Cognitive and Reflective Aspects” demonstrate higher mean accuracy, suggesting that these categories are more distinguishable based on the fMRI embeddings.

## 7 Discussion

#### Limitations and Future Directions

In this study, we explored a variety of settings: utilizing different encoders, LLM architectures, modalities, and datasets.
Our results demonstrate that LLMs are capable of performing Vector-ICL on both language and non-language inputs.
However, our experiments did not cover all possible combinations of these variables.
There are still many unexplored areas, such as additional modalities, tasks, and encoder-decoder configurations, that could further benefit from Vector-ICL.
Also, we only experimented with single-token encoders, while there exist encoders that produce variable-sized embeddings, that can potentially be more powerful and flexible.
We leave this extensive exploration for future research to fully understand the broader applicability and limitations of our approach across diverse domains.

#### Conclusion

In this work, we explore whether large language models trained only on text can perform in-context learning on continuous vectors from different domains.
Our findings suggest that LLMs can indeed understand and process continuous context via embedding projection.
Simple linear projections are often sufficient, though for cross-modal tasks—such as those involving non-textual data like time-series or graphs—non-linear transformations may be required.
In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and task-specific model or tuning.
We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.

## Appendix A Appendix

### A.1 Detailed Experiment Setup

#### Text Reconstruction

We use two datasets for the text reconstruction task, Parallel Sentence Talks’s English subset [] and Quora [].

Parallel Sentence Talks consist of sentences used in movie conversations, and Quora is built on a wide range of online questions.
They represent short pieces of text that convey clear information.
We aim to explore whether LLMs can decode the original message from the projected text embeddings.

We use the following prompt template:

Translate the text in brackets: $(\Box)$, translation: [Original Text]

We train the projectors on the training set of these two datasets and evaluate their performance on the test tests.
We measure the reconstruction performance with the BLEU score [, ].

#### Arithmetic and Function Regression

We created synthetic datasets of numerical data to pretrain our linear number projectors, experimenting with two configurations: one using 3-digit numbers and the other using 10-digit numbers. In Llama3.1-8B’s tokenizer, 3-digit numbers are represented as single tokens, while in Mistral-7B, Qwen2-7B, and Yi-1.5-9B, numbers larger than 10 are split into multiple tokens. Consequently, 10-digit numbers consistently span multiple tokens across all models, which increases the complexity of performing arithmetic operations.

To represent the numbers, we use a concatenated-and-flattened one-hot vector encoding for each digit.
For instance, a 3-digit number is represented as a $3\times 10$ matrix (one hot per digit place), which is then flattened into a 30-dimensional vector.
Similarly, a 10-digit number is represented as a $10\times 10$ matrix, flattened into a 100-dimensional vector.

The pretraining involves two tasks.
The first task is number reconstruction, we use the following prompt template, given the number is 128:

$x=\Box_{x}$, $x$ equals to (digits): 128

The second task is basic addition, we use the following prompt template, given the numbers are $x=128\;y=256,\;a=1,\;b=1$:

$x=\Box_{x},\;y=\Box_{y}$, $a*x+b*y$ equals to (digits): 384

Here, $a$ and $b$ are randomly sampled from $[0,1]$ with up to two decimal places, and the solution is the integer part of the sum.

For evaluation, we use a function regression task with a non-linear function: $f(x,y)=\sqrt{x}\cdot\sqrt{y}$.
The LLM is given inputs $x$ and $y$, along with the integer part of the output $f(x,y)$.
The model is then tasked with predicting the output for new pairs of $x$ and $y$.
The prompt for in-context learning is structured as follows:

$x=\Box_{x},\;y=\Box_{y}$, function($x,y$) equals to (digits): [Solution]

For few-shot ICL, the box tokens will be replaced with the actual numbers.
We measure the function regression performance with the mean relative error, where the relative error is computed as the $\ell_{1}$ difference divided by the ground truth value.

#### Text Classification

We use five datasets for the text classification task.
For binary classification, we include IMDB [], Rotten Tomatoes [], and the Stanford Sentiment Treebank (SST2) [].
For multi-class classification, we use the Emotion dataset [] and the Financial Phrasebank [].
The binary classification datasets (IMDB, Rotten Tomatoes, and SST2) involve classifying movie reviews as positive or negative.
The Emotion dataset classifies Twitter tweets into six categories: anger, fear, joy, love, sadness, and surprise.
The Financial Phrasebank categorizes financial news into positive, negative, or neutral sentiments.

We use the following prompt in ICL:

$(\Box)$’s sentiment is [Input Class]

For few-shot ICL, the box tokens will be replaced with the actual text. For soft prompt tuning, we use 20 virtual tokens and train for one epoch over the entire training set.
We measure the classification performance with accuracy on the test set.

#### Text Summarization

We use two datasets for the summarization task, XSum [] and the English subset of XLSum [].
They contain newspaper articles and their summaries.

We use the following prompt in ICL:

$(\Box)$’s summarization is: [Summary of the Input]

For few-shot ICL, the box token will be replaced by the article.
We measure the performance using the RougeL score with the ground truth summary on the test sets.

#### Molecule Captioning

We use the Language + Molecules-24 (LPM24) dataset for the molecule captioning task, it was created for the task of molecule-language translation, and contains 161K pairs of molecule strings and their captions in the combined training and test set.

A sample molecule-caption pair looks like the following:

Molecule: Cc1c(Cl)cccc1-n1ccn2c(SCC(=O)c3ccccc3C(F)(F)F)nnc2c1=O
Caption: The molecule is a pain treatment that impacts inflammatory disease treatment.

And we use the following prompt for ICL:

$(\Box)$’s molecule caption is: [Caption of the Input Molecule]

For few-shot ICL, we replace the box token with the actual molecule string.
We measure the performance using the BLEU score between the generated caption and the ground truth caption.

#### Brain fMRI Decoding and Classification

We analyze data from   and  , which consists of fMRI responses for 3 human subjects as they listen to 20+ hours of narrative stories from podcasts.
We preprocessed the data following  , by converting the fMRI responses into a 200-dimensional output using principal components analysis and labeling 10-grams of the story text at 2-second intervals using an ensemble of LLMs.

The data was separated into train set and test set by holding out the same three podcast stories from the three human subjects.
We use the same pretraining methodology as text to pretrain on the brain fMRI data.
As the data comes in as segments of text and the recorded fMRI, we randomly sample 20% of the segments to be in fMRI form and projected into box tokens, and we impose next token generation loss on the rest 80%.

We evaluate the projectors by giving them randomly sampled context from the train set, that could come from different human subjects or from a different story, and ask them to decode segments from the test set.
We use the following prompt in ICL in our decoding experiments:

What is the English translation of the input?
Input: $\Box$, Response: the input in English is [Text Corresponding to fMRI]

The random baseline is constructed by giving LLM the randomly sampled, shuffled text from the training set, and generating text according to it.
We measure the performance using the RougeL score between the generated text and the ground truth.

We construct the classification questions around the properties of the underlying text, for example, “Does the sentence contain a proper noun?”, “”Does the input mention anything related to arguing?”.
The ground truth is obtained via GPT4o [] as binary labels.
We use the following prompt in ICL in our classification experiments, using one of the example questions:

Does the input mention anything related to arguing?
Input: $\Box$, Response (Yes or No): according to the English text of the input, the answer is [Yes/No]

The random baseline is 50%, as we have downsampled and balanced the data. And we use accuracy as the performance metric.

#### Time-series

We use the Chronos [] time-series Transformers as the encoder.
Chronos was pretrained on large scale time-series and is designed to generate the next segments of the time-series.
It has proven effective on a wide range of time-series forecasting benchmarks.
We take the output of the last time step from Chronos-base as the time-series representation.

We use two datasets for time-series classification, FordA, and FordB, they are both part of the UCR Time Series Classification Archive [] ranging from 4000 to 5000 time-series for each dataset.
We use the following prompt in ICL:

$(\Box)$’s class (positive, negative) is: [Input Class]

We use the base encoder with trained classification head as the baseline and we measure the prediction performance with accuracy.

#### Graphs

We use Graphormer [] as the encoder model, specifically the one that was pretrained on large scale quantum chemistry graph datasets [].
Since the down-stream task (ogbg-molhiv []) is a molecule property prediction dataset, and with strong class imbalance (3% positive classes), we finetune the encoder on the training set to provide meaningful baselines and embeddings.
We take the output prior to the classification layer of the Graphormer as the graph embedding.

We use the ogbg-molhiv [] dataset for the graph classification task.
ogbg-molhiv is a molecule property prediction dataset consisting of a total 41.12K graphs with node features and edge attributes.
It has a strong class imbalance of having around 3% positive class and 97% negative class.

Hence weighted sampling is adopted in the finetuning of both the baseline Graphormer and the embedding projector to yield meaningful predictions.
We use the following prompt in ICL:

$(\Box)$’s class (positive, negative) is: [Input Class]

We use the finetuned Graphormer as our baseline and use the F1 score as the performance metric due to the significant class imbalance.

### A.2 Text classification and text summarization config in tables

Llama3.1-8B is used as the common LLM and NV-Embed-v1 is used as the text encoder for Vector-ICL. The ICL methods are prompted with up to 50 shots while the soft prompt is trained over the entire training set.

### A.3 Additional Experimental Results

Figure: Figure 5: Text Reconstruction
Refer to caption: x6.png

Figure: Figure 6: Function Regression - 10 digits (upper) and 3 Digits (lower)
Refer to caption: x8.png

Figure: Figure 7: Text Classification - Pretrained Projectors
Refer to caption: x10.png

Figure: Figure 8: Text Classification - Finetuned Projectors
Refer to caption: x11.png

Figure: Figure 9: Text Classification - Few-shot ICL
Refer to caption: x12.png

Figure: Figure 10: Text Summarization - Pretrained Projectors
Refer to caption: x13.png

Figure: Figure 11: Text Summarization - Finetuned Projectors
Refer to caption: x14.png

**Table 4: Hyperparameters for V-ICL training.**
| Hyperparameter | Value |
| --- | --- |
| Learning Rate | 1e-3 |
| Learning Rate Schedule | Cosine Annealing |
| Optimizer | AdamW |
| $\beta_{1}$ | 0.9 |
| $\beta_{2}$ | 0.999 |
| Training dtype | bf16 |
| Batch Size | 128 |
| Generation Temperature | 2e-1 |

### A.4 Brain fmri question categories

**Table 5: Question Categories and Their Associated Questions**
| Category | Questions |
| --- | --- |
| Sensory and Perceptual Descriptions | Does the input mention or describe a taste? |
|  | Does the input mention or describe a sound? |
|  | Does the sentence include a specific sound or auditory description? |
|  | Does the input mention or describe a visual experience? |
|  | Does the input mention or describe a texture? |
|  | Does the sentence describe a sensory experience? |
|  | Does the input mention anything related to color? |
|  | Does the input mention or describe a smell? |
|  | Does the input mention anything related to eyes? |
|  | Does the sentence describe a visual experience or scene? |
|  | Does the input describe a specific texture or sensation? |
|  | Does the sentence describe a specific sensation or feeling? |
| Social, Emotional, and Interpersonal Content | Does the input mention anything related to arguing? |
|  | Does the input mention anything related to empathy? |
|  | Does the sentence involve a discussion about personal or social values? |
|  | Does the input discuss a societal issue or social justice topic? |
|  | Does the input mention or describe high emotional intensity? |
|  | Does the sentence describe a relationship between people? |
|  | Does the input mention or describe highly positive emotional valence? |
|  | Does the input mention or describe highly negative emotional valence? |
|  | Does the input mention anything related to conflict? |
|  | Does the sentence describe a personal or social interaction that leads to a change or revelation? |
|  | Does the sentence express a philosophical or existential query or observation? |
|  | Does the sentence involve an expression of personal values or beliefs? |
|  | Does the sentence express a sense of belonging or connection to a place or community? |
|  | Is the sentence emotionally positive? |
| Cognitive and Reflective Aspects | Is the sentence reflective, involving self-analysis or introspection? |
|  | Does the input involve planning or organizing? |
|  | Does the text include a planning or decision-making process? |
|  | Does the sentence convey a decision or choice made by the narrator? |
|  | Does the sentence describe a personal reflection or thought? |
|  | Is the input about a discovery or realization? |
|  | Does the input contain a sense of ambiguity? |
|  | Is the sentence providing an explanation or rationale? |
|  | Does the input mention anything related to knowledge? |
| Language, Structure, and Syntax | Does the sentence contain a proper noun? |
|  | Does the sentence include a conditional clause? |
|  | Does the sentence contain a negation? |
|  | Does the sentence use a unique or unusual word? |
|  | Does the sentence include a direct speech quotation? |
|  | Does the sentence include dialogue? |
|  | Does the sentence contain a cultural reference? |
|  | Does the sentence involve a recount of a social or community event? |
|  | Does the input include a comparison or metaphor? |
|  | Does the sentence include technical or specialized terminology? |
|  | Is the sentence abstract rather than concrete? |
|  | Does the sentence include an account of a miscommunication or misunderstanding? |
|  | Does the text describe a mode of communication? |
| Physical Actions and Movements | Is the sentence conveying the narrator’s physical movement or action in detail? |
|  | Does the input mention anything related to motor movements? |
|  | Does the sentence describe a physical action? |
|  | Does the sentence describe a physical sensation? |
|  | Does the sentence describe an activity related to daily life or routine? |
|  | Does the input mention anything related to an action? |
|  | Does the sentence involve spatial reasoning? |
| Numbers and Quantitative Information | Does the input mention a number less than 5? |
|  | Does the input contain a number? |
|  | Does the input mention a number greater than 100? |
|  | Does the input mention anything related to arithmetic? |
|  | Does the input mention anything related to calculation? |
|  | Does the input contain a measurement? |
| Health, Disease, and Biological Aspects | Does the input mention anything related to diseases? |
|  | Does the input mention anything related to food? |
|  | Does the input mention anything related to age? |
|  | Does the input mention anything related to gender? |
|  | Does the input mention anything related to disgust? |
|  | Does the input mention anything related to children? |
| Conflict, Urgency, and Change | Does the sentence involve an unexpected incident or accident? |
|  | Does the input mention anything related to anger? |
|  | Does the sentence convey a sense of urgency or haste? |
|  | Does the sentence describe a change in a physical or emotional state? |
|  | Does the sentence describe a moment of relief or resolution of tension? |
|  | Does the sentence express the narrator’s opinion or judgment about an event or character? |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - How do you see vector-ICL applied in practical scenarios, considering decreasing inference cost and increasing context length of LLMs?  Given your experiments show that without task-specific fine-tuning, vector-ICL consistently outperforms tokens-based ICL?

- Did you explore instruction tuning after pre-training? Authors should discuss the tradeoffs between task-specific finetuning and more general approaches like instruction tuning/ human preference training.
soundness: 3
strengths: - The concept of using vectors for in-context learning is a new exploration, and the proposed methods for vector-ICL and evaluations show  promising results.

- Using light-weight trainable projectors with simple pre-training on general task is also not very expensive and can be integrated as a part of general LLM pre-training. 

- Experiments cover a wide range of tasks and modalities.
summary: The paper studies the feasibility of vector-ICL, that extends the in-context learning capabilities of LLMs to continuous vectors. Authors use light-weight projectors to align embedding space of input text embedding with LLM. To train the projectors, they use general language modeling objective followed by task-specific objectives. Authors experimented with multiple tasks and modalities, e.g., text classification, summarization, time-series classification, fMRI decoding, etc.
weaknesses: - Method lacks depth. Specifically, replacing any length text for any complexity task with a single embedding may not be sufficient. Including an ablation where text is replaced with a series of vectors (one vector per sentence/ chunk) would be helpful.

- Currently, the method requires task-specific fine-tuning to outperform token-ICL. I think authors should explore RLHF/ instruction finetuning datasets/ objectives to avoid task-specific finetuning.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: * Figure 4a: Why does the correlation vary so much between datasets and models of similar size?
* Figure 4b: What do the axes represent?
* lines 477-479: Can you expand on the block patterns and how they are explained?
* How does the finetuning work precisely? Do you finetune with multiple shots already? If so, how many?
soundness: 3
strengths: * The proposed approach is simple to understand and implement.
* The method is evaluated on a wide range of tasks and datasets, including multiple modalities.
* Parts of the results suggests that the models perform better with more shots.
* The paper is relevant to the ICLR audience.
summary: *Update after rebuttal*

After discussions with the authors my main concern was resolved. It appears that soft prompting with few-shot is a good baseline to demonstrate the power of Vector-ICL. I raised my score accordingly.

----------

The paper proposes a new method for in-context learning. By encoding examples into single vector representations via a separate, frozen encoder, and training a simple projection on top of the vector, LLMs are trained to perform ICL from continuous vector representations alone (Vector-ICL). The projection is trained a) via pretraining on unlabeled data and can optionally be b) finetuned on a labeled corpus.
The method is evaluated on a range of tasks, including text only tasks like text classification and summarization, as well as multi-modal tasks that utilize encoders for brain scans, graph data, or time-series.
weaknesses: * The paper claims that with finetuning the vector-ICL method outperforms standard ICL. However, The method's baseline is standard-ICL without any finetuning, i.e. the method compares a supervised method (albeit with a weak adapter projection P) with an unsupervised one. A baseline that is obviously missing is when finetuning the base model with standard ICL, perhaps using an adapter that is equally weak. For text classification and summarization, the soft prompting baseline may be adequate, but the reason for its inclusion is never discussed.
* The method is not well-motivated. In lines 34-38 it states that many data modalities cannot be well represented in natural language, making continuous vector representations necessary for in-context learning. While this is true, many of the experiments on non-textual either don't benefit from multiple shots or need to be trained on downstream data to work (e.g. time-series classification and graph classification in Figure 3, bottom). What's the value of Vector-ICL here? If you need to finetune a model to perform a task, can it still be called in-context learning?
* The paper structure could be better. The information pertaining to a particular task is scattered all over the paper, such that I found myself scrolling back and forth a lot. For example, I did not find Figure 3 particularly helpful because the information necessary to understand it fully is located far away from it (e.g. what the horizontal bar represents). I think it would be better to focus on fewer tasks in the main body of the paper, but explain these really well. The remainder could go into the appendix.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: q1: How does the proposed approach differ from soft prompt tuning (Li and Liang, 2021)?

q2: In 3.3 it is mentioned that the LLM would be trained with ``conditional generation loss''. However, in Figure 2b that is referred to, the LLM is supposed to be frozen. Could you clarify this point and explain the training process in more detail, including which components are frozen and which are updated during a) pre-training and b) fine-tuning?

q3: For Time Series (l 346), it is mentioned that you use the output of the last time step from Chronos-base. Doesn't this mean that the model is not able to use the full context of the time series but just the final state? What are the trade-offs between using the full context of the time series and just the final state?
soundness: 3
strengths: To the best of my knowledge, the proposed method is original (although some connections with soft prompt tuning should be discussed).
The paper is for the most part clear and the approach is interesting in the sense that it shows that LLMs can operate in-context on projected vectors. The results seem promising, as the learned projectors allow LLMs to tackle new tasks (graphs, fMRI etc) that are (assumed to be) not possible to tackle without projectors. The experimental part covers a wide range of tasks with different input types (8 tasks, with only 3 being classic text tasks and the proposed method beats its baselines in most cases. Clarity is generally good with some exceptions (see questions).
summary: The paper is about projecting input data into an LLM embedding space, such that the LLM can perform in-context learning on the projections. The paper claims that this approach can improve the performance of the LLM on downstream tasks and opens up new application domains with Graphs, fMRI, and Time Series.
weaknesses: 1. A weakness is that the projectors require pre-training with a language modeling objective and task-specific fine-tuning. It feels this defeats the purpose of in-context learning (i.e. not needing any training data to tackle a new task) to some extent. Authors can reflect if a change of the proposed methods name would be needed here.
2. Although the paper uses soft prompt tuning as a baseline, the relationship of the proposed approach with soft prompt tuning (Li and Liang, 2021; and follow-up work) is not explicitly discussed, i.e. as both methods learn some artificial token representations that are placed into the prompt.
3. The paper assumes that regular ICL is not applicable to Graphs, fMRI, and Time Series. While arguably not a natural fit, this assumption should be supported by some experiments, e.g. just using the numerical PCA vectors as text, or some flattened representation of the graph edges. As people have even thrown the parameters of another LM at an LM -- just in text format, I could imagine that even those non-text data types could be successfully encoded as text in a more straightforward manner without the need for training artificial tokens.
4. The experiments on time-series, graph classification and two fMRI tasks do not have a baseline, but only compare the pretrained-only projectors with fine-tuned projectors. It would be interesting to see how the proposed approach compares to some standard baseline of the respective domain, i.e. graph neural networks for graphs.


Minor point: In terms of presentation, the results description could directly link to the results, even if repetitive, e.g., Fig 3.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: N/A
soundness: 4
strengths: The paper identifies an interesting research problem and presents a well thought-out exploration. The idea of using embeddings for conditional generation has been explored n various indirect forms (e.g., [1]), but the paper's focus on ICL seems to be a useful contribution.

[1] Text Embeddings Reveal (Almost) As Much As Text (Morris et al., 2023)
summary: The paper explores using embeddings for examples in in-context learning (ICL), which may be more suited in non-textual tasks. The paper proposes a light-weight pretraining and finetuning scheme to obtain the encoder. The paper shows improvements over textual ICL on various non-textual tasks.
weaknesses: 1. The proposed framework is back to encoder-decoder. In this context, previous works like Flan-T5 have thoroughly explored the model's ICL capabilities [2]. While the paper differs since it focuses on a single embedding  and non-textual tasks, it's a bit odd to omit this context.

2. Intuitively vector-ICL should not work as well as textual ICL for a lot of tasks. Clearly if the task requires a lot of symbolic data ICL is strictly more powerful than the proposed approach. But the paper seems to mostly show positive results. It'd be good to do some exploration of the cases vector-ICL is worse than textual ICL and by how much. 

[2] Scaling Instruction-Finetuned Language Models (Chung et al., 2022)

Meta-review: This paper proposes Vector-ICL which encodes input demonstrations into continuous vectors via pretrained projectors and fine-tuning process to elicit LLMs' capability of understanding both vector and textual inputs. This strategy enables effective use of the ICL framework in domains lacking textual descriptions such as time-series, molecules, graphs. Extensive experiments on various domains and tasks (including across modalities) demonstrate the advantage of Vector-ICL over traditional few-shot ICL.

Strengths:
- Extending ICL capability to the continuous space is interesting and novel to some extent.
- Vector-ICL could obviously benefit diverse application domains where natural texts are missing or not feasible, such as molecules, time series data, etc.
- Extensive experiments have been conducted on a number of tasks and application domains, demonstrating a clear advantage of the framework over few-shot textual ICL. It is also shown that projector pretraining could have some extent of generalization beyond the training domain.

Weaknesses:
- This approach still requires a certain level of fine-tuning over the task-specific training data, limiting its generalizability and scalability towards general usecases (although the authors have shown that the method could outperform few-shot ICL in 4 out of 6 tasks without fine-tuning). Further exploration on more generic pretraining and evaluation without fine-tuning could be more interesting.
- Using a single vector to encode an input sample lacks capacity in representing more complex semantic meanings which could limit the adoption of Vector-ICL in complex scenarios or application domains. But given that this work pioneers the exploration of combining continuous vectors with ICL, it is acceptable to leave it as a future work.
- More discussions and comparisons between Vector-ICL and soft prompt tuning should be added, given that both paradigms combine continuous vectors with language generation.

Decision: Accept (Poster)