ELICIT: LLM Augmentation Via External In-context Capability

## Abstract

Abstract Enhancing the adaptive capabilities of large language models is a critical pursuit in both research and application.
Traditional fine-tuning methods require substantial data and computational resources, especially for enhancing specific capabilities, while in-context learning is limited by the need for appropriate demonstrations and efficient token usage.
Inspired by the expression of in-context learned capabilities through task vectors and the concept of modularization, we propose ELICIT , a framework consisting of two modules designed to effectively store and reuse task vectors to elicit the diverse capabilities of models without additional training or inference tokens.
Our comprehensive experiments and analysis demonstrate that our pipeline is highly transferable across different input formats, tasks, and model architectures. ELICIT serves as a plug-and-play performance booster to enable adaptive elicitation of model capabilities.
By externally storing and reusing vectors that represent in-context learned capabilities, ELICIT not only demonstrates the potential to operate modular capabilities but also significantly enhances the performance, versatility, adaptability, and scalability of large language models.
Our code will be publicly available 1 1 1 https://github.com/LINs-lab/ELICIT .

## 1 Introduction

Figure: Figure 1: Illustration of ELICIT, which dynamically retrieves and integrates task vectors from a capability library to augment a language model’s performance on arbitrary queries, without increasing token usage during inference.
Refer to caption: x1.png

Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP), demonstrating remarkable versatility in tackling a wide array of tasks and real-world challenges (Devlin, ; Brown, ; Han et al., ; Achiam et al., ; Touvron et al., ).
The power of these models lies in their ability to seamlessly integrate various *capabilities*, from logical reasoning (Bommasani et al., ) to common sense understanding (Talmor et al., ).
In our rapidly evolving world, a crucial aspect of LLM is the ability to efficiently adapt to new tasks or scenarios.

Traditional fine-tuning methods, while effective in enhancing specific model capabilities (Devlin, ; Thirunavukarasu et al., ; Gururangan et al., ), often fall short in providing the necessary adaptability.
These approaches are computationally intensive, leaving LLMs ill-equipped to handle the dynamic nature of real-world applications.
In-Context Learning (ICL) (Brown, ) has emerged as a promising alternative, allowing LLMs to adapt to new tasks without additional training by leveraging their inherent capabilities (Team et al., ; Vacareanu et al., ; Agarwal et al., ).
ICL, while effective, relies on carefully crafted demonstrations, requires extra overhead for each inference, and interrupts the context, potentially limiting its efficiency and flexibility (Lu et al., ; Zhao et al., ; Dong et al., ; Liu et al., ).

We envision a next step in LLM adaptation: *Can we elicit and harness the potential of LLMs’ inherent capabilities when adapting to new tasks, as demonstrated by ICL, while simultaneously maintaining efficiency and flexibility?*

Our research explores this question by introducing a novel approach ELICIT inspired by the concept of modularization (Pfeiffer et al., ; Fedus et al., ; Ding et al., ; Zhang et al., ; Xiao et al., ).
ELICIT involves the establishment of a comprehensive library of task vectors, each eliciting one in-context capability within the LLM.
For any arbitrary query text, ELICIT dynamically leverage this capability library to selectively elicit the capability, effectively and flexibly tapping into the model’s inherent capabilities on an as-needed basis.
We empirically verify the advantanges of ELICIT under $20$ tasks and $4$ models:

- •
Efficient Capability Elicitation: ELICIT aims to improve the model’s task-specific capabilities with minimal additional computational cost during inference.
Across $20$ tasks and $4$ models, ELICIT achieves an average improvement of $11.4$% over zero-shot performance while maintaining the same token usage (Section 4.2).
- •
Flexible Task Handling: ELICIT can adapt to various tasks without requiring task-specific demonstrations or rigid templates, enhancing performance on both in-domain and unseen tasks (Sections 4.2 and 4.4).
- •
Selective Capability Activation: ELICIT allows for targeted activation of specific model capabilities based on the input query In our experiments with a math-only capability library, ELICIT boosted Math performance dramatically (e.g., from $2.6$% to $21.3$% for Mistral) while maintaining or slightly improving performance in other domains (Section 4.3).
- •
Complementary Integration: ELICIT shows potential for complementary use with existing methods, offering further performance gains.
When combined with BM25 retrieval, ELICIT enhanced Pythia’s average performance from $21.9$% to $30.1$% (Section 4.5).

Our key contributions are summarized as follows:

- •
We introduce a novel, modular framework for enhancing the adaptive capabilities of LLMs on demand with minimal computational overhead.
- •
We conduct extensive experiments to evaluate our method, showcasing its effectiveness across different query formats, language models, and tasks.
- •
We provide a thorough analysis of our method, offering insights into the design choices and their contributions to overall performance.

## 2 Related Work

#### In-Context Learning.

While Brown () introduced In-Context Learning (ICL) as a simple yet effective way to enhance LLM performance by incorporating demonstrations into prompts, its applications have rapidly expanded across diverse domains.
ICL enables model to adapt to a wide array of tasks ranging from traditional NLP benchmarks to more specialized tasks such as egression (Vacareanu et al., ), kNN classification (Agarwal et al., ; Dinh et al., ), and even jailbreaking (Anil et al., ).
Researchers have actively explored various avenues to further enhance ICL’s adaptability and effectiveness. These efforts include increasing demonstration quantity (Bertsch et al., ; Agarwal et al., ; Zhang et al., ; Team et al., ), fine-tuning models for ICL (Min et al., ), leveraging prompt engineering (Nie et al., ), and implenmenting demonstration retrieval (Liu et al., ; Rubin et al., ; Li et al., ; Shi et al., ).

Concurrently, deeper insights into ICL’s underlying mechanisms have been sought through diverse perspectives. Some researchers view ICL as a process of compressing training data into task-specific vectors (Hendel et al., ), while others relate it to gradient descent (Von Oswald et al., ) or analyze it through the lens of repetition (Yan et al., ) and memorization (Golchin et al., ).
Building upon ICL advancements, we explored eliciting and harnessing LLMs’ inherent capabilities for new task adaptation, akin to ICL, while maintaining efficiency and flexibility.

#### Task representation for ICL.

Inspired by findings that intermediate representations in LLMs encode semantic meaning (Zou et al., ), researchers have explored injecting in-context learning demonstrations, encoded as function vectors, into intermediate representations to trigger desired predictions (Liu et al., ; Hendel et al., ; Todd et al., ; Li et al., ).
However, this line of work focuses on manipulating internal representations.
We are the first to comprehensively explore the modular approach of externally storing and retrieving such task representations to augment large language model capabilities.

#### Modular LLM.

Examining and understanding the modular nature of large language models (LLMs) has become a crucial area of study for researchers (Pfeiffer et al., ; Fedus et al., ; Ding et al., ; Zhang et al., ; Xiao et al., ). Initial investigations suggest that LLMs possess the capability to be broken down into distinct specialized components or modules.
Some approaches introduce additional modules or parameters for optimization, including parameter-efficient tuning techniques like adapter layers (Houlsby et al., ; Pfeiffer et al., ), prompt tuning (Liu et al., ; Ding et al., ), and parameter subset optimization methods such as BitFit (Zaken et al., ) and binary masks (Guo et al., ; Zhao et al., ).
Other approaches involve training dedicated models for task composition (Shao et al., ; Mu et al., ) or merging fine-tuned parameter adjustments (Ilharco et al., ; Panigrahi et al., ; Merullo et al., ; Yu et al., ).
Inspired by such modular perspectives, we explore the question of using task vectors in a modular way to dynamically elicit capabilities within the model.

## 3 Method

To elicit the hidden capability inside LLMs, we build our ELICIT by introducing a capability library which condenses each in-context learned capability into a task vector, and utilizing a retrieval module to strengthen the model when a task vector is helpful.

This section describes our implementation of ELICIT.
We first formally define in-context learning Task Vectors (Section 3.1), and motivate our work.
Then, we discuss the design choices of building capabilities libraries (Section 3.2), including the layer selection and intervention strategies.
Finally, we introduce our retrieval module (Section 3.3) to dynamic elicit and leverage model’s capability.

### 3.1 From ICL to Task Vectors: Formal Definitions

#### In-Context Learning (ICL).

Firstly, we define the framework for ICL.
Let $\mathcal{T}$ represent a collection of tasks.
For each task $t\in\mathcal{T}$, there exists a dataset $\mathbf{P}_{t}$ of in-context prompts.
Each prompt $\mathbf{p}_{i}^{t}\in\mathbf{P}_{t}$ is a sequence of tokens that represents the $i$-th prompt for task $t$.
Specifically, each prompt $\mathbf{p}_{i}^{t}$ consists of two components: (1) a set of $N$ input-output demonstrations $\mathbf{D}={(\mathbf{x}_{ij},\mathbf{y}_{ij})}_{j=1}^{N}$ from task $t$, where $j$ indexes the sequence of pairs ranging from $1$ to $N$, and (2) a query input $\mathbf{x}_{iq}$, which is distinct from the inputs in $\mathbf{D}$.
We formally represent an ICL prompt $\mathbf{p}_{i}^{t}$ as:

$$ $\displaystyle\mathbf{p}_{i}^{t}=[(\mathbf{x}_{i1},\mathbf{y}_{i1}),\ldots,( \mathbf{x}_{iN},\mathbf{y}_{iN}),\mathbf{x}_{iq}]\,.$ (1) $$

The Language Model (LM) aims to predict the corresponding target response $\mathbf{y}_{iq}$ for the query input $\mathbf{x}_{iq}$.
Through learning from the demonstrated input-output mappings in $\mathbf{D}$, ICL can enhances the model’s capability to perform this task.
We firstly introduce the hidden state in Transformers below.

#### Task Vector.

Previous research (Hendel et al., ) introduced the concept of a task vector in the context of ICL.
We build upon this foundation in our work.
We first introduce the definition of hidden state representations in transformer models and task vector is derived from it.

###### Definition 3.1 ( Hidden State Representation in Transformers ) .

Let $T$ be an auto-regressive transformer language model with $L$ layers.
For each layer $l\in{1,\ldots,L}$, we define $\mathbf{h}_{l}\in\mathbb{R}^{d}$ as the vector representation of the last token at layer $l$.
The computation of $\mathbf{h}_{l}$ follows the recurrence relation (Vaswani, ): $\mathbf{h}_{l}=\mathbf{h}_{l-1}+\mathbf{m}_{l}+\mathbf{a}_{l}$, where $\mathbf{m}_{l}$ is the output of a multilayer perceptron at layer $l$, and $\mathbf{a}_{l}$ is the projection of the attention output into the hidden state at layer $l$.

Having established the notion of hidden states in transformer models, we can now formally define the task vector within the ICL framework.

###### Definition 3.2 ( Task Vector $\boldsymbol{\theta}$ ) .

ICL functions by learning a task-specific mapping from demonstrations.
This mapping is represented as a task vector $\boldsymbol{\theta}$.
The task vector is derived from the activation state $\mathbf{h}_{l}$ (as defined in Definition 3.1) at a specific layer $l$, corresponding to the last token of the prompt.
This vector subsequently steers the transformer to yield pertinent outputs for given queries.

The task vector, as defined, encapsulates the essence of the task.
This leads to the following lemma, highlighting its role in simulating ICL behavior.

###### Lemma 3.3 ( Task Vector for ICL Simulation ) .

Given a task vector $\boldsymbol{\theta}$ that effectively captures the information from demonstrations in an ICL setting, we can simulate the behavior of regular ICL with only query as follows:

$$ $\displaystyle T[\mathbf{p}_{i}^{t}]\approx f(\boldsymbol{\theta};\mathbf{x}_{ iq})\,,$ $$

where:

- •
$T[\mathbf{p}_{i}^{t}]$ represents the output of the transformer model given a ICL prompt $\mathbf{p}_{i}^{t}$ defined as (1).
- •
$f(\boldsymbol{\theta};\mathbf{x}_{q})$ denotes a function that processes the query input $\mathbf{x}_{q}$ in a zero-shot manner, guided by the information encoded in the task vector $\boldsymbol{\theta}$.

###### Remark 3.4 ( Intervention of Task Vector $\boldsymbol{\theta}$ ) .

The function $f(\boldsymbol{\theta};\mathbf{x}_{q})$ mentioned in Lemma 3.3 is an abstract concept expressing that the task vector can be used to influence the model’s inference process.
In practice, $f(\boldsymbol{\theta};\mathbf{x}_{q})$ is implemented through operations on the hidden states $\mathbf{h}_{l}$ and the task vector $\boldsymbol{\theta}$. Specifically, these operations can take the following forms:

- 1.
Replacement (Hendel et al., ): The task vector $\boldsymbol{\theta}$ directly replaces the hidden state $\mathbf{h}_{l}$, i.e., $\tilde{\mathbf{h}}_{l}=\boldsymbol{\theta}$.
- 2.
Linear combination (Todd et al., ): The task vector $\boldsymbol{\theta}$ is combined linearly with the hidden state $\mathbf{h}_{l}$, i.e., $\tilde{\mathbf{h}}_{l}=\mathbf{h}_{l}+\alpha\boldsymbol{\theta}$, where $\alpha$ is an adjustable scalar parameter.

While previous research has demonstrated the existence and extractability of task vectors, it also has shown the potential for serving a technique to elicit the inherent capabilities when adapting to difference tasks as ICL, while simultaneously maintaing computational efficiency and flexibility.

We investigate the research problem through task vectors by proposing ELICIT, a framework designed to leverage these vectors for enhancing model capabilities. As shown in Figure 2, ELICIT consists of two main components:

- •
Build Capability Library: A capability library that stores task vectors representing various in-context learned capabilities.
- •
Dynamic Capability Elicitation: A dynamic retrieval module that selectively activates relevant task vectors based on the input query.

Figure: Figure 2: Overview of the proposed ELICIT framework for Large Language Model Augmentation. ELICIT consists of two modular components: (1) Build Capability Library - constructing a library of task-specific task vectors by learning from diverse task; (2) Dynamic Capability Elicitation - dynamically retrieving and integrating relevant task vectors from the library to augment the model’s capability for an arbitrary input query.
Refer to caption: x2.png

###### Definition 3.1 ( Hidden State Representation in Transformers ) .

###### Definition 3.2 ( Task Vector 𝜽 𝜽 \boldsymbol{\theta} bold_italic_θ ) .

###### Lemma 3.3 ( Task Vector for ICL Simulation ) .

###### Remark 3.4 ( Intervention of Task Vector 𝜽 𝜽 \boldsymbol{\theta} bold_italic_θ ) .

### 3.2 Building Capabilities Library

To investigate the idea of ELICIT, we first create a library of in-context learned capabilities $\boldsymbol{\Theta}=\{\{\boldsymbol{\theta}_{i}^{t}\}_{i=1}^{k}\}_{t\in
\mathcal{T}}$.
Each element in this library is represented by a task vector (as defined in Definition 3.2).
Here, $k$ denotes the number of ICL prompts for each task $t$, and we use $k=10$ for illustration.
Definition 3.2 describes $\boldsymbol{\theta}\in\mathbb{R}^{d}$ as a single layer’s hidden state.
In our implementation, we collect $\boldsymbol{\theta}\in\mathbb{R}^{L\times d}$, which includes representations from all $L$ layers, to enable the exploration of various designs for the sequential components of ELICIT.

The implementation of creating capability library involves two critical considerations: 1 *Dynamic Layer Selection for $l^{*}$*, and 2 *Intervention Strategies*.
1 determines the appropriate layer $l$ to intervene into during further reuse, utilizing the corresponding task vector, while
2 decides how to appropriately intervene the task vector to influence the model’s behavior (possible methods are described in Remark 3.4).
Our framework addresses these considerations as follows.

Dynamic Layer Selection for $l^{*}$.
The selection of the optimal layer for task vector intervention is crucial for maximizing the effectiveness of our approach (Todd et al., ; Hendel et al., ). Appendix B further illustrates the variation in the optimal layer across different tasks.
We implement a dynamic layer selection strategy to determine the optimal layer $l^{*}$ for task vector intervention.
While using a validation set to identify the optimal layer is not a novel concept, *our contribution lies in addressing the challenge of determining the intervention layer when applying the library in our proposed pipeline:
we equip each task vector with its corresponding optimal layer, pre-identified during the library construction phase, thereby enabling efficient and effective reuse of task vectors during inference*.
Our process is as follows:

- •
We obtain the task vector $\boldsymbol{\theta}\in\mathbb{R}^{L\times d}$ by processing the ICL prompt $\mathbf{p}_{i}^{t}$ (defined in (1), using $N=16$ demonstrations).
This is done by extracting the hidden states $\{\mathbf{h}_{l}\}_{l=1}^{L}$ from all layers of the language model.
We also store these ICL prompts for future use.
- •
Using the validation set, we perform a layer-by-layer analysis:
–
For each layer $l$, we intervene $\boldsymbol{\theta}_{l}$ during zero-shot query processing.
–
We measure zero-shot accuracy on the validation set for each intervention.
–
We identify $l^{*}$ as the layer yielding the best accuracy.
- •
We repeat this process for each task-model combination, creating a library where each task vector $\boldsymbol{\theta}$ has its corresponding best layer $l^{*}$.

When reusing task vectors in library, for any $\boldsymbol{\theta}\in\boldsymbol{\Theta}$, we intervene at the pre-identified optimal layer $l^{*}$ for each task-model combination.
This dynamic selection method ensures the performance of the task vector and provides a generalizable framework adaptable to different tasks.

Intervention Strategies.
The concept of intervention, formally introduced in Remark 3.4, also outlines two methods for incorporating the task vector $\boldsymbol{\theta}_{l}$ into the model’s inference process.
We evaluate these two intervention strategies:
1) linear combination of the original hidden state and task vector with varying intervene strength $\alpha$, and 2) direct replacement of the original hidden state with the task vector.
We examine the impact of these intervention strategies on both task performance and language modeling capability, with the latter measured using cross-entropy loss on the pre-training dataset (i.e, WikiText).

Figure 3 provides a detailed visualization of how varying $\alpha$ affect both accuracy and cross-entropy loss in the Llama3-8B model across a diverse set of $20$ tasks.
*Results reveal a clear trade-off between task performance and language modeling capability as intervention strength increases.*
Among the strategies tested, the additive approach $\tilde{\mathbf{h}}_{l}=\mathbf{h}_{l}+2\times\boldsymbol{\theta}_{l}$ consistently demonstrates superior performance across a wide range of tasks while minimizing degradation in language modeling ability.
Results for other models are presented in the Appendix A, showing similar trends.

In conclusion.
Our library contains $k\times\left\lvert\mathcal{T}\right\rvert$ items for each model, each consisting of three key components: (1) the ICL prompt $\mathbf{p}_{i}^{t}$, (2) the corresponding task vector $\boldsymbol{\theta}\in\mathbb{R}^{L\times d}$, and (3) the pre-identified optimal layer $l^{*}$.

Figure: Figure 3: Varying intervention strengths affect accuracy and cross-entropy loss in Llama3-8B on valid set of $20$ tasks across different layer. Higher in intervention strengths improve average task performance across layers but negatively impact language modeling capabilities. This reveals a trade-off between task-specific enhancement and general language modeling proficiency using task vectors.
Refer to caption: x3.png

### 3.3 Dynamic Capability Elicitation

After the creation of the capability library, as described in Section 3.2, we consider two considerations: i *Relevant Task Vector Selection*, and ii *Threshold-Based Filtering*.
i Relevant Task Vector Selection focuses on identifying the most relevant task vectors from the library for a given test query $q$.
We aim to find the most relevant task vectors $\boldsymbol{\theta}^{q}\in\boldsymbol{\Theta}$ stored in the library.
Unlike traditional in-context learning (ICL), we lack meta-information about the query.
ii Threshold-Based Filtering determines whether to utilize a retrieved task vector or not, to avoid compromising performance when no suitable task vectors are available in the library.
Our framework addresses these challenges as follows:

Relevant Task Vector Selection.
We address the challenge of selecting the most relevant task vectors by employing a binary classifier to calculate similarity scores.
This classifier is built upon the SimCSE RoBERTa model(^2^22princeton-nlp/sup-simcse-roberta-base), augmented with a $2$-layer Multi-Layer Perceptron (MLP) head.
The architecture incorporates ReLU activation functions and a dropout rate of $0.2$ for regularization.

We fine-tuned this model over 15 epochs using a learning rate of $2e{-}5$ on our curated dataset of $10{,}000$ examples.
The trained classifier is then used to compute similarity scores between a given query and each ICL prompt in our library.
These scores are used to rank all library items, producing a similarity list of size $k\times\left\lvert\mathcal{T}\right\rvert$.
The top-ranked task vector from this list is selected as our target for further processing.

Figure: (a) Precision-Recall Curves
Refer to caption: extracted/5921119/sources/figures/llama3-8b_all_layers_retriever_auc_natural.png

Threshold-Based Filtering.
To determine whether to utilize stored task vectors from our library, we implement a threshold-based approach using similarity scores.
This threshold is established through a comprehensive analysis of the recall-precision trade-off across our validation set, as illustrated in Figures 4(a), utilizing the aggregated similarity lists for all samples.
The AUC scores of our precision-recall curves (i.e., $0.96$) demonstrate the high effectiveness of our threshold-based approach in accurately determining whether stored task vectors require intervention.
Our evaluation of various recall levels, as shown in Figures 4(b), reveals that a recall of $0.8$ provides the optimal balance for our pipeline, other models’ results shown in Appendix D.

Our decision process of how to utilize the similarity list and threshold to choose whether to use the stored task vectors and what task vectors to apply is as follows:

- •
We implement Dynamic Top-K Thresholding (DTT). If the highest similarity score exceeds the threshold, we select the top $10$ task vectors from the ranked list for further processing.
- •
We then employ a majority voting mechanism among the optimal layers suggested by these top vectors to determine the final layer for intervention.
- •
In cases where the highest similarity score falls below the threshold, we refrain from using any stored task vector, relying instead on the model’s base capabilities.

## 4 Experiments

To comprehensively evaluate the effectiveness of ELICIT, we conduct a series of experiments designed to explore the following key questions:

- •
*Capability Elicitation Efficiency:* Can ELICIT effectively elicit the model’s capabilities without incurring significant additional computational costs?
- •
*Selective Activation:* Is ELICIT capable of selectively activating relevant capabilities as needed for specific tasks?
- •
*Complementarity:* How well does ELICIT integrate with and complement existing methods in the field?
- •
*Generalization:* Can ELICIT handle novel queries, particularly those that diverge significantly from the task vectors currently stored in the library?

### 4.1 Experiment Setup

#### Model.

We utilize decoder-only auto-regressive language models (Pythia-2.8B (Biderman et al., ), LLaMA3-8B (Dubey et al., ), and Mistral-7B (Jiang et al., )) and recurrent neural network (Mamba-2.8B (Gu & Dao, )).
Table 1 provides a comprehensive overview of these models, detailing their key characteristics including the number of parameters, layer numbers, and training context window size.
For all models, we use the corresponding huggingface implementations (Wolf et al., ).

**Table 1: Models used in this work. We consider decoder-only auto-regressive language models and recurrent neural networks that are capable of ICL. For each model, we present the number of parameters, context window during training, and the number of layers $\left\lvert L\right\rvert$.**
| Model | HuggingFace ID | Citation | Parameters | Train Length | $|L|$ |
| --- | --- | --- | --- | --- | --- |
| Llama 3 | meta-llama/Meta-Llama-3-8B | Dubey et al. () | 8B | 8k | 32 |
| Mistral | TIGER-Lab/Mistral-7B-Base-V0.2 | Jiang et al. () | 7B | 32k | 32 |
| Pythia | EleutherAI/pythia-2.8b | Biderman et al. () | 2.8B | 2k | 32 |
| Mamba | state-spaces/mamba-2.8b-hf | Gu & Dao () | 2.8B | 2k | 64 |

#### Tasks.

To assess the efficacy of our proposed pipeline across a diverse array of scenarios, we have meticulously constructed a benchmark comprising $20$ distinct tasks.
This benchmark is designed to evaluate the model’s performance on both classification and multiple-choice problems, spanning a wide spectrum of applications and complexities.
The tasks are categorized into five domains:

- •
Knowledge: CommonsenseQA (Talmor et al., ), OpenBookQA (Mihaylov et al., ), HellaSwag (Zellers et al., ), and BoolQ (Clark et al., );
- •
Reasoning: Four subsets from Big-Bench Hard (BBH) (Suzgun et al., ) and ARC-Challenge (Clark et al., );
- •
Mathematics: MathQA (Amini et al., ) and MMLU Pro-MATH (Wang et al., );
- •
Safety: Crows-Pairs (Nangia et al., ), BBQ-Age (Parrish et al., ), Ethics-Commonsense, and Ethics-Justice (Merity et al., );
- •
Natural Language Understanding (NLU): GLUE (SST-2, QNLI, MNLI) (Wang, ) and SuperGLUE (WIC, RTE) (Wang et al., ).

#### Evaluation.

To evaluate ELICIT under a real usage scenario, where the demonstrations can hardly be at the same format with the test query, we augment the test query with two additional formats different from the demonstration in library.
Furthermore, in our preliminary experiments, we find that zero-shot LLMs cannot answer properly with contextual guidance. Thus, to ensure a fair comparison with the zero-shot scenario, we add task templates before the test query. More details and examples can be found in Appendix E.

#### Baselines.

Our primary baseline is the zero-shot performance of LLMs, as our method maintains the same token usage.
For reference, we also include in-context learning (ICL) and BM25 (Robertson et al., ) retrieval of 16 examples from the same pool of examples used in constructing the capability library.
However, these are not directly comparable to our method, due to the raised nearly 20 times more tokens consuming.
The ICL baseline is task-specific, requiring knowledge of each query’s task type to use corresponding demonstrations.
In contrast, our method is task-agnostic, applicable across various tasks without needing task-specific information or prompts.

### 4.2 Efficient Capability Elicitation

**Table 2: Performance of ELICIT across model and tasks. ELICIT significantly enhances performance while maintaining the same token usage as Zero-shot, often achieving results comparable to or better than 16-shot and BM25 ICL retriever methods. This improvement is consistent across various models and tasks, demonstrating ELICIT’s efficiency and effectiveness in boosting model capabilities without increasing computational demands.**
| Model |  | # Tokens | NLU | Reasoning | Knowledge | Math | Safety | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 16-shot | 1574.7 ± 0.2 | 42.4 ± 0.9 | 36.1 ± 0.4 | 59.9 ± 0.2 | 21.1 ± 1.9 | 50.9 ± 0.6 | 42.1 ± 0.2 |
|  | bm25 | 2145.5 ± 29.1 | 44.3 ± 1.4 | 47.1 ± 0.3 | 61.6 ± 0.8 | 25.7 ± 1.8 | 45.6 ± 2.1 | 44.9 ± 0.4 |
|  | Zero-shot | 108.2 ± 1.4 | 32.2 ± 1.2 | 32.9 ± 0.2 | 42.5 ± 1.2 | 14.0 ± 1.0 | 35.5 ± 1.2 | 31.4 ± 0.7 |
| Llama3 | ELICIT | 108.2 ± 1.4 | 38.1 ± 0.9 | 46.1 ± 0.3 | 60.7 ± 1.2 | 19.4 ± 1.1 | 49.4 ± 2.1 | 42.7 ± 0.8 |
|  | 16-shot | 1797.3 ± 2.8 | 50.0 ± 0.9 | 41.6 ± 0.5 | 60.8 ± 0.9 | 17.6 ± 1.6 | 55.5 ± 1.3 | 45.1 ± 0.5 |
|  | bm25 | 2403.5 ± 32.1 | 44.5 ± 1.3 | 43.1 ± 0.4 | 60.1 ± 1.0 | 18.6 ± 0.8 | 52.3 ± 2.6 | 43.7 ± 0.7 |
|  | Zero-shot | 123.4 ± 1.7 | 29.5 ± 1.2 | 26.2 ± 0.4 | 45.5 ± 1.3 | 2.8 ± 0.1 | 36.1 ± 0.3 | 28.0 ± 0.5 |
| Mistral | ELICIT | 123.4 ± 1.7 | 36.9 ± 1.0 | 47.1 ± 0.6 | 60.4 ± 1.1 | 20.4 ± 1.1 | 47.9 ± 1.6 | 42.5 ± 0.5 |
|  | 16-shot | 1597.6 ± 2.4 | 48.1 ± 0.4 | 22.2 ± 0.8 | 12.5 ± 0.7 | 10.3 ± 0.9 | 28.2 ± 0.9 | 24.3 ± 0.4 |
|  | bm25 | 2190.5 ± 30.2 | 38.8 ± 0.6 | 20.3 ± 0.1 | 12.5 ± 0.6 | 11.7 ± 0.8 | 26.2 ± 1.3 | 21.9 ± 0.2 |
|  | Zero-shot | 109.8 ± 1.5 | 42.9 ± 0.3 | 18.9 ± 0.2 | 21.8 ± 1.6 | 7.3 ± 0.2 | 32.5 ± 1.2 | 24.7 ± 0.4 |
| Pythia | ELICIT | 109.8 ± 1.5 | 60.1 ± 0.1 | 25.7 ± 0.9 | 20.9 ± 1.2 | 14.4 ± 1.3 | 40.9 ± 2.5 | 32.4 ± 0.4 |
|  | 16-shot | 1601.5 ± 1.7 | 31.0 ± 0.9 | 24.0 ± 0.7 | 27.2 ± 1.1 | 12.6 ± 1.0 | 36.4 ± 0.4 | 26.2 ± 0.3 |
|  | bm25 | 2190.5 ± 30.2 | 34.6 ± 1.8 | 23.3 ± 0.6 | 23.8 ± 1.1 | 12.9 ± 2.1 | 29.3 ± 1.6 | 24.8 ± 0.3 |
|  | Zero-shot | 109.8 ± 1.5 | 36.1 ± 1.5 | 19.5 ± 0.5 | 17.3 ± 1.2 | 5.8 ± 1.2 | 30.2 ± 0.2 | 21.8 ± 0.2 |
| Mamba | ELICIT | 109.8 ± 1.5 | 45.6 ± 1.4 | 29.3 ± 0.7 | 30.9 ± 1.4 | 15.4 ± 1.4 | 47.3 ± 1.6 | 33.7 ± 0.7 |

#### ELICIT achieves efficiently eliciting models’ capabilities.

From Tables 2, comparing the zero-shot baseline and ELICIT, we observe that ELICIT significantly elicits model capabilities across most tasks without increasing token usage.
Across the 20 tasks, ELICIT achieves an average improvement of 11.4% across different models.
For Llama3, ELICIT improves over zero-shot by 15.1% while using the same 108.2 tokens.
ELICIT demonstrates substantial gains in Reasoning (e.g., +14.4% for LLama3) and Safety tasks (e.g., +14.0% for LLama3).
In many cases, ELICIT’s performance is comparable to or surpasses that of 16-shot and BM25 methods, despite their higher token requirements.
Furthermore, it exhibits robustness across various template formats, highlighting its versatility.

### 4.3 Selective Adaptive Activation of Capabilities

Figure: Figure 5: Performance on ELICIT across different domains when the library only contains math-related task vectors on Mistral.
Refer to caption: extracted/5921119/sources/figures/performance_distribution__mistral_math_.png

#### ELICIT elicits capability when necessary.

We demonstrate selective activation by constructing a library containing only math-related task vectors, as shown in Figure 5.
The results clearly illustrate that ELICIT significantly boosts performance in the Math domain, with a dramatic increase from $2.6$% to $21.3$%, while maintaining performance in other domains for Mistral.
Notably, the Reasoning domain also shows a slight improvement, increasing from $22.3$% to $22.4$%.
Results for other models, presented in Appendix F, demonstrate a similar trend.
The striking improvement in Math performance, coupled with the subtle gain in Reasoning and the stability in other domains, demonstrates ELICIT’s capacity for targeted capability activation, making it a flexible and efficient performance enhancer.

### 4.4 ELICIT generalize to Unseen Tasks without addtional information

#### ELICIT generalizes across unseen tasks.

In Table 3, we observe that ELICIT significantly improves model performance on unseen tasks (GLUE-COLA, BBQ Religion, Deepmind (Saxton et al., ), MMLU-Psychology, and BBH-Logical-Deduction-Five-objects) not present in its capability library.
Across all models, ELICIT consistently outperforms the Zero-shot baseline.
In several cases, it even approaches or surpasses the BM25 retrieval baseline, despite using substantially fewer tokens.
For instance, on the MMLU-Psychology task, ELICIT achieves a $21.7$% absolute improvement over Zero-shot, surpassing BM25 retrieval by $5.7$% for Llama3 Model.
These results are achieved without additional token usage and task information, demonstrating ELICIT’s efficiency, flexibility, and generalization ability.

**Table 3: ELICIT can generalize to unseen tasks. ELICIT achieves significant performance gains without additional token usage across different models and unseen tasks.**
|  |  | # Tokens | GLUE COLA | BBQ Religion | Deepmind | MMLU-Psychology | BBH-five-objects | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BM25 | 2066.9 ± 22.5 | 83.6 ± 0.8 | 54.5 ± 0.7 | 29.1 ± 1.9 | 81.6 ± 0.4 | 29.6 ± 0.0 | 55.7 ± 0.2 |
|  | Zero-shot | 102.9 ± 0.7 | 72.0 ± 0.7 | 38.6 ± 1.1 | 17.5 ± 2.6 | 54.2 ± 0.3 | 14.3 ± 0.0 | 39.3 ± 0.8 |
| Llama | ELICIT | 102.9 ± 0.7 | 73.4 ± 0.6 | 47.8 ± 0.8 | 23.1 ± 3.4 | 75.9 ± 0.2 | 26.2 ± 0.0 | 49.3 ± 0.5 |
|  | BM25 | 2300.3 ± 23.5 | 84.6 ± 1.2 | 57.6 ± 0.6 | 19.5 ± 2.2 | 74.7 ± 0.9 | 15.1 ± 0.0 | 50.3 ± 0.5 |
|  | Zero-shot | 114.6 ± 0.8 | 43.3 ± 1.1 | 35.4 ± 3.3 | 9.0 ± 0.4 | 57.9 ± 0.7 | 6.9 ± 0.0 | 30.5 ± 1.0 |
| Mistral | ELICIT | 114.6 ± 0.8 | 46.2 ± 2.0 | 45.0 ± 2.7 | 25.4 ± 1.7 | 71.2 ± 0.4 | 24.0 ± 0.0 | 42.4 ± 1.1 |
|  | BM25 | 2140.4 ± 23.8 | 67.4 ± 0.7 | 30.4 ± 2.9 | 19.6 ± 1.3 | 15.9 ± 0.4 | 14.6 ± 0.0 | 29.6 ± 0.8 |
|  | Zero-shot | 105.9 ± 0.7 | 70.3 ± 1.0 | 21.3 ± 2.9 | 10.9 ± 0.7 | 13.9 ± 0.5 | 6.2 ± 0.0 | 24.5 ± 0.4 |
| Pythia | ELICIT | 105.9 ± 0.7 | 72.4 ± 0.4 | 33.2 ± 2.7 | 23.3 ± 1.7 | 26.4 ± 0.4 | 13.9 ± 0.0 | 33.8 ± 0.4 |
|  | BM25 | 2140.4 ± 23.8 | 54.3 ± 1.3 | 29.7 ± 0.5 | 13.1 ± 2.0 | 15.1 ± 0.3 | 5.8 ± 0.0 | 23.6 ± 0.4 |
|  | Zero-shot | 105.9 ± 0.7 | 48.5 ± 0.6 | 22.0 ± 2.0 | 9.8 ± 1.3 | 19.9 ± 0.9 | 10.0 ± 0.0 | 22.0 ± 0.2 |
| Mamba | ELICIT | 105.9 ± 0.7 | 51.5 ± 1.0 | 31.0 ± 4.5 | 13.5 ± 3.0 | 24.1 ± 1.6 | 18.3 ± 0.0 | 27.7 ± 1.1 |

### 4.5 Complementary Intergration

#### ELICIT serves as a versatile plug-and-play performance booster.

Tables 4 and 9 showcase ELICIT’s remarkable ability to function as an orthogonal, plug-and-play enhancement to existing solutions like BM25 retrieval.
In Table 4, we observe consistent performance gains across all models and tasks when ELICIT is combined with BM25 retrieval.
Notably, Pythia demonstrates substantial improvements, with its average performance increasing from 21.9% to 30.1% for in-domain tasks, and from 23.6% to 28.4% on unseen tasks.
For instance, the Llama3 model on the MMLU-Psychology task achieves $81.2$% with ELICIT, surpassing BM25 retreival alone ($81.6$%), persisting across various models and tasks.
The combination with BM24 Retrieval results on unseen tasks is shown in H.
Its plug-and-play nature allows for seamless integration with existing methods, enhancing model performance in both in-domain and unseen task scenarios without compromising the benefits of current approaches.

**Table 4: ELICIT as an plug-and-play performance booster: Performance Gains When Combined with BM25 on In-Domain Tasks.**
| Model |  | NLU | Reasoning | Knowledge | Math | Safety | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama | BM25 | 44.3 ± 1.3 | 47.1 ± 0.3 | 61.6 ± 0.8 | 25.7 ± 1.8 | 45.6 ± 2.1 | 44.9 ± 0.4 |
|  | BM25+ELICIT | 43.9 ± 1.7 | 49.9 ± 0.5 | 60.5 ± 1.0 | 25.2 ± 1.2 | 48.7 ± 3.2 | 45.6 ± 0.7 |
| Mistral | BM25 | 44.5 ± 1.3 | 43.1 ± 0.4 | 60.0 ± 1.0 | 18.6 ± 0.8 | 52.3 ± 2.6 | 43.7 ± 0.8 |
|  | BM25+ELICIT | 47.7 ± 1.3 | 47.1 ± 0.6 | 61.8 ± 1.5 | 23.5 ± 2.2 | 49.2 ± 3.0 | 45.9 ± 0.2 |
| Pythia | BM25 | 38.8 ± 0.7 | 20.4 ± 0.2 | 12.6 ± 0.6 | 11.7 ± 0.7 | 26.2 ± 1.3 | 21.9 ± 0.2 |
|  | BM25+ELICIT | 60.9 ± 1.0 | 23.4 ± 1.0 | 15.4 ± 0.8 | 14.3 ± 1.5 | 36.3 ± 0.9 | 30.1 ± 0.3 |
| Mamba | BM25 | 34.6 ± 1.8 | 23.4 ± 0.6 | 23.7 ± 1.1 | 12.9 ± 2.0 | 29.3 ± 1.6 | 24.8 ± 0.3 |
|  | BM25+ELICIT | 44.4 ± 1.9 | 31.1 ± 0.3 | 30.4 ± 1.7 | 15.9 ± 2.7 | 42.3 ± 2.1 | 32.8 ± 0.3 |

## 5 Ablation Study

### 5.1 Similarity-Based Retrieve

We also explored similarity-based retrieval methods, such as cosine similarity, t-SNE distance, and Euclidean distance between the query embedding and the task vectors $\boldsymbol{\theta}$ in capability library.
However, as illustrated in Figure 6, the precision-recall curves for these methods on Llama3 exhibit very low AUC scores, with the highest being a mere 0.28.
These poor AUC values indicate that the discrimination ability of these similarity-based approaches is inadequate for effectively identifying relevant task vectors from the library.
The precision-recall curves for similarity-based methods on other models are presented in Appendix G, further highlighting their suboptimal performance.
In stark contrast, the trained retriever in our proposed design can achieve a remarkably high AUC of 0.96 (Figure 4(a)).
This substantial improvement in retrieval performance underscores the benefits of our design, which effectively leverages learning-based techniques to intelligently retrieve and integrate relevant capabilities from the library.

Figure: (a)
Refer to caption: extracted/5921119/sources/figures/llama3-8b_all_layers_cosine_auc_natural.png

### 5.2 Selection after threshold filtering

**Table 5: Ablation Study on DDT on Valid for Llama3-8B.**
|  | TP |
| --- | --- |
| zs-shot | 33.62 |
| $n$=5 | 44.71 |
| $n$=15 | 45.02 |
| $n$=10,w/o DTT | 45.17 |
| $n$=10 | 45.29 |

In Table 5, we investigate the impact of the number of selected states $n$ when the top1 similarity score reaches the threshold, as well as the effect of using Dynamic Top-K Thresholding (DTT) or not as Section 3.3 mentioned.
The results show that selecting $n=10$ and using Dynamic Top-K Thresholding achieves the best performance on the validation set.
Choosing fewer vector quantities (e.g., n=5) would limit the method’s potential, while selecting too many (e.g., n=15) could introduce irrelevant noise, thereby degrading performance.
Dynamic Top-K adaptively sets the similarity threshold to ensure that only sufficiently relevant vectors are utilized.
This ablation study highlights the rationale and effectiveness of our design choices.

## 6 Conclusion

In this paper, we explore the vision of eliciting and harnessing the potential of large language models’ inherent capabilities when adapting to new tasks, akin to in-context learning (ICL), while maintaining efficiency and flexibility.
We propose ELICIT, a novel framework consisting of two key modules: Build Capability Library and Dynamic Capability Elicitation.
ELICIT achieves consistent improvements across diverse tasks, input formats, and model architectures.
Our results show that ELICIT not only has the potential to harness models’ latent abilities without introducing substantial additional computational cost, but also advances language models’ performance, versatility, adaptability, and scalability.

## Appendix A Intervention Strategies

As mentioned in Section 3.2, we choose $\tilde{\mathbf{h}}_{l}=\mathbf{h}_{l}+\alpha\cdot\boldsymbol{\theta}_{l}$ as the intervention strategy, where $\alpha$ is a scaling factor that controls the intervention strength.
We observe the performance and cross-entropy loss across a diverse set of 20 tasks by varying $\alpha$.
The results for Mistral, Mamba, and Pythia are shown in Figures 7, 8, and 9, respectively.
The results reveal a similar trade-off between task performance and language modeling capability as the intervention strength increases.
Among the strategies tested, the additive approach consistently demonstrates superior performance across a wide range of tasks while minimizing degradation in language modeling ability.
Across different models, $\alpha$ can be set to 2.0 to achieve a good balance between task performance and language modeling capability.

Figure: Figure 7: Varying intervention strengths affect accuracy and cross-entropy loss in Mamba on valid set of 20 tasks across different layer.
Refer to caption: x4.png

Figure: Figure 8: Varying intervention strengths affect accuracy and cross-entropy loss in Mistral on valid set of 20 tasks across different layer.
Refer to caption: x5.png

Figure: Figure 9: Varying intervention strengths affect accuracy and cross-entropy loss in Pythia on valid set of 20 tasks across different layer.
Refer to caption: x6.png

## Appendix B Best Layer for different Tasks.

As mentioned in Section 3.2, we identify the optimal intervention layer for different tasks.
Figures 10, 11, and 12 illustrate that the optimal intervention layer varies significantly across tasks.
For instance, the ARC challenge task achieves the best performance when intervening in the middle layers, while the GLUE SST2 task performs best when intervening in the later layers for the Llama-3 model.
Furthermore, intervening at different layers leads to substantial performance variations.
Therefore, instead of fixing the intervention layer as in Todd et al. (), we propose a dynamic layer selection approach to identify the optimal intervention layer for each task.

Figure: Figure 10: Performance distribution varying intervention layer on Llama3 8B in 20 tasks on Valid Set.
Refer to caption: extracted/5921119/sources/figures/datasets_layer_distribution_llama3-8b.png

Figure: Figure 11: Performance distribution varying intervention layer on Mistral in 20 tasks on Valid Set.
Refer to caption: extracted/5921119/sources/figures/datasets_layer_distribution_mistral.png

Figure: Figure 12: Performance distribution varying intervention layer on Mamba in 20 tasks on Valid Set.
Refer to caption: extracted/5921119/sources/figures/datasets_layer_distribution_mamba.png

## Appendix C Data Construction for Retriever

As mentioned in Section 3.3, we train a retriever on the constructed data.
To train the retriever as a classifier, we construct data pairs to determine whether a task-specific prompt and an ICL prompt belong to the same task.
For each task and each template, we sample two examples from the validation set.
We create positive and negative pairs, where each pair consists of a task-specific prompt and an ICL prompt from the library.
Negative examples are formed by randomly pairing task-specific prompts with ICL prompts from different tasks.
We balance the data distribution across different tasks, ensuring that each task has an equal number of positive and negative pairs.

## Appendix D Recall Sweep

As mentioned in Section 3.3, we sweep the threshold determined by recall, and as shown in Figure13, the results reveal that a recall of 0.8 provides an optimal balance between accuracy and recall for our pipeline across Pythia, Mamba, and Mistral models.

We observe a similar increasing trend in performance as recall increases for all three models. At lower recall values, the intervene accuracy is close to the zero-shot accuracy, indicating that the retrieved prompts may not be relevant to the task. As recall increases, the intervene accuracy improves significantly, demonstrating the effectiveness of the proposed approach in selecting appropriate prompts for intervention.

Based on these observations, we choose a recall value of 0.8 to determine the threshold for filtering prompts across different models, as it strikes a balance between maximizing accuracy and maintaining a reasonable recall level.

Figure: (a) Pythia
Refer to caption: extracted/5921119/sources/figures/pythia_recall_vs_acc.png

## Appendix E Evaluation Setting

### E.1 ICL setting

As 4.1 states, we first primarily implement ELICIT on the traditional In-Context Learning (ICL) setting.
We find that zero-shot Large Language Models (LLMs) cannot answer properly with contextual guidance.
Although ELICIT works on such a traditional ICL setting, as shown in Table 6 and 7, the zero-shot accuracy is almost 0, which is not plausible to evaluate model’s performance.
Therefore, we think it’s not fair to augment model performance on such traditional zero-shot queries.

**Table 6: Performance of ELICIT across model and tasks in ICL setting.**
| model |  | #Tokens | NLU | Reasoning | Knowledge | Math | Safety | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 16-shot | 1553.4 ± 2.8 | 60.3 ± 1.2 | 55.6 ± 0.2 | 69.2 ± 2.0 | 27.0 ± 0.0 | 60.9 ± 0.5 | 54.6 ± 0.6 |
|  | BM25 | 1799.2 ± 26.8 | 59.4 ± 0.3 | 54.7 ± 0.2 | 66.7 ± 0.2 | 30.3 ± 0.9 | 55.4 ± 1.2 | 53.3 ± 0.5 |
|  | Zero-shot | 87.6 ± 0.8 | 0.0 ± 0.0 | 21.6 ± 0.2 | 26.2 ± 0.2 | 0.0 ± 0.0 | 3.1 ± 0.6 | 10.2 ± 0.1 |
| Llama | Ours | 87.6 ± 0.8 | 45.5 ± 1.5 | 45.2 ± 0.2 | 57.6 ± 0.1 | 12.7 ± 0.9 | 43.6 ± 2.0 | 40.9 ± 0.1 |
|  | 16-shot | 1779.0 ± 3.4 | 59.5 ± 1.5 | 51.7 ± 0.7 | 69.4 ± 1.6 | 24.0 ± 3.7 | 62.2 ± 1.8 | 53.4 ± 1.3 |
|  | BM25 | 2045.0 ± 27.9 | 57.5 ± 1.3 | 51.8 ± 0.5 | 66.2 ± 2.5 | 23.8 ± 1.6 | 59.2 ± 1.3 | 51.7 ± 0.6 |
|  | Zero-shot | 100.9 ± 1.7 | 0.1 ± 0.1 | 22.6 ± 0.8 | 16.9 ± 0.9 | 0.7 ± 0.5 | 2.7 ± 0.4 | 8.6 ± 0.1 |
| Mistral | Ours | 100.9 ± 1.7 | 31.8 ± 0.5 | 44.4 ± 0.6 | 48.0 ± 0.7 | 17.0 ± 3.1 | 41.8 ± 1.0 | 36.6 ± 0.1 |
|  | 16-shot | 1581.1 ± 0.3 | 52.7 ± 1.7 | 21.7 ± 1.0 | 13.7 ± 1.2 | 12.2 ± 1.5 | 34.1 ± 0.2 | 26.8 ± 0.2 |
|  | BM25 | 1848.6 ± 26.4 | 47.9 ± 1.7 | 20.8 ± 0.8 | 20.2 ± 0.8 | 12.3 ± 2.8 | 36.4 ± 1.0 | 27.5 ± 0.3 |
|  | Zero-shot | 88.3 ± 1.5 | 0.2 ± 0.0 | 7.7 ± 0.3 | 3.2 ± 0.4 | 0.3 ± 0.2 | 2.0 ± 0.0 | 2.7 ± 0.1 |
| Pythia | Ours | 88.3 ± 1.5 | 46.3 ± 0.8 | 23.2 ± 1.1 | 11.1 ± 0.8 | 15.2 ± 2.0 | 36.9 ± 3.0 | 26.5 ± 0.8 |
|  | 16-shot | 1581.1 ± 1.3 | 40.4 ± 1.1 | 31.9 ± 1.1 | 34.2 ± 1.0 | 15.0 ± 2.9 | 40.9 ± 2.1 | 32.5 ± 0.3 |
|  | BM25 | 1848.6 ± 26.4 | 43.7 ± 1.9 | 31.4 ± 0.0 | 25.2 ± 1.9 | 14.7 ± 2.1 | 38.3 ± 0.8 | 30.7 ± 0.5 |
|  | Zero-shot | 88.3 ± 1.5 | 0.2 ± 0.2 | 12.3 ± 0.2 | 0.6 ± 0.2 | 0.0 ± 0.0 | 5.1 ± 1.5 | 3.6 ± 0.3 |
| Mamba | Ours | 88.3 ± 1.5 | 33.7 ± 0.6 | 26.5 ± 0.6 | 25.6 ± 1.1 | 16.3 ± 2.2 | 39.2 ± 0.5 | 28.3 ± 0.2 |

**Table 7: Unseen task for ICL Setting**
|  |  | # Tokens | GLUE COLA | BBQ Religion | Deepmind | MMLU-Psychology | BBH-five-objects | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BM25 | 1684.3 ± 3.3 | 33.3 ± 1.9 | 69.3 ± 2.4 | 32.0 ± 1.4 | 83.7 ± 0.5 | 35.0 ± 0.0 | 50.7 ± 0.1 |
|  | Zero-shot | 76.5 ± 0.2 | 0.7 ± 0.5 | 5.3 ± 2.4 | 0.0 ± 0.0 | 62.7 ± 0.5 | 0.0 ± 0.0 | 13.7 ± 0.7 |
| Llama | Ours | 76.5 ± 0.2 | 1.3 ± 0.5 | 24.3 ± 3.8 | 19.3 ± 2.4 | 66.0 ± 0.0 | 5.0 ± 0.0 | 23.2 ± 1.1 |
|  | BM25 | 1913.3 ± 20.0 | 27.0 ± 1.6 | 69.0 ± 1.4 | 28.3 ± 3.9 | 79.7 ± 1.2 | 26.2 ± 0.0 | 46.1 ± 0.6 |
|  | Zero-shot | 85.5 ± 0.8 | 1.7 ± 1.2 | 1.0 ± 0.8 | 1.3 ± 1.2 | 35.3 ± 1.2 | 0.0 ± 0.0 | 7.9 ± 0.3 |
| Mistral | Ours | 85.5 ± 0.8 | 1.3 ± 1.2 | 18.3 ± 2.1 | 20.0 ± 0.8 | 54.7 ± 1.2 | 10.0 ± 0.0 | 20.9 ± 1.1 |
|  | BM25 | 1747.6 ± 20.2 | 13.7 ± 1.9 | 33.7 ± 1.7 | 20.3 ± 2.6 | 18.7 ± 1.7 | 6.2 ± 0.0 | 18.5 ± 1.2 |
|  | Zero-shot | 78.1 ± 0.7 | 43.0 ± 2.9 | 0.0 ± 0.0 | 0.3 ± 0.5 | 3.3 ± 0.5 | 0.0 ± 0.0 | 9.3 ± 0.7 |
| pythia | Ours | 78.1 ± 0.7 | 37.0 ± 2.4 | 14.7 ± 4.6 | 15.0 ± 0.8 | 13.0 ± 1.4 | 6.2 ± 0.0 | 17.2 ± 0.8 |
|  | BM25 | 1747.6 ± 20.2 | 36.7 ± 1.2 | 33.3 ± 2.6 | 25.3 ± 4.5 | 25.3 ± 0.5 | 21.2 ± 0.0 | 28.4 ± 1.1 |
|  | Zero-shot | 78.1 ± 0.7 | 19.7 ± 2.5 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 3.9 ± 0.5 |
| Mamba | Ours | 78.1 ± 0.7 | 18.0 ± 1.4 | 19.3 ± 3.4 | 13.7 ± 3.8 | 13.0 ± 0.8 | 7.5 ± 0.0 | 14.3 ± 1.1 |

### E.2 Task Specific Prompt

To ensure fair comparisons in zero-shot scenarios, we prepend task-specific prompts before each test query. These task-specific prompts are manually crafted following guidelines from lm-harness (Gao et al., ) and the chain-of-thought-hub (^3^33https://github.com/FranxYao/chain-of-thought-hub.git).
The complete set of prompts used is provided in Figure  LABEL:tab:all_prompts.

**Table 8: Task Specific Prompts for Various Tasks**
| Task | Prompts |
| --- | --- |
| bbh_date_understanding | •<br>Infer the date from context. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Determine the date based on contextual clues. End your response with ’X’, where X represents the correct option.<br>Question: {input}<br>•<br>Use the given context to deduce the date. Conclude your answer with ’X’, X being the right letter choice.<br>Question: {input} |
| bbh_boolean_expressions | •<br>Evaluate the result of a random Boolean expression.<br>Question: {input}<br>•<br>Calculate the outcome of a given Boolean expression.<br>Question: {input}<br>•<br>Determine the result of the provided Boolean logic statement.<br>Question: {input} |
| bbh_date_understanding | •<br>Infer the date from context. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Determine the date based on contextual clues. End your response with ’X’, where X represents the correct option.<br>Question: {input}<br>•<br>Use the given context to deduce the date. Conclude your answer with ’X’, X being the right letter choice.<br>Question: {input} |
| bbh_boolean_expressions | •<br>Evaluate the result of a random Boolean expression.<br>Question: {input}<br>•<br>Calculate the outcome of a given Boolean expression.<br>Question: {input}<br>•<br>Determine the result of the provided Boolean logic statement.<br>Question: {input} |
| mmlu_pro_math | •<br>The following are multiple choice questions (with answers) about math. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice math questions. Conclude your response with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Answer these math multiple-choice questions. Answer with ’X’, where X is the right letter choice.<br>Question: {input} |
| superglue_rte | •<br>{input}<br>Question: Is the hypothesis correct based on the premise? Answer True or False.<br>•<br>{input}<br>Question: Based on the premise, is the hypothesis accurate? Respond with True or False.<br>•<br>{input}<br>Question: Does the premise support the hypothesis? Answer True or False. |
| bbh_temporal_sequences | •<br>Answer questions about which times certain events could have occurred. Finish your answer with ’X’ where X is the correct letter choice.<br>Q: {input}<br>•<br>Determine possible occurrence times for specific events. Answer with ’X’, X being the correct option.<br>Q: {input}<br>•<br>Identify when certain events could have happened. Conclude with ’X’, where X is the right letter choice.<br>Q: {input} |
| bbh_reasoning_about_colored_objects | •<br>Answer extremely simple questions about the colors of objects on a surface. Finish your answer with ’X’ where X is the correct letter choice.<br>Q: {input}<br>•<br>Respond to basic questions about object colors on a surface. Answer with ’X’, X being the correct option.<br>Q: {input}<br>•<br>Address simple queries regarding the colors of items on a surface. Conclude with ’X’, where X is the right letter choice.<br>Q: {input} |
| ethics_commonsense | •<br>{input}<br>Question: Is this wrong? Answer Yes or No.<br>•<br>{input}<br>Question: Is this ethically incorrect? Respond with Yes or No.<br>•<br>{input}<br>Question: Does this violate ethical norms? Answer Yes or No. |
| ethics_justice | •<br>{input}<br>Question: Would most people believe this reasonable or unreasonable to say? Answer Yes or No.<br>•<br>{input}<br>Question: Would the majority consider this statement reasonable or unreasonable? Answer Yes or No.<br>•<br>{input}<br>Question: Is this statement generally perceived as reasonable or unreasonable? Respond with Yes or No. |
| glue_mnli | •<br>{input} True, False or Neither?<br>•<br>{input} Answer selecting from: True, False, or Neither?<br>•<br>{input} Choose one as answer: True, False, or Neither? |
| glue_sst2 | •<br>{input}<br>Question: Is this sentence positive or negative? Answer positive or negative.<br>•<br>{input}<br>Question: Does this sentence convey a positive or negative sentiment? Answer positive or negative.<br>•<br>{input}<br>Question: Is the tone of this sentence positive or negative? Respond with positive or negative. |
| glue_qnli | •<br>{input}<br>Question: Does this response answer the question? Answer Yes or No.<br>•<br>{input}<br>Question: Is the response answer to the question? Answer Yes or No.<br>•<br>{input}<br>Question: Does the given response address the question? Respond with Yes or No. |
| hellaswag | •<br>The following are multiple choice questions (with answers) about commonsense NLI. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>The following are multiple-choice questions about commonsense natural language inference. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Answer these multiple-choice questions on commonsense language understanding. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| crows_pairs | •<br>{input}<br>Question: Which sentence is more stereotypical? Answer A or B.<br>•<br>{input}<br>Question: Which sentence contains more stereotypical content? Answer A or B.<br>•<br>{input}<br>Question: Identify the sentence with stronger stereotypical elements. Respond with A or B. |
| boolq | •<br>{input}<br>Answer True or False.<br>•<br>{input}<br>Respond with True or False.<br>•<br>{input}<br>Is this statement correct? Answer True or False. |
| math_qa | •<br>The following are multiple choice questions (with answers) about math word problem. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice questions about math word problems. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Solve these multiple-choice math word problems. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| superglue_wic | •<br>{input}<br>Question: Is the word used in the same way in the two sentences above? Answer Yes or No.<br>•<br>{input}<br>Question: Is the word used similarly in both sentences above? Respond with Yes or No.<br>•<br>{input}<br>Question: Does the word have the same meaning in the two given sentences? Answer Yes or No. |
| openbookqa | •<br>The following are multiple choice questions (with answers) about multi-step reasoning. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>The following are multiple-choice questions testing multi-step reasoning. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Answer these multiple-choice questions involving multi-step logical thinking. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| commonsense_qa | •<br>The following are multiple choice questions (with answers) about commonsense knowledge reasoning. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice questions about commonsense reasoning. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Respond to these multiple-choice questions on commonsense knowledge. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| bbq_age | •<br>The following are multiple choice questions (with answers) about social bias on age. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>The following are multiple-choice questions about age-related social bias. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Answer these multiple-choice questions on social bias regarding age. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| arc_challenge | •<br>The following are multiple choice questions (with answers) about science. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice science questions. Answer with ’X’, X being the correct option.<br>Question: {input}<br>•<br>Respond to these multiple-choice questions on scientific topics. Conclude with ’X’, where X is the right letter choice.<br>Question: {input} |
| glue_cola | •<br>{input}<br>Question: Does this sentence make sense? Answer Yes or No.<br>•<br>{input}<br>Question: Is this sentence logically coherent? Respond with Yes or No.<br>•<br>{input}<br>Question: Evaluate if this sentence is meaningful. Reply with Yes or No. |
| bbh_logical_deduction_five_objects | •<br>A logical deduction task which requires deducing the order of a sequence of objects. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>This challenge involves logically determining the sequence of a set of objects. Conclude your response with ’X’, where X is the appropriate letter option.<br>Question: {input}<br>•<br>In this logical reasoning exercise, deduce the correct order of a series of objects. End your answer with ’X’, X being the right letter choice.<br>Question: {input} |
| mmlu_high_school_psychology | •<br>The following are multiple choice questions (with answers) about high school psychology. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice questions testing high school level psychology knowledge. Conclude your response with ’X’, X representing the correct option.<br>Question: {input}<br>•<br>These questions assess understanding of high school psychology concepts. End your answer with ’X’, where X is the letter of the correct choice.<br>Question: {input} |
| bbq_religion | •<br>The following are multiple choice questions (with answers) about social bias on religion. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Here are multiple-choice questions addressing social biases related to religion. Conclude your answer with ’X’, X being the correct letter option.<br>Question: {input}<br>•<br>These questions explore social biases in the context of religion. End your response with ’X’, where X represents the right letter choice.<br>Question: {input} |
| deepmind | •<br>The following are multiple choice questions (with answers) about algebraic word problems. Finish your answer with ’X’ where X is the correct letter choice.<br>Question: {input}<br>•<br>Below are multiple-choice questions testing algebraic word problem solving skills. Conclude your answer with ’X’, X being the correct option letter.<br>Question: {input}<br>•<br>These questions assess your ability to solve algebraic word problems. End your response with ’X’, where X is the letter of the right choice.<br>Question: {input} |

## Appendix F Adaptive Elicitation

As mentioned in Section 4.3, we show that when provided the library with only math-related task vectors, performance shows a significant improvement on the math domain while retaining or slightly improving in other domains for Mistral.
Figure 14 illustrates similar results on other models such as Mamba, Pythia, and Llama3.

Figure: (a) Mamba
Refer to caption: extracted/5921119/sources/figures/performance_distribution__mamba_math_.png

## Appendix G Similariy-Based Retrieve Method

Section 5.1 demonstrates the poor precision-recall performance of similarity-based retrieval methods on the Llama3 model.
Figure 15 presents the Precision-Recall curves for Mistral and Mamba under different similarity-based approaches, which also exhibit poor results.
In contrast, our proposed retrieval module achieves significantly higher precision and recall across all models.
This highlights the effectiveness of our method in accurately retrieving relevant task vectors to support different tasks.

Figure: (a)
Refer to caption: extracted/5921119/sources/figures/mistral-7b-base-v0.2_all_layers_cosine_auc_natural.png

## Appendix H Orthogonal Augmentation in Unseen Tasks.

As shown in Section 4.5, ELICIT demonstrates its plug-and-play capability by seamlessly integrating with existing methods such as BM25 Retrieval for in-domain tasks.
Table 9 showcases ELICIT combined with BM25 in unseen tasks, combined with BM25 for unseen tasks, where we can observe performance improvements across various models and tasks.
This highlights the versatility and effectiveness of ELICIT in augmenting existing methods and tasks.

**Table 9: Orthogonal Augmentation in Unseen Tasks.**
| Model |  | GLUE COLA | BBQ Religion | Deepmind | MMLU-Psychology | BBH-five-objects | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama | BM25 | 83.6 ± 0.8 | 54.5 ± 0.7 | 29.1 ± 1.9 | 81.6 ± 0.4 | 29.6 ± 0.0 | 55.7 ± 0.2 |
|  | BM25+ELICIT | 84.7 ± 0.9 | 54.4 ± 0.1 | 30.0 ± 4.3 | 81.2 ± 0.3 | 33.5 ± 0.0 | 56.7 ± 0.6 |
| Mistral | BM25 | 84.6 ± 1.2 | 57.6 ± 0.6 | 19.5 ± 2.2 | 74.7 ± 0.9 | 15.1 ± 0.0 | 50.3 ± 0.5 |
|  | BM25+ELICIT | 85.6 ± 1.4 | 54.3 ± 2.9 | 26.1 ± 2.4 | 74.2 ± 1.0 | 23.8 ± 0.0 | 52.8 ± 0.9 |
| Pythia | BM25 | 54.5 ± 1.0 | 29.7 ± 0.7 | 13.3 ± 2.2 | 14.9 ± 0.2 | 5.7 ± 0.1 | 23.6 ± 0.3 |
|  | BM25+ELICIT | 56.1 ± 0.9 | 33.9 ± 4.0 | 13.9 ± 2.8 | 22.3 ± 1.3 | 16.0 ± 0.0 | 28.4 ± 1.2 |
| Mamba | BM25 | 67.4 ± 0.7 | 30.3 ± 2.7 | 19.4 ± 1.2 | 16.0 ± 0.4 | 14.5 ± 0.1 | 29.5 ± 0.7 |
|  | BM25+ELICIT | 69.3 ± 0.9 | 30.8 ± 1.6 | 23.0 ± 1.4 | 22.8 ± 0.2 | 16.5 ± 0.0 | 32.5 ± 0.1 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How are the ICL prompt $p_i^{t}$ chosen? (e.g., Are they a group of randomly selected examples?) If there is a technique to maximize the diversity of the prompts in a given library, would it also boost the model performance?
2. Figure 5 shows that ELICIT boosts performance on relevant tasks while minimally compromising performance on non-related tasks. Is this because task vectors are not applied for unrelated tasks, or does the model perform well even when unrelated task vectors are given?
3. The current framework augments (at most) a single task vector at a single layer. Could this be extended to multiple vectors or layers to improve performance? (e.g., in cases where a given query is highly relevant to two different task vectors)
4. Tables 2 and 3 show that ELICIT performs significantly better than the ICL baselines for Pythia and Mamba. What would be the reason for this? For example, can ELICIT be more beneficial for models with weaker capabilities? Or could this be related to a specific training recipe, as Pythia and Mamba are trained under the same setup?
soundness: 3
strengths: 1. The writing is clear and easy to follow. The motivation and design of each component is straightforward to understand.
2. Dynamically augmenting task vectors is significantly more efficient than in-context learning while showing competitive or even better performance.
3. The proposed approach can be applied to existing LLMs in a plug-and-play manner, making ELICIT easy to deploy.
summary: This paper proposes ELICIT, a framework that stores the task vectors corresponding to different in-context-learning (ICL) prompts and dynamically augments the given question with retrieved task vectors to provide ICL abilities without explicitly forwarding the long ICL prompts. ELICIT shows comparable or better performance than ICL baselines on diverse tasks while being significantly more efficient.
weaknesses: 1. Some details regarding the experiment setup need to be included. For example, the paper does not describe how the ICL prompts $p_i^{t}$ are chosen.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: See weakness above
soundness: 4
strengths: 1. The paper comes up with an interesting and intuitive solution to improve LLMs' abilities using the task vectors.
2. Extensive experiments over various models and tasks.
3. Exprimental results show great advantage of the method over others.
4. This novel plug-and-play framework could benefit other methods on the same task.
summary: This paper proposes ELICIT. It's a framework aims at improving LLMs capabilities by introducing an external ICL capacity library. This library stores task vectors, which represent in-context learned abilities, enabling models to retrieve relevant skills dynamically without additional training tokens or fine-tuning. The approach allows LLMs to handle diverse tasks by selectively activating specific capabilities when needed, thus improving both versatility and computational efficiency.
weaknesses: 1. I would expect the proposed ELICIT method to be integrated into more existing strategies such as few-shot learning.
2. Have you tried using the capacity bank with other creation method other than ICL?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why isn't the experiment conducted on instruction-tuned models but base models?
2. Did you compute the decrease in inference efficiency caused by the introduction of a new module?
soundness: 3
strengths: 1. The new framework with the use of task vector demonstrates effectiveness in improving zero-shot performance.
2. The paper is generally well written and easy for readers to understand.
summary: The article introduces ELICIT, a novel framework to enhance the adaptive capabilities of large language models (LLMs) without the need for extensive fine-tuning or in-context learning demonstrations. ELICIT consists of two key modules: a capability library that stores task vectors representing various in-context learned capabilities, and a dynamic retrieval module that selectively activates these vectors based on input queries. Experimental results demonstrate the effectiveness of the model.
weaknesses: 1. The novelty is limited in some aspects, including the use of task vector and the retrieval module. 
2. Experiments on different models of different sizes should be conducted as the study would better demonstrate that this method is also effective for large models.
3. More comprehensive experiments on more datasets are expected, such as MMLU, GSM8K, HumanEval, etc.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: While the paper presents an interesting direction within ICL with modular task vectors and shows better performance than zero-shot and classical ICL, further improvement of clarity, related work comparison, and validation dependency could strengthen the paper even more. I would recommend acceptance if the authors address these points and the questions within the weakness section and the following one:
- Have you tried aggregating task vectors by task, and if so, what were the results?
- Have you considered using multiple layers to represent task vectors instead of relying on a single optimal layer? If yes, how did this affect performance?
- Can you please clarify if only the most similar task vector is used for intervention in the end? If so, does this mean many task vectors remain unused?
- Can you include comparisons to related task vector approaches from previous ICL research you mentioned in the related work? Can you also compare your method against PEFT modular methods with a few-shot regime?
- How well does your approach scale to larger LLMs?
- Can ELICIT be extended for multi-modal applications or other, more complex tasks?
- Does ELICIT support compositionality, such that combining different task vectors can represent new tasks?
soundness: 3
strengths: - **S1:** Paper explores a novel and promising direction for in-context learning by leveraging task vectors with an external modular concept which is really interesting and aligns well with the recent work in the field of in-context learning and task-vectors.
- **S2:** Proposed method ELICIT demonstrates strong performance and generalization across diverse tasks and models.
- **S3:** This paper has a clear mathematical presentation with good explanations of ICL and task vectors.
summary: This paper introduces ELICIT, a novel framework for adapting Large Language Models (LLMs) to diverse tasks, using task vectors and in-context learning (ICL). Inspired by the concept of modularization, ELICIT comprises two core components—Build Capability Library and Dynamic Capability Elicitation. By building a library of task vectors, each representing one in-context capability, ELICIT dynamically leverages this library to selectively retrieve and activate capabilities based on any given query, ensuring efficient and flexible elicitation of the model’s capabilities. 

To build the capability library, task vectors are stored across layers for each task, along with prompts for future reuse, and the position of the optimal layer, which is determined based on the hold-out validation set and zero-shot inferences with task vector interventions to the model. Interventions within the model can be as direct replacement or a linear combination, with the latter shown to perform better. Later, task vectors are retrieved with the Dynamic Capability Elicitation module that employs a SimCSE RoBERTa model for relevant task vector selection and a threshold-based filtering approach based on AUC scores from the validation set.

ELICIT shows good performance across 20 ICL tasks and four models, outperforming other baselines across tasks, models and query formats, while also showing good generalization on unseen tasks.
weaknesses: - **W1:** The paper lacks specifics on dataset curation regarding the train-val-test splits and sizes, as well as the sample sizes used to calculate the performance. Can you please clarify how many examples were used for the task calculation? 
- **W2:** ELICIT selects the optimal layer for task vectors for each given task and prompt, but assuming that the whole task encoding is stored just within a single layer may not be optimal. Have the authors tested multi-layer interventions, and if so, how did they compare? Moreover, could grouping and averaging task vectors per task simplify optimization steps and yield efficient task vector retrieval without extra steps?
- **W3:** The paper does not compare with previous task vector approaches mentioned in the related work (Hendel et al., Todd et al., Liu et al., Yan et al.), as well as with other parameter-efficient fine-tuning methods based on modularity (also mentioned in the related work like Hu et al.) with 16 shots. I believe such comparisons should be performed and included in the paper. Additionally, the paper does not mention task vector work for visual ICL and multi-modal ICL [1, 2] and including these would provide a comprehensive context and overview of the field. 
- **W4:** Figure 3 illustrates the trade-off between stronger interventions and language modeling performance on WikiText, which is an expected observation since ICL and general language modeling operate with different circuits [3, 4, 5]. Having stronger interventions steers the activations further from the pretrained task, thus resulting in the worse performance, which is what previous works also showed. Authors do not analyze or explain this observation, but just comment how the strength of interventions affects the ICL and language modeling performances. I believe further discussion and explanation should be included.
- **W5:** The proposed method relies heavily on validation data to select optimal hyperparameters and determine the filtering threshold. And the similarity-based model for task vector retrieval is further trained. Does relying on the validation tuning affects the scalability and efficiency? Can you please explain how this similarity model was trained, and with what data?
- **W6:** Tables 1 and 2 contain additional bolded entries, and captions are not descriptive enough, missing information about the sample size for the BM25 and evaluation in general. Further, figure 6 does not have a clear explanation of components labeled a, b, and c, while also missing a description in general. Finally, there is a typo in the appendix in the title for the Similarity based retrieval methods.

[1] https://arxiv.org/abs/2404.05729

[2] https://arxiv.org/abs/2406.15334

[3] https://arxiv.org/abs/2205.05055

[4] https://arxiv.org/abs/2404.07129

[5] https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html

Meta-review: This paper proposes a new approach that improves LLMs capabilities by introducing an external ICL capacity library.

Three reviewers support the contributions of this paper with clear cceptance scores while one reviewer gave clear rejection.

AC carefully read the paper, reviewers' comments and author feedback.

While R-yopv pointed out major concerns such as limited novelty and insufficient experiments, the authors provide the additional experimental results on those issues. Although yopv does not show his/her feedback, AC thinks that the authors successfully addressed the concerns. 

So, AC recommends accepting this paper.

Decision: Accept (Poster)