In-Context Learning State Vector with Inner and Momentum Optimization
Authors: Dongfang Li
zhenyu liu
Xinshuo Hu
Zetian Sun
Baotian Hu
Min Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: in-context learning
language models
task representations
mechanistic interpretability
Primary area: natural_language_processing

Abstract: Large Language Models (LLMs) have exhibited an impressive ability to perform In-Context Learning (ICL) from only a few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization of these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters trained with gradient descent, and introducing the concept of state vector. Inspired by the works on model soup and momentum-based gradient descent, we propose inner and momentum optimization methods that are applied to refine the state vector progressively as test-time adaptation. Moreover, we simulate state vector aggregation in the multiple example setting, where demonstrations comprising numerous examples are usually too lengthy for regular ICL, and further propose a divide-and-conquer aggregation method to address this challenge. We conduct extensive experiments using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The experimental results show that our optimization method effectively enhances the state vector and achieves the state-of-the-art performance on diverse tasks.

In-Context Learning State Vector with Inner and Momentum Optimization
Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu Min Zhang
Harbin Institute of Technology (Shenzhen), Shenzhen, China
{ (crazyofapple, liuzhenyuhit (egmall.cm com (hubaotian, zhangmin2021 .du cn
Abstract
Large Language Models (LLMs) have exhibited am impressive ability to perform In-Context Learning (ICL) from only few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters trained with gradient descent, and introducing the concept of state vector. Inspired by the works model soup and momentum- based gradient descent, we propose inner and momentum optimization methods that are applied refine the state vector progressively as test-time adaptation. Moreover, we simulate state vector aggregation in the multiple example setting, where demonstrations comprising numerous examples are usually too lengthy for regular ICL. and further propose a divvid-anddcongeee address this challenge. We conduct extensive experiments using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The experimental results show that our optimization method effectively enhances the state vector and achieves the state-of-the-art performance on diverse tasks. Code is available at https: 7/github, com/HITTz TMG/TCI, -CL--tatee
aggregation method
Introduction
In-Context Learning (ICL) has emerged powerful capability in tandem with the scaling of large language models (LLMs) [rownn et al. 2020 By simply conditioning as demonstrations, LLMs yield significant improvement and deliver remarkable results in various downstream Natural Language Processing (NLP) tasks (Wei et al.) 2022 Liu et al., 2023a]. For example, model prompted with the input "gaor + goat, sakne + snake, brid + can produce the output "bird". Given these successes, it is worthwhile to inquire about the exact internal working mechanisms of ICL. Considering the opaque operation of ICL. within the ruto-regressive transformer, it is plausible that ICL might function as a general mechanism that leverages both demonstrations and the query to yield the prediction [Dong et al. 2023].
few input-label pairs
Recently, some studies have found that the ICL mapping function exists in the outputs of the attention layers or attention heads [Liu al. 2023b Dai al. 2023 when applying causal effects analysis on a different set of models and tasks, such as the task vector [Hendel et al. 2023) and the function vector (Todd al. 20233. These works show that the functionalities learned through ICL can be encapsulated in compressed vectors derived from transformers, which then can be used to intervene in the transformer to handle queries without demonstrations. This revelation suggests the potential mechanism of ICL that first utilises demonstrations to learn the mapping function from inputs to
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

labels in shallow transformer layers, and then uses the ICL function in deeper transformer layers to make predictions (Hendel et al. 2023]. However, while these compressed vectors encapsulate learned information in a more condensed form and show significant promise in applying ICI., there still exists a considerable gap in understanding the operational mechanisms and optimization strategies of these vectors. This significant gap hinders the further grasping and utilization of ICI.
In this paper, we aim bridge the existing gap by presenting a comprehensive analysis of compressed vectors. Specifically, we investigate their similarities with parameters trained via gradient descent and introduce the formulation of state vector that encapsulates the processing state of ICL stored in the attention activations. Building on the concept of state vector, and drawing insights from the model soup [Worsmman al., [2022] and Polyak momentum- based gradient optimization algorithms [Qian] 1999. Sutskever et al. 2013], we propose inner optimization and momentum optimization strategies which are progressively applied to enhance the state vector. Moreover, we further exploit the demon- stration compression capabilities of the state vector to address the practical challenges encountered when applying ICL settings with multiple examples, where demonstrations are typically too lengthy for standard ICL, such as in the 100-shot setting which is prevalent in practice. Specifically, we introduce a divide-and-conquer aggregation method that effectively aggregates the ICL functions of these extensive examples. This approach enables us to scale up for processing extended examples by compressing them into a single state vector. We conduct extensive experiments using Llama- 2 [Touvron et al., (2033 and GPT-J [Wang and Komatsuzaki, 2021] in both zero-shot and few-shot settings. The experimental results show that our method effectively enhances the state vector and achieves state-of-the of-t----rr-r-tttt performance on diverse tasks. This not only manifests the effectiveness of our approach but also paves the way for a more comprehensive understanding of ICI..
Our contributions are summarized as follows:
We delve into the working mechanism of compressed vectors in ICL and highlight their similarities with parameters trained via gradient descent. Building on this observation, we propose the formulation of the state vector.
We propose inner and momentum optimization to progressively refine the state vector as an efficient test-time adaptation. Additionally, we introduce a divide-and-conquer aggregation to effectively scale up to large numbers of examples.
We show the practicality of our proposed methods across wide range of tasks through extensive experiments. Our results also offer insights for future research aiming to fully understand the functionalities of ICL.
2 Related Work
Mechanistic Interpretability. Recent works have focused the working mechanisms of ICL [Chan et al. 2022, Xie et al. 2022 Wang et al. 2023 Olsson et al. [2022] argue that in- duction heads may be the mechanistic source of general ICL in transformers, Akyürek et al. [2022]
show that transformer- based in-context learners can implicitly implement standard optimization algorithms on linear models. A mainstream assumption posits that ICI. has similarity with the gradient descent. von Oswald et al. [2023] demonstrate how a linear attention-only transformer model can perform gradient descent- procedure implicitly. Dai et al. (2023) compare standard gradient descent based fine-tuning and ICL, and figure out that the transformer attention of ICL exhibits dual form of gradient descent-based optimization. Moreover, some works revisit and modify this theory on the layer causality dependence [Natan et al., [223) or training batch size (Shen et al.) 2023). In contrast, we focus on the application of the dual form of gradient descent and ICL and present optimization methods with inspiration from the dual form.
Task Representation. Numerous studies have extensively explored the concept of compressing various tasks into task representations as means of effectively manipulating tasks within ICL
ability. Notably, [Shao al. [2023] and Mu al. (023) have successfully yielded compositional task representations by trainig composition model. In a slightly different vein, some researchers have delved into the art of devising methodologies to compose minor parameter adjustments acquired through task fine-tuning [ihhacco et al. 2022 Panigrahi et al. 2023 Yu et al. 2023 Hu et al. 2024
Merullo et al. 2023) An alternative line of research finds that the task representation could be extracted in ICL Liu et al. 2023b Hendel et al. 2023 Todd al. 2023
Yang et al. 2023 Different

---

from these approaches, our work avoids the need for additional training and focuses more on analysing why these compressed vectors work and how improve their performance.
Formalization
In this section, we first provide a detailed examination of attention activation which is found to
contain the compressed ICL function by previous works [Hendel et al., 2023 Todd et al. 2023)
Then, we highlight its inherent similarities with parameters trained through gradient descent. Finally,
we introduce the concept of the state vector drawing inspiration from these observations.
classic template of ICL has the following necessary components: (1) N examples that are used to form the demonstrations and each example contains an input query X' and its corresponding label y (2) Separate tokens S that separate the input query and the label for each example (e.g., -). (3) A query Xg for prediction. With the above components, the contextual model input of ICL could be written as follows:
&, S, , S.N. Ry.
Here we analyse the attention activation of the last separate token. In the I-th transformer layer, the output activation a' of the attention heads of the last separate token is:
a' = Wv X] softmax (Wx[X'xx)' Vd
(I)
where X" denotes the hidden state of demonstrations, X denotes the hidden state of the query and the last separate token (called zero-shot input). q denotes the attention query vector of the last separate token, [X'; X denotes the matrix concatenation, vd is the scaling factor, WK and Wy are parameter weight matrix.
Consistent with previous works DDa et al. 2023 Natan et al. 2023], we omit the softmax operation and the scaling factor to approximate standard attention relaxed linear attention for qualitative analysis. Consequently, the activation can be simplified as follows:
a' = Wv(X'; X X] (Wx[X';
= =(WvX (W&X)" + WvX' (WxX'*")
(2)
WZSt + Σ ((Wx)) & (Wxx))) q.
We define Wzs1. Wv X XW* as the initialized parameters since it is the attention result in the Zero-Shot Learning (ZSL.) setting.
To draw meaningful comparison between attention activation and parameters trained through gradient descent, we now shift our focus towards analyzing a simple linear transformation represented by y, Wx,, Given loss function C and the learning rate 7), the gradient of linear weight is:
C(y.) aCly:) θy:
Qy, OW
Denoting the pack-propagated errors as e; = V C. we can get the full batch gradient with training examples:
AWGD Σ e, x
where x is the input training examples. Hence, in the previous Eqn. 2 we substitute Wxxx as
training examples. and take Wvx, x e; corresponding to some meta gradients (Dai et al. 2023 Natan et 2023). The activation can be written as:
a' = W2SL+ + Ee, Ce.oWxx) q= (WZsL + AWGD) q. -(W2LL
(5)
Hence, can be inferred that the output activation a' can be regarded as parameters trained via gradient descent which utilizes the demonstrations training instances.
3

---

With the above dual form between activation and trained parameters, and in light of observations that transformers tend learn the ICL function primarily their first layers [Wang al. 2023]. we have the following hypothesis: During the process of ICI., the first I. layers progressively update the flow of information using each example the demonstration through forward computation. The processing state of ICL is then stored within the activation of the attention head. The subsequent layers access and utilize the processing state to reinstate the ICL function, which is used implicitly for predicting the queries. Therefore we concatenate the activation in the initial L layers and introduce the notation of the state vector:
V
(6)
where L is the number of layers and N the number examples the demonstration. || denotes
concatenation operation. Note that have completely different construction strategy and usage compared to the function vector [odd et al. 2023]. Although the task vector [Hendel et al. [2023]
may be functionally equivalent in the forward process, the proposed state vector differs sigmificantly in its integration into the model, making it easier and more effective to analyse and interpret.
Method
Extraction
Progressively Optimization
Intervention
Transfomerr Layer
Transformer Layer
Feed forward layer
Momentum
Feed forward layer
Inner Optimization
Optimization
Divide and Conquer Aggregation
Illebel
Divide Stage
Conquer Stage
layer
layer
10akea
Pistaction
Plersction
Fatraction
10whhe
state vector
First layers
RESICERAD SCSEED
First looress
Figure 1: The overall framework of the proposed state vector. The state vectors are extracted from the output activations of attention heads. These state vectors progressively optimized by inner optimization and momentum optimization, be aggregated through divide-and-conquer (D&C) aggregation. Finally, the processed state vector utilized to intervene the inference forward pass.
4.1 Overview
As illustrated in Figure I our approach initially extracts the state vector from the attention head that corresponds to the final separate token in the first L layers using demonstration and a dummy query. Then, with the view of treating the state vector as trained parameters, coupled with drawing inspiration from the model soup and the momentun-based gradient optimization algorithm, we introduce two methods that progressively optimize the state vector as test-time adaptation [Liang et al. 2023]: (1) inner optimization ($4.2) and (2) momentum optimization ($4.3). Moreover, we propose a divide-and-conquee
(D&C) state vector aggregation method for efficiently compressing the ICL function in the multiple example setting ($4.4).
After the state vector optimization or aggregation, we utilize the processed state vector intervene the model during the forward inference pass. In particular, we first input test query in the zero-shot setting or with the demonstration in the few-shot setting. During the forward pass in the first layers, we replace the attention activation of the last separate token with the corresponding activation in state vector. other words, the state vector is leveraged to intervene the output of the first I transformer layers, blocking the attention of the last separate token to the previous context. With state vector intervention, the transformer learns the ICI. function from the processing state stored in the state vector, and continues to make the prediction on the test query.

---

4.2 Inner Optimization
Inspired by the works on the model soup [Wortsman et al., 2022 Chronopoulou al. 2023] which show that weight-space averaging not only yields performance improvement but also often enhances robustness, we thus ask the following research question (RQ1): vector using the model soup approach? To explore this question, we propose an inner optimization method to improve the effectiveness and robustness of state vector. Specifically, we not only extract the state vector in each separate token of the dummy query but also extract the state vector from each example. Formally, with a forward pass in an N shot ICL setting, we extract the N state vector V (1 < i S N) from last N separate token. Subsequently, we apply uniform averaging process to these state vectors as follows:
possible to optimize our state
V
VIC
where V is the inner optimized state vector, which can be directly utilized for inference intervention serves as the initial state vector for later momentum optimization.
4.3 Momentum Optimization
Since we view the state vector as parameters trained gradually through demonstration examples, the difference between two state vectors with adjacent corresponding separate tokens can also be regarded as the influence of the middle example, akin to the gradient. Motivated by this understanding, coupled with extensive studies of the gradient optimization algorithm (Sutskever et al. 2013 Duchi et al. 2010 Loshchilov and Hutter 2019], we direct our focus toward a simple momentum- based gradient optimization algorithm, seeking to answer the following research question (RQ2): Can our state vector be optimized using momentun- -ased optimization algorithm? To answer this question, we propose a momentum optimization. Formally, we first extract the influence of each example by subtracting two adjacent state vectors:
E = V V--1
(8)
where ET is the influence of i-th (1 < S N) example in the early L layer. Then, we apply the momentum gradient optimization algorithm to obtain optimized influence E/", and add it to the last state vector:
b - - V'x + opt [B)))
(9)
where Di is the momentum optimized state vector and V is the inner optimized state vector. opt()) (-) denotes the momentum gradient optimization algorithm. We also explore various other gradient optimization algorithms in 1
4.4 Divide-and-Conquee Aggregation
In addition to optimizing the state vector to more effectively represent the ICL function from a small number of examples, also explore its capacity encapsulate multiple examples within a single vector. However, regular ICL can not be directly used on multiple examples due to the context length limitation of current LLMs. This leads us investigate the following question (RQ3): Can we use the state vector to represení multiple examples that are unmanageable for regular ICL? To address this question, we propose a divide-anddconquer method for state vector aggregation. As depicted in Figure D our approach involves distinct aggregation processes (i.e. the divide stage and the conquer stage). In the divide stage, examples are randomly divided into groups, termed grouped demonstrations. Within each group, a random example is selected to serve as a dummy query, which allows us extract a group-specific state vector. In the conquer stage, these dummy queries are paired with their corresponding labels to form input-label pairs. From these input-label pairs, we form an aggregated demonstration, add an additional dummy query, and subsequently extract the aggregated state vector. It is worth noting that during the forward pass of aggregated state vector extraction, we utilise the group-specific state vector to intervene the attention activation of the separate tokens of their corresponding examples. The divide and conquer approach allows us to aggregate the ICI. function of each grouped demonstration into its respective group-specific state vector, and subsequently aggregate the ICL function of each group-specifi state vector into single,
5

---

comprehensive aggregated state vector. This aggregated vector is then utilized for interventions during inference, similarly the optimized state vector discussed in $4.2 and $4.3. Moreover, in the few-shot setting, the aggregated demonstrations are treated as inference demonstrations. The divide-and-conquee approach effectively circumvents the context-length constraints inherent in L.LMs, thereby enabling more effective and efficient aggregation of information across multiple examples.
5 Experiment
5.1 Setup
We conduct the evaluation across 12 datasets that encompass different domains.
Linguisties includes Antonym Nguyen et al. 2017]. Capitalize, Present-Past, and Singular-
Plural (Todd et al. 2023). focusing transformations in the form meaning of words.
Translation is represented by the English-Frende French [Lmplee et al., 2018 dataset, which involves translating English words into their French counterparts.
Knowledge comprises Country-Capital [Todd al. [202]], AG News (Zhang et al. 2013). Person-Sport, Person-Instrument, Person- Person-Occupation, CoopuuttCompany.. and Landmark- Country (Hernande al. 2023 which are centred around question-to-answer mappings
for commonsense knowledge queries.
We employ Llama-2-7B and GPT-J-6B
L.MMs, chosen for their moderate model sizes, open- source and capability for ICL. We also provide the results with larger models (i.c., Llama-2-13BB in the Appendix H We use Llama-2-7B as the default model unless otherwise specified. Our method is orthogonal to the choice of transformer-bat
decoder-only autoregressive LLMs.
restrict to single-token output and use first output token accuracy the evaluation metric as in previous work [Hendel et al. 2002 Todd et al. 2023
For simplicity evaluation,
5.2 Baseline
In the paper, we compare with the following methods:
Regular the baseline for the zero-shot setting that uses only the given query as input, while ICL, baseline [Wei et al.) [2022] makes predictions on the label by taking both the
demonstrations and the given query.
Function vector [Todd al. 2023 extracted from attention activation using the causal mediation method and is then added to the hidden state of certain transformer layers during inference.
Task vector (Hendel al. 2023 is extracted from the hidden state of the separate token and is leveraged for blocking the layer when inference.
5.3 Inner (ptinization(RQ1)
As shown in Table the performance of our inner optimized state vector has significant improvement comparing the task vector and function vector in both zero-shot and few-shot settings. Our state vector with inner optimization. In the zero-shot setting, the inner optimization shows average improvement of 10.2% on Llama-2 and 5.9% on GPT-J across six datasets. In the few-shot setting, the inner optimization also achieves 1.2% improvement on Llama-2 and 1.7% on GPT-J. The improvement demonstrates the effectiveness of inner optimization. However, although state vector (inn.) outperforms task vector, its few-shot performance on some datasets inferior to the ICL baseline. We attribute this primarily to the introduction of query information from examples. While inner optimization enhances task-relevant information for the state vector, also introduces noise of other dummy queries, hindering the model's ability to focus on the current predictive query, thereby reducing performance. In addition the performance improvements, our inner optimization approach also effectively alleviates the phenomenon of high variance in the original task vector in the zero-shot setting. In practical use, the performance of the task vector influenced by demonstrations and dummy queries, leading to weaker robustness. Our proposed inner optimization approach effectively

---

Model
Method
Anym
Eng-Pr
Pers-Iust
Pers-Oce
Prod-Comp
Land-Cout
Average
Regular
1.0=
0.120.1
0.00000
0.020.0
0.4002
0.0200
0.3
Function vector
45.1420
21.6120
11.31007
0.1
25.61443
32.92216
22.8
Zero-shot
Tack vector
5622228
63.2436
61.8244
27.9
25.5
27.8
53.7
State (inn.)
61.0+10
66.5432
674416
42.7444
14.52106
81.0=117
Llama 2-7B
State (mom..
67.5411
68.7416
45.6455
71.34-36
77.74118
ICL buseline
64.8248
74.3408
71.7437
56.1227
80.820.88
87.0=003
72.5
Function vector
54.5209
65.2
60.84:5
54.2422
76.02113
84.2
65.8
Row-shot
Task vector
65.7+18
73.8+00
66.64:12
56.44-23
81.94-188
86.7+0.9
71.8
State veclor (inn.)
66.22116
74.6:09
70.14443
57.0+22
82.81:16
87.5
73.0
State (mom.
65.8177
74.3+1.1
74.929
58.2404
82.01:10
87.6
Regular
8.1
0.2
0.0:20.0
0.0:0..0
19400
0.9:::0.
3.0
Function vector
33.12118
29. 1185
4.13388
11.1233
46.34577
22.52100.2
24.4
Zero-sbot
Task
23.6.3388
32.2:15.1
44.41500
18.311866
43.8.:50
11.3
35.6
State vector (inn.)
13
31.74388
49.3:42.0
30,01162
42.84:4.3
41.5
State vector (mom.)
31.1
35.1
50.34:10
42.41:55
44.2
60.3=009
43.9
GPT-J-6B
ICL bascline
59.22114
69.9120
44.71667
29.3110
0251110
09,3
55.8
Function vector
16.421.9
65.8
49.1222
30.3#199
58.5
69.2
54.9
Hew-shot
vector
58.521.6
70.62122
42.3464
27.8433
66.012.6
63.125.3
State
58.7:22
70.91113
46.51499
29.41177
66.31:2.11
66.41228
56.4
State (mom..)
19,6
70.1
51.924
30.4+1.1
63.8+0.88
68.62003
57.4
Table I: Performance of state vector optimization. The best results in the zero shot setting are in underline and the best results in the few shot setting are in bold. The result of basic state vector is mathematically equivalent to task vector. Note that we only present the results across six tasks here and leave the rest in the Appendix. We also report standard deviation the results are passed with significance test (p .05).
60
70
50
500
2.50
NEE
Nigaar
20
Zemoher Ang
20
Zenathe Aug
20
Zesanc
20
Zenanee Auy
Zexoric DMC
Zeniiel DeC.
12
02
20:00
60 50300
40
50
Nummbrr evamppe
Number exampee
Kumber ecomple
Numbhrr example
(5)
Antonym
Llama-2-77 Instrument
(c) Llama 2.70 English French
(d)
60
00
70
70-
70
70
60
60
50
60
50
2.50
2.50
40
30
Auguir
Aapu
30
repuir
Jareene Ag
20-
DSC
Fun
35 92 Numbor oxampic
St
53 Number sxampli
Number exampl:
Number champl::
(c) GPT-J-6B Antonym
( GPT-J-6B Person-Instrument
(g) GPT-J-6B English-French
(h) GPT-J-6B AG News
Figure 2: Performance of aggregation across number of examples. Arg. denotes the average aggregation baseline and D&C. denotes the divide-ane -nd---uue- aggregation. The axis represents the number of examples, and the axis represents the accuracy.
mitigates this issue, similarly motivated as the model averaging method, thereby enhancing the robustness of the state vector.
5.4 Momentum Optimization (RQ2)
As depicted in Table D building upon the inner optimized state vector, our proposed momentum optimization algorithm further enhances the effectiveness of the state vector, achieving the best performance on average in all settings. In the zero-shot setting, the momentum optimization boosts the performance of the inner-optimized state vector with an average increase of 1.3% on Llama-2 and 2.4% on GPT-J. In the few-shot setting, state vector with momentum optimization achieves 0.8% average increase on llama-2 and 1.0% on GPT-J. This reveals the effectiveness of our momentum optimization. With the combination of inner optimization and momentum optimization, state
7

---

100
90
Task Vector
Method
Zero-shot
Few-shot
80
State Vector (inn.)
State Vector (mom.)
70
ICL baseline
0.2+
71.0+
60
Task vector
52.9 94
68.5+ 10.5
Accurac)
50
State vector (mom.)
65. + 10.2
72.2+
40
State vector (adag.)
11.7a 12.0
16. 1 10.2
30
State vector (rms.)
0.8 0.9
1.5
20
State vector (adam.)
6.7
10.6
10
Table 2:
Performance comparison of gradient opti- mization algorithms. The method means the optimiza- tion algorithm applied to the opt (-) in Eqn.
0
2 4
10
12
14 16
28
22
24 26
28 30
Layers
Figure 3: Average zero-shot performance across six datasets for each choice the intermediate layer L. The solid line means the average value, while the shaded area indicates the standard deviation.
vector (mom.) surpasses the original variant, showcasing a remarkable improvement of 11.5% for 1lama-2 and 8.3% for GPT-J in the zero-shot setting. In the few-shot setting, our state vector (mom.) still outperforms the task vector with 2.0% improvement for Llama-2 and 2.7% for GPT-J. Furthermore, without inputting demonstration during inference, the state vector (mom.) achieves an impressive 90% ICL performance on Llama-2 and 78% ICL performance on GPT-J. When compared to ICL with the same examples as the demonstration, state vector (mom.) outperforms ICL in both Llama-2 and GPT-J. These improvements verify the effectiveness of our progressive optimization strategy. Note that applying momentum optimization directly task vectors does not yield average improvements across tasks in our preliminary experiment. We speculate that this inconsistency stems from the poor robustness of the task vectors, which hinders the stable optimization by momentum optimization and leads to poor performance in some tasks.
5.5 Divide-and-Conquee Aggregation (RQ3)
In this experiment, we explore the performance of D&C state vector aggregation across varying numbers of examples. Besides the regular and ICL baseline mentioned, introduce average aggregation as a strong baseline. This approach first extracts state vectors from the example group and subsequently employs their mathematical average for aggregation. We compare our D&C aggregation method with the baseline ranging from 10 to 100 examples across two models. Due to limited computational resources, we
able do an exhaustive search over all datasets. Thus, we only present the results for four tasks.
As illustrated in the Figure both the D&C aggregation and average aggregation exhibit similar trends in both few-shot and zero-shot settings. The performance of both aggregation methods initially falls short of the ICL bascline. However, their performance boosts when examples increase. The initial poor performance can be attributed to the limited number state vectors. Additionally, although the performance of the D&C aggregation initially falls behind that of the average aggregation, it exhibits a more substantial performance improvement when examples increase, ultimately outperforming average aggregation in the multiple example setting, highlighting the efficiency of D&C aggregation.
6
Analysis
6.1
Ablation with Other Optimization Methods
We present an ablation study to investigate various classical gradient optimization algorithms, aiming to delve deeper into the inner state vector optimization. We compare the momentum- based gradient optimization algorithm with following additional first-order gradient optimization algorithms: Ada- grad (adag-) (Duchi et al. 2010), RMSprop (rms.) [Gavees 2013) and Adam(adam.) (Kingma and
Ba 2015
8

---

1.0
1.0
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
100
0.2
0.4
0.6
0.8
1.0
Antonym
(b) English-French
Product-Company
Figure 4: The 2D PCA visualization of the state vector in the Antonym, English- and Product-Comppntt tasks, where each color represents the state vector corresponding to examples occupying specific positions in the demonstration, with the position of each point indicated by the adjacent number.
As shown in Table 2 we observe a significant decrease in state vector performance when using first- order gradient optimization algorithms, in contrast to the more stable results obtained with momentum- based optimization. This discrepancy suggests that current first-order gradient optimization algorithms may not be well-suited for state vector optimization. We believe there are two main reasons for this. First, first-order gradient optimizers typically rely on adaptive learning rates, which depend on significant amount of historical information. This can lead to instability and reduced effectiveness, especially when the available data is limited. Second, first-order gradient optimizers involve more complex dyper-parameter identify optimal settings. Our experimental results confirm that directly applying hyper-paranee configurations commonly used in gradient descent leads to suboptimal performance in state vector optimization.
compared to the Momentum Optimizer, making it more difficult to
6.2 Layer Selection
We investigate the impact of layer selection the extraction of state vectors in transformer models. We evaluate the average performance across different datasets in the zero-shot setting, as illustrated in Figure 3 Our results reveal dual-phase trend: initially, increasing the number of layers for state vector extraction improves performance, but this improvement reverses beyond the 14th layer. We correlate this with the dynamics of ICL function processing in transformers in line with previous works [Voita et al. 2019 Wang et al. 2023). In the initial layers, transformers are primarily engaged in learning and incapsulating the ICI. function within state vector, where additional layers enhance the richness of the functional information in the state vector. In contrast, the later layers prioritize applying this learned information for prediction tasks. Here, additional layers tend introduce noise, especially from predicted labels of dummy queries, which may negatively impact performance.
6.3 Qualitative Study
We provide the visualization by Principal Component Analysis (PCA) of the original state vector in the Antonym, English-French and Product- Canduct-Compnyy task. As depicted Figure 4 we have three observations: (1) State vectors corresponding the examples occupying the same position tend to form distinct clusters. This clustering pattern suggests a high degree of similarity among state vectors within each example position, despite different contexts. (2) A notable separation is evident between the state vectors originating from the first example and other position examples. This demarcation implies that ICL may begin to effectively function with few examples. (3) interesting trend is observable in the movement of these clusters as the example position increases. This trend may be indicative of an accumulation of task-specific information, where each additional example contributes to more nuanced understanding of the model. These findings suggest a progressive enhancement in the ability of model to internalize and reflect the subtleties of the task at hand. Moreover, these observations reflect the efficacy of momentum optimization to leverage the observed clustering trend.

---

7 Conclusion
In this paper, we reveal that ICL compressed vector can be viewed as parameters trained through gradient descent on the demonstrations. Then, we introduce the concept of state vector coupled with two optimization methods to enhance the capability of ICL and conduct comprehensive experiments across two popular LLMs and multiple tasks to support our claim. Furthermore, our approach demonstrates the ability to compress context while maintaining lower variance. In the future, we aim to extend our methods to more complex ICL scenarios and apply them to larger LLMs and call for more nuanced and realistic studies of ICL.
Acknowledgements
This work is jointly supported by grants: National Natural Science Foundation of China (No. 62376067), National Natural Science Foundation of China (No. 62406088) and Guangdong Basic and Applied Basic Research Foundation (2023A1515100075)
References
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm in-context learning? investigations with linear models. ArXiv preprint, abs/2211.15661, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33; Annual Conference Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
Stephanic Chan, Adam Santoro, Andrew K. Lampinen, Jane Wang, Aaditya Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, 2022.
Alexandra Chronopoulou, Matthew Peters, Alexander Fraser, and Jesse Dodge. AdapterSoup: Weight averaging to improve generalization of pretrained language models. In Findings of the Association for Computational Linguistics: EACL. 2023, pages 2054-2063, 2023.
Damai Dai, Yutao Sun, Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent meta-optimizers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4005-4019, 2023.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiee exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. ArXiv preprint, abs/2301.0033 .00234, 2023.
John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. In COLT 2010 The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010. pages 257-269, 2010.
Alex Graves. Generating sequences with recurrent neural networks. ArXiv, abs/1308.0855 2013.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. ArXiv preprint, 1tt/22310.11911 2023.
10

---

Evan Hermandez, Arnb Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. ArXiv preprint, abs/2308.09 09124, 2023.
Xinshuo Hu, Dongfang Li, Zihao Zheng, Zhenyu Liu, Baotian Hu, and Min Zhang. Separate the wheat from the chaff': Model deficiency unlearning via parameter-efficien module operation. In Proc. of AAAI, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ArXiv preprint, abs/2212.04089, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Machine Learning, 2015.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSPP 2023, Koblenz, Germany, October 23-26, 2023, pages 61 1-26... ACM, 2023. doi: 10.1 10.1145/ 1600006.3313115.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference Machine Learning, 2018.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribu- tion shifts. ArXiv preprint, 0bbs/2333.1361.. 2023.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023a.
Sheng Liu, Lei Xing, and James Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering, ArXiv preprint, abs/231 012311106666. 2023b.
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Machine Learning, 2019.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec- style vector arithmetic. ArXiv preprint, abs/2305.16130, 2023.
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. Learning to compress prompts with gist tokens. ArXiv preprint, abs/2304. h0d/2334.08467,, 2023.
Tomer Bar Natan, Gilad Deutch, Nadav Magar, and Guy Dar. In-context learning and gradient descent revisited. ArXiv preprint, abs/2311.00777) 2023.
Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Distinguishing antonyms and synonyms in pattern-based neural network. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 76-85,22 2017.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. ArXiv preprint, abs/2209.1 1895, 2022.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, 2023.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1):
145-151, 1999.
11

---

Nan Shao, Zefan Cai, Hanwei Xu, Chonghua Liao, Yanan Zheng, and Zhilin Yang. Composi- tional task representations for large language models. In International Conference on Learning Representations, 2023.
Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? ArXiv preprint, abs/2310.08540, 2023.
Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In Proc. of ICML, volume 28 of JMLR Workshop and Conference Proceedings, pages 11 1139-118 147, 2013.
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. ArXiv preprint, abs/2310.15215 2023.
Hugo Touvron, Louis Martin, and Kevin R. Stone et al. Ilaaa 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307. 009288., 2023.
Elena Voita, Rico Sennrich, and Ivan Titov. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proc. of EMNLP, pages 4396-4406, 2019.
Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. jax 2021.
com
Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. Label words are anchors: An information flow perspective for understanding in-context learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9840-9855, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffll, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 23965-23999 2022.
Sang Michael Xie, Aditi Raghunathan. Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Machine Learning, 2022.
Jiaxi Yang, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Iterative forward tuning boosts in-context learning in language models. ArXiv preprint, abs/2305.13014 2023.
Le Yu, Yu Bowen, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. ArXiv preprint, abs/2311.03099, 2023.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657, 2015.
12

---

A Implementation Details
In this paper, we random sampling to create subsets for cach dataset. Each subset consists of 10 instances for demonstrations and one instance for a dummy query since we employ a 10-shot as the default ICL setting. The remaining instances split into test and development sets with 7:3 ratio. For experiments with multiple examples, we sample 100 instances instead of 10. We use "->" as the separate token similar previous works. We tried other tokens but significant difference. All the experiments are reported over 5 random seeds. The inference mechanism with state vector we describe in $4.1 has key hyper-parameter (i.e.the layer L). Previous studies (Hendel et al. 2023) have shown that the choice of L has an influence on performance. We find the best layer for different tasks via the accuracy of the development set. For the inner optimization in $4.2 we choose the last seven state vectors to optimize. This is because the early state vectors yield subpar performance, primarily due to limitations in the available examples. For the momentum optimization, we choose 0.5 as the retention rate for historical momentum from the options of 0.25, 0.5 and 0.75. We run all the experiments on single NVIDIA A100 80G GPUs. Each of our experiments consumes between 10 minutes to 8 hours of GPU time, depending on the dataset.
B More Details about Baseline
In this section, we present an in-depth and comprehensive analysis of two baselines (i.e. task vector [Hendel et al. 2023) and function vector [Todd et al. [023]). Furthermore, we offer a more nuanced comparison with proposed state vector, highlighting the distinct differences and advantages of our approach.
The task vector is designed to extract the ICL function from a specific layer's ' hidden state within the transformer model. This achieved by directly replacing the corresponding hidden state during inference for intervention. On the other hand, Todd et al.) [2023) first extracts the ICL function from the output activations across all attention heads all transformer layers. These activations are then prioritized based on their causal effect, quantified by the variance in the model's 's output space with or without individual activation interventions. The mathematical average of the top 10 causal effect activations is the function vector, which is subsequently added to the hidden state of a specific layer during the inference stage.
In contrast to these methods, our approach for state vector extraction focuses on procuring the ICL procession state from the output activations of the attention heads within the first L layers. During inference, we replace the corresponding activations with optimized ones. While functionally equivalent the forward process of the task vector when disregarding state vector optimization (i.e.. the vanilla state vector), our approach offers enhanced mechanical explainability. This is attributable to its motivation from the dual form of in-context learning and gradient descent, as explicated in previous work (Dai et al. 2023 Natan et al. 2023). Furthermore, inspired by the dual form, we focus on the further optimization process. On the other hand, unlike the function vector which extracts activations based on the causal effects resulting from individual interventions, our method is rooted in the underlying mechanisms of ICL. This strategy not only improves mechanical explainability but also demonstrates greater performance evidenced by extensive experiments. Experiments also show notably poor performance of the function vector on certain knowledge-baid based datasets, such as Person Prrson-Occupation.
C More Details about Datasets
Here, we describe in detail the tasks that we use to evaluate the state vectors.
Antonym Nguyen et al. 2017 contains 2398 word pairs that are antonyms of each other (e.g. "massive" - "tiny" "). We apply the dataset processed version from the function They filter the word pairs where both words can be tokenized as a
vector [Todd et al. 20033
single token.
Capitalize [Todd et al.
contains 813 word pairs that capitalize the first letter of the given input word (e.g. "plan" - "Plan").
2023
Present-Past [Todd et al. 2023 contains 293 word pairs, where simple past tense verbs are output when given simple present tense verbs (c.g. "adapt" - "adapted")..
13

---

Regular
3.5
Zero-shot Zero shot mom. inn. Few-shet mom.
2.78
3.0-
2.0
1.5
0.99
0.96
0.5
0.0
Liama-2-78
GPT/-68
Figure 5: Time efficiency analysis Llama-2-7 and GPT-J-6B. Inn denotes our state vector with inner optimization. Mom denotes our state vector with momentum optimization
Singular-Plural [Todd et al.
2023
contains 205 word pairs, where the plural form of "wallets").
given singular word (e.g.. "wallet"
contains 4698 pairs of words, which consists of word in English and its translation into French (e.g.. "circle" "cercle"). We apply the processed version from the function vector[ et al. 2023].
English-French [Lample et al. 2018
Country-Capital [Todd al. 2023 contains 197 instances, which output the of the capital city the given country (e.g. "Luanda" + "Angola").
AG News Zhang al. 2015] contains 7600 instances. Each instance contains the news headlines and the first few sentences of an article as input, and output corresponding labels include Business, Science, Sports, and World.
Person-Sport [Hernandez et al. 2023 contains 318 instances. Each instance contains the name of a professional athlete and the sport that they play (e.g. "Hank Aaron" "basketball"".
contains 510 instances. Each instance contains the name of a professional musician and the instrument they play (e.g. "Tom Fletcher" "guitar").
Person- Instrument [Hernandez et al. 2023
Person-Ocaun -ecupaiOnn [Hernandez al., 2023 contains 821 instances. Each instance contains the name of well-known individual and their occupation (e.g. "Tom Fletcher" + "guitar").
Product-Company [Hernandez al. 2023 contains 522 instances. Each instance contains the name of a commercial product and the company that sells the product (e.g. ''Tom Fletcher" "guitar").
Landmark-Counte [Hernandez et al. 2023 contains 836 instances. Each instance con- tains the name of landmark and the country in which it is located.
D Efficiency Analysis
In this section, we present an efficiency analysis of two proposed optimization methods. We evaluate the average inference time using 1000 test data on single NVIDIA A100 (80G) GPU, covering six main datasets and 10 random seeds per dataset. The results are illustrated Figure 3 In the zero-shot setting, compress the ICL function into the state vector which eliminates the need concatenate demonstrations during inference. As shown in the Figure 5] the proposed inner optimization and momentum optimization, which, while tripling the inference speed, achieve 89% of the regular ICL performance on Llama-2-7B and 78% on GPT-J-6B (see Table in the paper). In the few-shot setting, the proposed inner optimization and momentum optimization achieve better results than standard ICL at the cost of minimal loss in inference speed (e.g., 99% and 96%). Moreover, our method orthogonal to attention speedup techniques, such as flash attention [Dao et al. 2022] and page attention won et al. 2023]. Therefore, our approach can also benefit from the achievements of these works and achieve further efficiency improvement. We leave the exploration of alternative enhancement future work.
14

---

Prompt
Llama-2-7B
+SV
What instrument did X play?
8.7
67.3+
Can you tell me which musical instrument was played by X? What was the primary instrument of X in their music career?
69.0+44
17.3
70.31
Table 3: Text portahility of momentum optimized state vector. The templates are provided with "X" replaced by query word. "+SV" denotes adding momentum optimized state vector
English-French
What is the meaning of biography? Llama-2-7B
Prompt
A written account of someone's 's life. It is biographie.
state vector
Antonym
Prompt
When think of upright, I think of think of person who is standing up for what they believe in.
Llama-2-7B
state vector
I think of down.
Table Natural prompt cases with momentum optimized state vector on Antonym task and English-French task.
E Natural Text Completions
In this study, we evaluate the effectiveness of the momentum optimized state vector on natural text completions. Given a natural text template, we instruct the model to greedily generate 5 tokens with or without intervention in the zero-shot setting. We use exact match accuracy as the metric. Table shows the result of natural text completions on Llama-2. The performance boosts observed with the coomentum-optimizze state vector on the separate tokens indicate that can guide the model to generate answers correctly. We include more examples of natural text completions in the Appendix.
F
Case Study
In this section, we present a case study shown in Table to demonstrate the efficacy of the momentum- optimized state vector in natural text completions. Consider the query: "What is the meaning of biography?", The vanilla Llama-2 model would directly answer this question. However, when influenced by an English-Frende state vector, Llama-2 changes its response, translating the question into French instead. Similarly, when presented with the sentence "When think of upright, think of".. Influenced by Antonym state vector, Llama-2 completes the sentence with an anonymous pattern. These instances exemplify the model learning the ICL function stored in the momentum optimized state vector, enabling it to generate context relevant to the specified task.
Full Result G
In this section, we provide the additional result with Ilama-2-7B GPT-J model. We first present the main result of optimization on the other six tasks except the main result, and the average performance across all tasks. As shown in Table 3, our inner optimization and momentum optimization effectively enhance the state vector.
Moreover, we provide result of state vector aggregation on two additional datasets. As shown in Figure 6, the trends of both D$C and average aggregation follow similar pattern to the main result shown in Figure as the number of examples increases, illustrating the effectiveness of our aggregation methods.
15

---

Model
Method
Capitalio
Country-Cpitall
Present-Past
Singubr-Plural
Person-Sport
AG News
Average (ATI)
Rogalar
0.0:1
0.0:1
000:
0:0:0:0
0:01
Pasction vector
98.6+ 04
67.4 20.7
50.24 4.5
94.24 06
1.4+00
52,24+00
44.7
Zar-shot
Task vector
92.9155
02.8123
95.2211
95.3119
869245
47.82113
60.4
State (inn.)
99.6234
94.02-13
97.1210
520=55
Lluma-22
Staee vecter (mom)
99. 1101
04. 04.5100
06.5207
966110
88 18.1
50.0.5.
76.3
99.9
18
98.3205
98.5201
94:3:02
76.0:33
Function vector
99.71
97.1
88.41
M0.7444
78.
Few-ebot
Task vector
18.0100
12.91:4
98. 98.2500
98.5115
954
64.3:394
81.5
State (inn.)
99.7+01
94,44113
76.0+8.5
State vector (mom))
99.34001
04.9100
98.3
98.800
057
76.32159
83.N
Rogalar
Pasetion voctor
66.34 9-
$7.0 98
63.1 1.1
60:1 2.1
46.41:4
57.4
Zero shot
Task
31.62:43
37.0
616412
464-40
55.0=33
State vector (inn.)
58.2+133
45.5+553
41.3+a20
619407
51.2-18
59,7=55,
47.8
GPTJ-60
State vector (mnam.)
584420
52.9454
45,92802
625507
514
61.32443
49.7
reguiar
59,300
82,4-35
Pasetion vector
98.622 0.6
78.61 5.1
90.81: 1.3
95.91 0.
81.6:
72.72:32
70.6
Fou-dbot
Task
99.3+-0.3
89,8+11
91.5+110
99.3+005
85.3-36
63.3+43
State (inn.)
99.42:03
89.21116
97.31:0
99.3105
83
75.72:12
State vector (mam..)
99.4
90.14:3.5
97.6-t:09
99.4
83.7=30
78.0-222
74.4
Table Additional performance of vector optimization all task. The hest results zero shot setting are underline and best results the few shot setting are bold. The result of basic state vector is mathematically equivalent to task vector, Average (AI.I.) represents the average performance across all 12 datasets.
other six tasks and average performance of
62
Som
12
Zom aud E6C
50 20 30
40
50
10
70 90
20.22 30 30
50 50
73 60 91
20 40 50 60 00.00 03
10 20 30 53 60 72 00
(a) Llama2-7B Porson-Occupation Llama2-7B Product-Company GPT-J-6B Person-Occupation
GPT-J-6B Product-Company
Figure Additional performance of aggregation across number of examples. Avg. denotes the average aggregation baseline and D&C. denotes the dividc-and and---und aggregation. The X axis represents the number of examples, and the Y axis represents the accuracy.
H Result on Larger Model
In this section, we provide the optimization and aggregation results on the larger model. Here we choose Alama-2-13BB and Llamm-2-70BB (quantized by GPTQ) their memory requirements suit our hardware conditions. We present the result of the optimization method on three representative datasets shown in T and the result of the aggregation method four representative datasets shown in Figure 7 and Figure [ The result shows that our inner and momentum optimization and D&C aggregation method could also benefit the state vector on the larger model setting.
I Qualitative Study
In Figure we present Principal Component Analysis (PCA) visualization of the original state vector GPT-J, applied both the Antonym task and the English-Frente French translation task. Note that the cluster distributions observed in GPT-J closely mirror those of Llama-2. This similarity indicates consistent and progressive enhancement the model capacity, originally identified in Llama-2 in $6.3 which is also shown on GPT-J. Such findings demonstrate the broad applicability and generalizability of our momentum optimization approach across different models.
J Robustness Analysis
In this appendix, we examine the robustness of the state vector with inner optimization. Specif- ically, we evaluate the task vector and the inner optimized state vector on the Llama-2 dataset, focusing three tasks. We measure and report the performance standard deviation using 100
16

---

Model
Method
Antonym
English-French
Person-nnttrument
Average
Regular
1.2.t 0.7
0,2 0.2
0.0% 0.0
0.5
Function vector
47.1 1.6
23.244.3
0.1 0.1
23.5
Zcro-shot
Task vector
46.01 2.4
43.1172
68.2.6.3
49.1
State vector (inn.)
47.0# 1.2
50,5+199
66.6=3.1
54.7
State vector (mom.)
47.9 1.1
55.9:13.4
68.5+22
57.4
Alama-2-13B
ICL baseline
67.0 0.1
1..5+
75.0=002
72.2
Function vector
65.7 1.7
75.2412
72.2
71.3
Few-shot
Task vector
64.8 1.2
70.5+33.
70.6-3.1
State vector (inn.)
65.5+ 0.8
75.8+116
1..0=1.3
72.8
State vector (mom.)
65.91
75.6
78.6
73.4
Regular
2.4
0.6::0.5
0.0 0.0
1.0
Function vector
48.3499.
29.940.3
5.340.3
27.8
Zero-shot
Task
63.34:2.1
48.8
63.5
58.5
State vector (inn.)
6..5
56.016.6
67.3
62.9
State vector (mom.)
66.7
57.8+
71.9
65.5
Llama-2-70B
ICL baseline
68.61228
81.6114
80.621.
76.9
Function vector
64.8+:2.3
81.74221
82.8
76.4
Few-shot
Task vector
67.1128
81.5+18
82.2=4.7
76.9
State vector (inn.)
68.04
82.9+199
82.6#2.3
State vector (mom.)
68.54222
82.542.1
83.2-26
78.1
Table 6: Performance of state vector optimization across three tasks on Llama-2-13B and Llama-2-70B. The best results in the zero shot setting are in underline and the best results in the few shot setting are in bold. The result of basic state vector mathematically equivalent task vector.
kayde
Sapde
Jar dux
10c
32 Number d0
Number noamble
Number coumple
Number
AG News
(b) Antonym
(c) English-French
(d) Product- Prodact-Compann Company
Figure 7: Performance of aggregation Llama-2- number of examples. Avg. denotes the average aggregation baseline and D&C. denotes the divide-and-on ann-uuuuuer aggregation. The X axis represents the number of examples, and the Y axis represents the accuracy.
diverse demonstrations or dummy queries. As illustrated in Figure 10 our analysis yields three key observations:
The task vector and state vector exhibit greater sensitivity to dummy queries than to demon- strations. This finding suggests that dummy queries compared demonstrations, underscoring the importance of reducing the noise from dummy queries to enhance state vector performance.
have a greater impact on performance
In the few-shot setting, both the task vector and the state vector (inn.) indicate significantly greater robustness compared to their performance the zero-shot setting. There noticeable reduction in the standard deviation across diverse demonstrations or dummy queries when applying demonstrations during ICL inference. This improvement may be attributed to the richer ICL function information provided by demonstrations, which in turn bolsters performance stability.
Compared to the task vector, our inner optimized state vector shows markedly enhanced robustness to the variations in demonstrations and dummy queries, in both zero-shot and few-shot settings. This highlights the effectiveness of our proposed inner optimization in improving the robustness of the state vector.
17

---

sasele
23
20-
70 15E
Number of example
Number example
Numberat manmple
Number esampple
(a) Person- Oeccpppttnotiion
(b) Anlonym
(c) English French
(d) menoon-nstrument
Figure 8: Performance of aggregation on 1lama-2-70B across number of examples. Avg. denotes the average aggregation haseline and C& denotes the divide-and -nd-uuu aggregation. The X axis represents the number of examples, and the Y axis represents the accuracy.
0.8
0.8
0.6
0.6
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) Antonym
(b) English-French
Figure The PCA visualization of the state vector in the Antonym task and English-French task GPT-J. where each color represents the state vector corresponding to examples occupying specific positions in the demonstration and the outlier is of the first order.
Zero-shot
Zero-shat task vector
Zero-shot state vector (inn.)
Zero-shot state vector (inn.)
Standard Deviation
(In..)
Standard Deviation
Few-shot vector Unn.)
Lit
2
1-
0
Antorym
Enn Paith
Antonym
Perrer Intimmmen Eng Fnghhhhwwhhh
(a) Demonstration Robustness
(b) Dummy Robustness
Figure 10: Standard deviation of performance on Llama-2 across three datasets.
K
Limitation
The definition of state vectors contingent upon specific assumptions and lacks a rigorous theoretical foundation, which may impact its generalizability and reliability across different NLP tasks. Addi- tionally, the experiments were conducted on a limited scale with moderate-size models and datasets. These constraints may affect the applicability of the results to larger models Further research will explore these aspects to establish a more robust validation of the proposed methods.
complex datasets.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
we provide our discussion of limitations in the Appendix K
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: the paper fully disclose all the information and details of main experimental results in the Section 5.1 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: we provide the instructions, data, codes and scripts of the main experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp) cc/
publ ic tttp /eedddeee//eeennnnme ss Loonolic cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ ctclpp..ccpuubicce
CooeeoodeSSSa Submisbbbsss ss cassoooPiiyyy for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we provide all the training and test detailed description of the experimental setup and details Section 5.1 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: we provide the standard error of the main results in Table Table
Table 5
Table O
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide the information on the computer resources in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pww/...eeeeopp.cco/ol//ea
Answer: [Yes]
Justification: the research conducted in the paper fully conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: all creators and original owners of assets used the paper, including code, data, and models, properly credited, and the licenses and terms of use are explicitly mentioned and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, baapeettsppprrssseeee paperswws thoode. com datasstt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "WMajupmtqL", "reviewer": "Reviewer_pJQh", "summary": "The paper considers the problem of using compressed vectors to replace explicit demonstrations, with potential benefits of providing new perspectives to better understand the mechanism of In-Context Learning (ICL), and addressing the issue of overly long demonstrations by summarizing them into a vector. The main challenge is the drop in accuracy. The paper proposes new methods to enhance compressed vectors, using ideas of ensemble averaging, momentum (though details are insufficient), and Divide-and-Conquer aggregation. Many ablation experiments are conducted to prove the effectiveness and answer different research questions.", "questions": "1. What is the strict definition of the momentum gradient optimization algorithm used as opt? Is it Polyak Momentum or Nesterov momentum?\n2. In the code, what do FixOptimizer, OneStepOptimizer, and FixOneStepOptimizer represent respectively? Which one represents momentum? \n3. For Divide-and-Conquer aggregation, how does the performance compare to directly averaging the different group-specific state vectors?", "limitations": "1. The presentation needs improvement.\n2. It is worth considering larger models, e.g., Llama2-70B. But existing experiments on Llama-2-7B, Llama-2-13B and GPT-J-6B is strong enough.\n3. It is worth discussing the performance on more complex ICL tasks, such as math and text summarization.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper conducts a large number of ablation experiments.\n2. The methods proposed are quite novel, including the heuristic approaches and Divide-and-Conquer aggregation. \n3. In practice, the methods perform very well on certain tasks.\n4. The work provides new perspectives to better understand the mechanism of ICL.", "weaknesses": "1. The development is still in its early stages, mostly heuristic, lacking any theoretical guarantees. However, I think this is not a critical issue, as the final performance is good in practice for certain tasks, and a rigorous analysis is very hard due to the complexity of transformers.\n2. The presentation needs improvement, e.g., Figure 4 should have a color legend for different positions. The specific parameter $L$ used should be clarified in Section 5.1\n3. The strict definition of the momentum gradient optimization algorithm used needs to be clearly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `pJQh`,  \n\nThank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf) that contains new figures and tables. We provide the analysis in `General Response` (shown in a separate comment on this page).\n\n\n***We would like to address your concerns and questions in detail below.***\n\n***\n\n## **Concern 1: Limitation of Model Size**\n\nThank you for your valuable feedback and for highlighting the potential limitations related to the model sizes. \n\nIn response to your concern, we have conducted additional experiments using the larger Llama2-70B model to evaluate the effectiveness of the proposed optimization method and the D&C aggregation. Please refer to the \"Experiment on Llama-2-70B\" in our `General Response` for detailed experiments and analysis. Notably, while the performance of ICL improves with the larger Llama2-70B, our method still effectively enhances the state vector and outperforms regular ICL in a few-shot setting.  \n\nWe appreciate your feedback and hope these additional results address your concerns.\n\n## **Concern 2: Limitation of Datasets**\n\nThank you for your feedback and for highlighting the potential limitations associated with the tasks.  \n\nTo explore the effectiveness of state vectors on more complex tasks, we conducted experiments on the alignment task, which involves aligning LLMs with human preferences and is inherently more complex and diverse. Please refer to the \"State Vector for Alignment\" in our `General Response` for detailed experiments and analysis. \n\nNotably, our state vectors can achieve 85% of the performance of mainstream alignment-tuning methods (i.e., instruction-finetuning and reinforcement learning from human feedback) without training. With Mistral-7B and Llama-2-70B, our state vectors can achieve 80% of the alignment performance of GPT-4-0613.  \n\nWe hope these additional experiments provide a thorough understanding of our method's capabilities on complex tasks.\n\n## **Question 1: Definition of  Momentum Gradient Optimization**\n\nThank you for your valuable feedback and for raising an important point regarding the strict definition of the momentum gradient optimization algorithm used in our study.  \n\nIn this paper, we employed the Polyak Momentum Optimization algorithm. To ensure clarity and enhance the reader's understanding, we have provided the update rules for all optimization algorithms used in Table 2, corresponding to the $opt(*)$ function calculation method described in Equation 9 in the paper.   \n\nTo maintain consistency with the notation used in the paper, in the following formulas, $E_i$ denotes the $i$-th ($1 \\le i \\leq N$) state vector (we ignore the hyper-parameter $L$ for simplicity), and $N$ is the number of state vectors. Additionally, let $v_{t}$ represent the update vector at $t$-th iteration, initialized as $v_0=\\boldsymbol{0}$. Here, $\\alpha$ denotes the learning rate, $\\beta$ denotes the momentum coefficient, and $\\epsilon$ is a very small constant for calculation stability. Below are the detailed update rules for each optimizer:  \n\n1. Polyak Momentum Optimizer (mom.)\n\n$g_t = E_{t}-E_{t-1}$\n\n$v_{t}=\\beta v_{t-1}+(1-\\beta)g_t$\n\n2. AdaGrad Optimizer (adag.)\n\n$g_t = E_{t}-E_{t-1}$\n\n$s_t=s_{t-1}+g_t \\cdot g_t$\n\n$v_{t}=\\frac{1}{\\sqrt{s_t+\\epsilon}} \\cdot g_t$\n\n3. RMSprop Optimizer (rms.)\n\n$g_t = E_{t}-E_{t-1}$\n\n$s_t=\\beta s_{t-1} + (1-\\beta) g_t \\cdot g_t$\n\n$v_{t}=\\frac{1}{\\sqrt{s_t+\\epsilon}} \\cdot g_t$\n\n4. Adam Optimizer (adam.)\n\n$g_t = E_{t}-E_{t-1}$\n\n$s_t=\\beta_1 s_{t-1} + (1-\\beta_1) g_t \\cdot g_t$\n\n$h_{t}=\\beta_2 h_{t-1}+(1-\\beta_2) g_t$\n\n$\\hat{s}_t=\\frac{s_t}{1-\\beta_1^t}$\n\n$\\hat{h}_t=\\frac{h_t}{1-\\beta_2^t}$\n\n$v_{t}=\\frac{1}{\\sqrt{\\hat{s}_t+\\epsilon}} \\cdot \\hat{h}_t$\n\nAfter $N$ iterations, we use the final update vector $v_{N}$ to optimize the state vector:  \n\n$opt([E_i]^N_{i=1})=\\alpha * v_{N}$\n\nWe hope this clarifies the strict definition and update rules for the momentum gradient optimization algorithm used in our study. Thank you once again for your insightful question.\n\n## **Question 2: Comparison between D&C and Averaging Aggregation**\n\nIn our aggregation experiment, we compared our D&C aggregation with the average aggregation method. The average aggregation method involves averaging the group-specific state vectors. As shown in Figures 2 and 6, although average aggregation benefits from an increasing number of examples, D&C aggregation outperforms average aggregation when the number of examples is the same. This is primarily due to the fact that D&C aggregation better captures the information in the group-specific state vector, leading to more robust and better performance.  \n\nThank you for your question.  We hope this comparison adequately addresses your question.  \n\n## **Question 3: Optimization in Code**\n\nIn the code, *FixOptimizer* denotes the inner optimization. *OneStepOptimizer*  denotes the direct momentum optimization  on the original state vector, which we did not apply.  *FixOneStepOptimizer*  denotes the momentum optimization on the inner optimized state vector , and this represents the \"momentum optimization\" mentioned in the paper. \n\n## **Suggestion of Presentation**\n\nWe will revise `Figure 4` to include a color legend for different positions to enhance its clarity. For the specific parameter used in the paper, we provided them in the \"implementation details\" in `Appendix A`. We appreciate your suggestions and hope these could address your concerns.  \n\n***\n\nThank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AuJkQtCKGa", "reviewer": "Reviewer_FZSw", "summary": "This paper aims to revealing the working mechanism of the compressed state vector in context learning. They first prove the state vectors are similar with parameters trained via gradient descent. Then, they propose three methods to optimize such kind of parameters: (1) inner optimization averaging each vector of the separate token; (2) momentum optimization using momentum-based optimization algorithm; (3) divide-and-conquer strategy aggregating several vector groups. They test these methods on some medium-sized LLMs. Experiments show some improvements against other compressed vector baselines.", "questions": "1.\tDoes the state vector without optimization actually function the same as the task vector? If not, I think there should be another baseline called state vector (w/o optimization). The authors should claim whether this is a new compressed vector method or just a prove of similarity to gradient descent.\n2.\tCan the authors provide more rationality of optimizing the state vector? I understand the state vector is a compression of previous context, but what it would present after the optimization?\n3.\tExperiments show that the inner optimization and momentum optimization prefer different tasks. Is there a conclusion when to use which? Or is there a combination to adaptively cover all the situations?\n4.\tCan the authors elaborate more on the differences between averaging and D&C? Why D&C is worse when the number of examples equals 10 (Figure 2)? In my understanding, they should be the same since the authors use 10-shot ICL (there should be only 1 group).\n5.\tThe authors should check the correctness of the experiment results. For example, in Table 5, they report Llama-2 Zero-shot Regular with six zeros but with an average of 0.2.", "limitations": "See questions", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The contribution of this paper is significant. It attempts to provide understanding of the mechanism of ICL via a gradient descent perspective. The proposed state vector can be viewed as trained parameters so that they can be optimized with future methods. \n3. The proposed optimization methods are reasonable. Experiments with these methods support their claims. The performance can be improved in most scenarios.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, and avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc. Please keep in mind that the rebuttal period is not necessarily enough to run new experiments. As a result, asking for new results is rather unrealistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `FZSw`,  \n\nThank you for your review. According to the feedback from other reviewers, we have conducted additional experiments and analysis. We have uploaded a [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf) that contains new figures and tables. We provide the analysis in `General Response` (shown in a separate comment on this page).\n\n***We would like to address your concerns and questions in detail below.***\n\n***\n## **Question 1: Difference with Task Vector**\nThank you for your feedback. Since the task vector aims to extract the hidden state in the initial $L$layers for intervention, it is functionally equivalent to our state vector without optimization that extract attention activations in initial $L$layers. However, we must emphasize that the proposed state vector differs significantly in its integration into the model. In our theoretical framework, the state vector is defined based on attention activations of the separator token, which can be viewed as updated parameters. Therefore, even though it is mathematically equivalent to the task vector, the state vector possesses a stronger theoretical foundation and interpretability.  \n\n## **Question 2: Contribution Clarify**\nWe would like to accurately describe our work and contributions. Previous work [1] proposed the dual view of ICL and gradient descent and provided evidence from the perspective of attention pattern similarity. Our work extends this understanding and applies it to the working mechanism of ICL compressed vectors. Based on our understanding and theoretical framework, we propose the definition of the state vector and introduce its optimization and aggregation methods.  Our paper can be considered as primarily proposing a new compressed vector method, inspired by the similarity between ICL and gradient descent.  Therefore, our work establishes a bridge between the study of ICL's working mechanism and the application of ICL compressed vectors.  \n\n[1] *Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers*\n\n## **Question 3: Rationality of Optimization**\nThank you for your question. In our perspective, we consider the state vector not simply as a compression of demonstration text information, but rather as a storage of the implicit ICL task function information contained within the demonstrations. Our optimization methods have enhanced the accuracy of this task information.  The experiments in `Section 6.1 Qualitative Study` can intuitively demonstrate this point. As shown in `Figure 4`, a trend is observable in the movement of these clusters as the example position increases, indicating that as the number of demonstrations increases,  the state vector becomes closer to the ideal task representation. Our momentum optimization effectively utilizes this trend and captures the underlying task dynamics within the demonstrations, thereby enhancing the state vector.    \n\n## **Question 4: Choice of Optimization Method**\nThank you for your insightful question. \n\nBased on the experimental results, we find that momentum optimization performs better on Knowledge tasks, the reason may be that more examples can effectively promote the model's recall of internally stored knowledge, while 10 examples may be insufficient.  Therefore, our momentum optimization can enhance the accuracy of the knowledge-based task function stored in the state vector by extending its trend. \n\nOn the other hand, inner optimization shows better average performance on Linguistics and Translation tasks. This might be because, for these tasks, 10 examples are sufficient for the task,  and momentum optimization could over-edit the state vector, thereby introducing noise. \n\nIn light of these findings, we suggest using momentum optimization when increasing the number of examples effectively enhances ICL performance. Conversely, when an increase in examples does not contribute much to performance improvement, using only inner optimization is recommended.  \n\nWe hope this explanation addresses your question. Please let us know if you have any further questions; we would be happy to discuss them with you.\n\n## **Question 5: Difference of Aggregation Result in 10-shot**\nThank you for your question. We would like to elaborate on the differences between averaging aggregation and D&C aggregation in the 10-shot setting.\n\nIn the 10-shot setting, we have only one demonstration group. For averaging aggregation, we directly extract the group-specific state vector from this single demonstration group, which becomes the final aggregated state vector.\n\nFor D&C aggregation, the process is slightly different. In the divide stage, we extract the group-specific state vector from the group-specific dummy query. In the conquer stage, we first pair the group-specific dummy query with its label to form a one-shot demonstration. Then, we input a new dummy query with this one-shot demonstration and extract the aggregated state vector from the dummy query. Thus, the D&C aggregated state vector and the averaging aggregated state vector are not the same.  \n\nOur analysis suggests that the D&C aggregated state vector performs worse when the number of examples is small because the conquer stage relies on a limited number of examples. This limitation prevents the model from effectively compressing the information from the group-specific state vector into the final D&C aggregated state vector.   \n\n## **Typo in Table 5**\n\nWe would like to clarify that the average result in Table 5 represents the average performance across 12 tasks, not just the 6 additional tasks. We appreciate your feedback and will revise our manuscript to ensure this is more clearly communicated.  \n\n*** \n\nThank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7z6pmiWczM", "reviewer": "Reviewer_yzeU", "summary": "The authors focus on the compressed vectors in In-Context Learning (ICL). They highlights the similarities between the compressed vectors and parameters trained via gradient descent, proposing the formulation of the state vector. Then they applies two optimization algorithm to progressively refine state vector. The proposed method achieves SotA performance on diverse tasks.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The authors introduce the formulation of state vector that encapsulates the processing state of ICL stored in the attention activations.\n2.The proposed two optimization methods for the state vector seem to be effective.\n3. The research topic is significant and intriguing", "weaknesses": "1.The proposed state vector seems to be benefit from examples (as shown in Eqn. 5), could you explain how it works without demonstration in the zero-shot setting (as shown in Table 1)?\n2.As ICL baseline makes predictions conditioned on the demonstrations, could you explain why its performance remains unchanged with different number of examples in Figure 2?\n3.The proposed method is mainly draw inspiration from trainable parameters and gradient optimization algorithm. However, the performance of applying other classical gradient optimization algorithms has deteriorated significantly in Table 2. And the authors argue that “The outcome indicates a discrepancy between the state vector and updated parameters with gradient descent.” This confuses me a lot. The authors should state their similarities and differences more clearly and precisely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `yzeU`,  \n\nThank you for your review. According to the feedback from other reviewers, we have conducted additional experiments and analysis. We have uploaded a [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf) that contains new figures and tables. We provide the analysis in `General Response` (shown in a separate comment on this page).\n\n***We would like to address your concerns and questions in detail below.***\n\n***\n\n## **Question 1: State Vector Usage in Zero-shot**\n\nWe would like to clarify the input of our optimization method. When extracting the state vector, we  use demonstrations and a dummy query as input. Then we extract attention activations from the forward pass to form the state vector. Thus, the process is the same for both the zero-shot and few-shot settings. Subsequently, we obtain the processed state vector through optimization or aggregation methods. The extraction and the following optimization or aggregation are performed only once since they are unrelated of the test data.  \n\nDuring testing, our experiments are divided into zero-shot and few-shot settings, which differ based on the input to the test data. In the zero-shot setting, we only input the query, whereas in the few-shot setting, we input both the demonstrations and the query. During the forward pass, we use the attention activations stored in the state vector to replace the ones computed in the transformer.  \n\nWe hope this explanation addresses your question.  Should you have any further inquiries, we are at your disposal for any additional discussions.  \n\n## **Question 2: Baseline Result In Figure 2**\n\nThank you for your question. \n\nIn Figure 2, the ICL baseline denotes the 10-shot ICL result. Due to the input length limitations of the model, regular ICL cannot be directly used on multiple examples (e.g., the average input length of AG News in the 10-shot setting is 3872, while the maximum input length of Llama-2 is 4096). Therefore, we only use the 10-shot ICL  as baseline, while we provide the averaging aggregation as a stronger baseline. We hope this explanation addresses your question.\n\n## **Question 3: First-Order Gradient Optimizer Result**\n\nHere, we would like to provide a more detailed analysis. Firstly, we present the calculation formulas for all optimizers. In the following formulas, $g_t$ and $v_{t}$ denote the initial gradient and the optimized gradient at the $t$-th iteration, respectively. $\\beta$ denotes the momentum coefficient.  \n\n**1. Polyak Momentum Optimizer (mom.)**\n\n$v_{t}=\\beta v_{t-1}+(1-\\beta)g_t$\n\n**2. AdaGrad Optimizer (adag.)**\n\n$s_t=s_{t-1}+g_t \\cdot g_t$\n\n$v_{t}=\\frac{1}{\\sqrt{s_t+\\epsilon}} \\cdot g_t$\n\n**3. RMSprop Optimizer (rms.)**\n\n$s_t=\\beta s_{t-1} + (1-\\beta) g_t \\cdot g_t$\n\n$v_{t}=\\frac{1}{\\sqrt{s_t+\\epsilon}} \\cdot g_t$deacay\n\n**4. Adam Optimizer (adam.)**\n\n$s_t=\\beta_1 s_{t-1} + (1-\\beta_1) g_t \\cdot g_t$\n\n$h_{t}=\\beta_2 h_{t-1}+(1-\\beta_2) g_t$\n\n$\\hat{s}_t=\\frac{s_t}{1-\\beta_1^t}$\n\n$\\hat{h}_t=\\frac{h_t}{1-\\beta_2^t}$\n\n$v_{t}=\\frac{1}{\\sqrt{\\hat{s}_t+\\epsilon}} \\cdot \\hat{h}_t$\n\nWe believe there are two main reasons why first-order gradient optimizers are not well-suited for state vector optimization: \n\n- First-order gradient  optimizers use adaptive learning rates, which require a significant amount of historical information. This strategy often shows instability and reduced effectiveness when data is limited. \n\n- First-order gradient  optimizers have more complex hyper-parameters than the Momentum Optimizer, making it challenging to find the optimal hyper-parameters. Experimental results indicate that directly using hyper-parameter settings commonly used in gradient descent results in poor performance.  \n\nWe hope this explanation  resolves your concerns. If you have any further questions, please feel free to let us know, and we would be happy to discuss them with you.  \n\n*** \n\nThank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5SRi4FwZ2", "reviewer": "Reviewer_4gb3", "summary": "The paper presents an in-depth analysis and optimization of In-Context Learning (ICL) within large language models (LLMs). In particular, it proposes inner and momentum-based optimization techniques to increase the performance of in-context learning.", "questions": "In principle, ICL plays a more significant role in larger models (also with better long-sequence performance). Does this method benefit these models?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. In-depth analysis of ICL vectors across 12 different datasets (linguistic, translation, and knowledge). The analysis provides insights into ICL vectors and motivates the optimization.\n2. The optimization method can be divided into inner and momentum-based stages, each of which has shown promising improvement.\n3. The ablation study is abundant.\n4. The presentation of the research question is clear to read and follow.", "weaknesses": "1. The method itself is somewhat simple. Specifically, the inner optimization is merely a uniform averaging.\n2. The definition of state vectors lacks a rigorous theoretical basis, making it difficult to infer the approach's generalizability and reliability across different NLP tasks.\n3. The experiments are conducted on Llama-2-7B and GPT-J-6B, which are limited in terms of scale and performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `4gb3`,  \n\nThank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf) that contains new figures and tables. We provide the analysis in `General Response` (shown in a separate comment on this page).\n\n***We would like to address your concerns and questions in detail below.***\n\n***\n## **Concern 1: Method is too Simple**\nWe respectfully disagree with the comments on the simplicity of our work. Before we dive into the concrete points, we would like to clarify our motivations and contributions:  \n- We have conducted a comprehensive study of the working mechanism of the compressed vector in ICL. We demonstrated that attention activations can be equivalent to updated parameters via gradient descent under specific conditions. Leveraging this understanding, we proposed the definition of the state vector. To the best of our knowledge, we are the first to analyze the compressed vector in ICL through the dual view of ICL and gradient descent.  \n- Inspired by the concepts of model soup and momentum gradient optimizer, we proposed the Inner Optimization and Momentum Optimization methods for the state vector. Extensive experiments on a range of tasks have shown that our methods significantly enhance the effectiveness and efficiency of the state vector.  \n- To address the example limitation caused by model input length constraints, we introduced the Divide-and-Conquer Aggregation method. Extensive experiments demonstrate that our approach effectively aggregates information from state vectors, successfully scaling up to a large number of examples.  \n\nAlthough our inner optimization can be viewed as a kind of uniform averaging, it is a simple yet effective method to enhance the state vector within a single forward pass. Additionally, our proposed momentum optimization further enhances the inner optimized state vector. Our work not only provides a novel theoretical perspective on ICL compressed vectors but also introduces practical applications for the study of ICL's working mechanism. **In this way, our work, as the first to establish a bridge between the theoretical understanding of ICL's working mechanism and the application of ICL compressed vectors, offers a novel integration in the field.**  \n\nWe hope this explanation addresses your concerns. If you have any further questions, please feel free to let us know, we would be happy to discuss them with you.\n\n## **Concern 2: Limitation in Theory Basis**\nWe acknowledge that our theoretical development is still in the early stages. Due to the complexity of transformers, it is very challenging to rigorously prove the mechanisms of ICL is very challenging. However, our method demonstrates that under specific conditions, ICL can execute gradient descent algorithms. Previous works [1,2] have also built upon this understanding, conducting experiment to support it through attention pattern similarity.  \n\nWhile we cannot mathematically infer the generalizability and reliability of our method, we have extensively validated it across 12 datasets. The experiments demonstrate that our method is both generalized and robust.  Additionally, we have provided results on more complex and realistic alignment tasks. Please refer to the\"State Vector for Alignment\" in our `General Response` for detailed experiments and analysis. Notably, our optimized state vectors can achieve 85% of the alignment performance of GPT-4 on Mistral-7B and Llama-2-70B. \n\nWe hope this explanation and the additional experiments address your concerns regarding the theoretical basis and practical applicability of our approach.  \n\n[1] *Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers*\n\n[2] *In-context Learning and Gradient Descent Revisited*\n\n## **Concern 3: Limitation in Model Size**\nThank you for your insightful feedback and for pointing out the potential limitations concerning model sizes.  \n\nIn response to your concerns, we have performed additional experiments using the larger Llama2-70B model to evaluate the effectiveness of our proposed optimization method and the D&C aggregation. Due to resource constraints, we were unable to extend our method to even larger models. Detailed results and analysis can be found in the \"Experiment on Llama-2-70B\" in our `General Response`. Importantly, although the performance of ICL exhibits some improvement with the larger Llama2-70B, our method continues to significantly enhance the state vector and surpass regular ICL in few-shot settings.  \n \nWe hope that these additional results satisfactorily address your concerns.  \n\n*** \n\nThank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion. If you find that our response addresses your concerns, would you kindly consider raising your rating score for our paper? We greatly appreciate your consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtZh7Sykkh", "reviewer": "Reviewer_MvM2", "summary": "This work introduces the novel concept of the state vector. It encapsulates the information of ICL examples from separate tokens as anchors. Drawing inspiration from the duality between transformer attention and gradient descent, the authors implement inner and momentum optimization to generate task-specific state vectors. To address the context length limitation of LLMs, a divide-and-conquer strategy is employed to aggregate information from multiple examples into a single state vector. Experimental results demonstrate that patching the state vector during the inference forward pass outperforms previously proposed task vector and function vectors.", "questions": "## Summary of review\n\nIn general, the paper presents an interesting concept, and the overall submission is technically sound. Therefore I am positive to this paper at this point.\n\nHowever, I hope the authors can address the several concerns raised in the weakness section by providing extra explanations and discussions regarding the unclear aspects. Also, please response to the additional questions shown as follows. \n\nIf the authors can well address my concerns/questions, I will keep the positive accept rating.\n\n## Additional questions\n1. I would like to know whether the ICL state vector can be used in actual alignment scenarios. If we switch to a long-context version of the model, how will the ICL state vector be different?\n\n2. The previous study demonstrates that the label words are information anchors [1], while this work aggregates the information of separate tokens in the state vector, is there any evidence that the separate tokens can also gather in-context information in a progressive manner?\n\n[1]Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9840–9855, Singapore. Association for Computational Linguistics.", "limitations": "n/a", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Although the idea of using vectors to capture in-context learning is not new, the proposed approach of optimizing the state vector using inner/momentum optimization is well-motivated.\n\n2. The paper is well-organized and sound with experiments and ablation studies.\n\n3. The experimental results and the analyses provide some useful insights.", "weaknesses": "1. It is not clear whether larger models or more complex datasets can still benefit from this work. Is this work only applicable to relatively small models?\n\n2. It is unclear why a dummy query has to be included when extracting the state vector, whose task is to encapsulate the examples only. What if not? \n\n3. Can we use this vector to align LLMs further, considering encapsulating them into memory/vector bases for reuse and continuous learning of new knowledge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer `MvM2`,  \n\nThank you for your review. According to the feedback from you and other reviewers, we have conducted additional experiments and analysis. We have uploaded a [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf) that contains new figures and tables. We provide the analysis in `General Response` (shown in a separate comment on this page).\n\n***We would like to address your concerns and questions in detail below.***\n\n***\n## **Concern 1: Limitation in Model Size and Datasets**\n\nTo address your concerns, we have conducted additional experiments using the larger Llama-2-70B model to evaluate the effectiveness of our proposed optimization method and the D&C aggregation. Detailed results and analysis can be found in the   \"Experiment on Llama-2-70B\" in our `General Response`.  Notably, the performance of ICL shows some improvement with the larger Llama2-70B model; our method continues to significantly enhance the state vector and surpass regular ICL in few-shot settings.  \n\nRegarding the dataset limitations, please refer to the response to \"Question 2\" where we present experiments on more complex alignment tasks.  \n\nWe hope that these additional results satisfactorily address your concerns.\n\n## **Question 1: Reason of Introducing Dummy Query**\n\nIn our preliminary experiments, we observed a decline in performance in state vector extraction when we omitted the dummy query and simply added a separate token right after the demonstration. This finding aligns with earlier studies [1,2]. The issue likely arises because omitting the query alters the input format significantly from regular ICL. Therefore, we follow the previous settings by introducing a \"dummy query\" and extracting the state vector from the separator token that follows this dummy query.\n\nThank you for your insightful question. We appreciate your feedback and hope this clarifies our approach.\n\n[1] *In-context learning creates task vectors*\n\n[2] *Function vectors in large language models*\n\n## **Question 2: Performance on Alignment Task**\n\nThank you for your valuable question.  We present the performance of the optimized state vector on the alignment task in a zero-shot setting. The results are shown in `Table 8` in the `Rebuttal PDF`. Our findings indicate that although the state vector is slightly inferior to the regular ICL baseline, it still demonstrates significant potential as an effective alignment approach. By omitting the demonstration in the input, our method significantly reduces the time cost (e.g., by 5.4$\\times$ on Llama-2-7B) while achieving 90% of the performance of regular ICL.  \n\n Moreover, compared to mainstream alignment-tuning methods such as instruction fine-tuning and reinforcement learning from human feedback, our method achieves an average of 85% of their performance without requiring any additional training. Notably, with models like Mistral-7B and Llama2-70B, our state vectors can achieve 80% of the alignment performance of GPT-4-0613. These results demonstrate that state vectors can enable efficient and effective model alignment, and that inner optimization is beneficial for complex alignment tasks.  \n\nWe appreciate your feedback and hope this addresses your question comprehensively. \n\n## **Question 3: Performance on Longer Model**\nThank you for your insightful question. Switching to a long-context version of the model will result in a slightly different ICL state vector due to several factors:\n\n1. **Extended Token History:** The state vector can access a more extended history of tokens, allowing it to retain more contextual information and improve performance on tasks with long-term dependencies.\n\n2. **Capturing Intricate Patterns:** With a longer context, the state vector can capture more intricate patterns and relationships within the data, leading to more accurate and contextually appropriate responses.\n\n3. **Coherent Outputs:** Long-context models generally produce more coherent outputs, as the state vector maintains a stable understanding over extended sequences, reducing the risk of losing context.\n\n4. **Enhanced Complexity Handling:** The state vector in a long-context model can handle complex tasks more effectively, benefiting from the comprehensive integration of extensive contextual information.\n\nWe appreciate your feedback and hope this explanation clarifies the potential impacts of using a long-context model.\n\n## **Discussion about information anchors**\nThank you for your insightful comments. We are pleased to discuss the mechanism of information gathering in ICL.\n\nOur research provides further evidence that separate tokens can progressively gather in-context information. In this paper, we investigate the impact of layer selection on state vector extraction in `Section 6.2 Layer Selection.` We observe that initially, increasing the number of layers for state vector extraction improves performance, and it reaches peak performance around the middle layers. This indicates that ICL information is progressively integrated into the separate token. In the work \"Label Words are Anchors,\" it is found that information flows first from the example to its corresponding label words in the early layers, and then from the label words' representation to the final separate token representation in different layers. This also shows that ICL information progressively flows to the separate token and peaks around the middle layers. These results are consistent with our observations.\n\nWe hope this explanation addresses your concerns. If you have any further questions or need additional clarification, please let us know. We are more than happy to discuss this further.\n\n[1] *Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning.*\n\n*** \n\nThank you very much again for your great questions and suggestions. Please let us know if you have any further questions, as we are happy to continue the discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We greatly appreciate your insightful reviews and are delighted that you have acknowledged our paper's strengths. We briefly summarize them as follows:

- **Novelty:** "The methods proposed are quite novel.", "The research topic is significant and intriguing.", "The paper presents an interesting concept.", etc.
- **Deep Analysis and Abundant Experiment:** "The paper presents an in-depth analysis and optimization of ICL.", "The paper conducts a large number of ablation experiments.", etc.
- **Writing:** "The paper is well written and easy to follow.", "The presentation of the research question is clear to read and follow.", "The paper is well-organized.".
- **Fully Motivated:** "The proposed optimization methods are reasonable.", "the proposed approach of optimizing the state vector using inner/momentum optimization is well-motivated.", etc
- **Effectiveness:** "The proposed two optimization methods for the state vector seem to be effective.", "The optimization method has shown promising improvement.", etc

***
We present the additional experiments on a larger model and more complex datasets. **Please check out the [Rebuttal PDF](https://openreview.net/attachment?id=ulPGXOjfvv&name=pdf)**. The following are the brief analysis and findings: 

### **Experiment on Llama-2-70B**

We provide the optimization and aggregation results on the Llama-2-70B model. Due to resource constraints, we are unable to conduct our method on very large models. The results of the optimization method are presented in `Table 7`, and the aggregation method results are shown in `Figure 10`. We find that, compared to smaller models, all results improve when applied to Llama-2-70B.  Furthermore, both the inner optimization and momentum optimization effectively enhance the state vector, either outperforming or being comparable to regular ICL in zero-shot and few-shot settings. For the aggregation results, the trends observed in smaller models remain evident in the larger model, with the D&C method still outperforming the averaging aggregation in multiple example settings. These results indicate that our inner and momentum optimization, as well as the D&C aggregation method, can also benefit the state vector on the larger Llama-2-70B model.  

### **State Vector for Alignment**

We present the performance of the inner optimized state vector on the alignment task in a zero-shot setting. We evaluate our method using two automatic alignment benchmarks: alpaca-eval (2.0)[1] and just-eval[2]. The results shown in `Table 8` indicate that although our state vector is slightly inferior to the regular ICL baseline, it still demonstrates significant potential as an effective alignment approach. By omitting the demonstration in the input, our method significantly reduces the time cost (e.g., by 5.4$\times$ on Llama-2-7B), but achieves 90% performance of regular ICL. Compared to mainstream alignment-tuning methods (i.e., instruction finetuning and reinforcement learning from human feedback), our method achieves an average of 85\% of their performance without requiring any training. Notably, with Mistral-7B and Llama2-70B, our state vectors can achieve 80\% of the alignment performance of GPT-4-0613. These results demonstrate that state vectors can enable efficient and effective model alignment, and that inner optimization is beneficial for complex alignment tasks.

[1] *Length-controlled alpacaeval: A simple way to debias automatic evaluators.*

[2]  *The unlocking spell on base llms: Rethinking alignment via in-context learning.* 

***

We sincerely thank the reviewers for their constructive suggestions and questions to enhance our paper. We will address your questions and concerns below and incorporate your valuable suggestions into our revisions. Please reply if you have any further questions, and we will be more than happy to continue the discussion.

Meta-review: **Summary:**

The paper explores compressed vectors to replace explicit demonstrations in In-Context Learning (ICL), aiming to improve understanding and address the issue of lengthy demonstrations. It introduces novel methods like ensemble averaging, momentum, and Divide-and-Conquer aggregation, supported by extensive ablation experiments.

**Strengths:**

- Innovative methods for enhancing compressed vectors in ICL.
- Extensive ablation studies provide valuable insights.
- Strong practical performance on certain tasks.

**Weaknesses:**

- Methods are largely heuristic and lack theoretical guarantees.
- Presentation needs improvement, with clearer explanations of key details.

**Reason to Accept:**
The paper's novel approaches and solid experimental validation offer valuable contributions to ICL, making it a worthwhile acceptance despite some early-stage development and presentation issues.
Decision: Accept (poster)