Improving Input-label Mapping with Demonstration Replay for In-context Learning
Authors: Zhuocheng Gong
Jiahao Liu
Qifan Wang
Jingang Wang
Xunliang Cai
Dongyan Zhao
Rui Yan
Venue: EMNLP 2023 Findings (2023)
Keywords: in-context learning
causal language modeling

Abstract: In-context learning (ICL) is an emerging capability of large autoregressive language models where a few input-label demonstrations are appended to the input to enhance the model's understanding of downstream NLP tasks, without directly adjusting the model parameters. The effectiveness of ICL can be attributed to the strong language modeling capabilities of large language models (LLMs), which enable them to learn the mapping between input and labels based on in-context demonstrations. Despite achieving promising results, the causal nature of language modeling in ICL restricts the attention to be backward only, i.e., a token only attends to its previous tokens, failing to capture the full input-label information and limiting the model's performance. In this paper, we propose a novel ICL method called Repeated Demonstration with Sliding Causal Attention, (RdSca). Specifically, we duplicate later demonstrations and concatenate them to the front, allowing the model to `observe' the later information even under the causal restriction. Besides, we introduce sliding causal attention, which customizes causal attention to avoid information leakage. Experimental results show that our method significantly improves the input-label mapping in ICL demonstrations. We also conduct an in-depth analysis of how to customize the causal attention without training, which has been an unexplored area in previous research.

Improving Input-label Mapping with Demonstration Replay for In-context Learning
Zhuocheng Gong'; Jiahao Liu', Qifan Wang
Jingang Wang', Xunliang Cai?, Dongyan Zhao Rui Yan
'Wangxuan Institute of Computer Technology, Peking University
2 Meituan; 'Meta AI; "National Key Laboratory of General Artificial Intelligence BBijjing Institute for General Artificial Intelligence
"Gaoling School of Artificial Intelligence, Renmin University of China (gzhch, zhaody } @pku. edu en, ruiyanerue edu. cn, wqfcrê fb. com (liujiahaol2, wwnggiingnggenee caixun ling (eweemmttuu .aaaanccoo com
{
Abstract
tasks in different fields. One of the most re- markable abilities of LLMs is in-context learn- ing (ICL) (Brown et al.. 2020). By providing a few demonstrations and instructions into the input, along with the input queries, LLMs can perform well in new tasks without requiring fine-tuning. The secret of ICL to formulate the input as the natural language generation task, then the LLM can be activated to prompt knowledge learned the pre-trane pre-training stage.
In-context learning (ICL) is an emerging ca- pability of large autoregressive language mod- els where a few input-label demonstrations are appended to the input to enhance the model's understanding of downstream NLP tasks, with- out directly adjusting The effectiveness of ICL can be attributed to the strong language modeling capabilities of large language models (LLMs), which enable them to learn the mapping between input and labels based on in-context demonstrations. De- spite achieving promising results, the causal nature of language modeling in ICL restricts the attention to be backward only, i.c., to- ken only attends to its previous tokens, failing to capture the full input-label information and limiting the model's performance. In this pa- per, we propose novel ICL method called Repeated Demonstration with Sliding Causal Attention, (RDSCA). Specifically, cate later demonstrations and concatenate them to the front, allowing the model to 'observe' the later information even under the causal re- striction. Besides, we introduce sliding causal attention, which customizes causal attention avoid information leakage. Experimental results show that our method significantly im- proves the input-label mapping in ICL demon- strations. We also conduct in-depth analysis how to customize the causal attention with- out training, which has heen unexplored area in previous research.
model parameters.
Despite the promising results demonstrated by existing ICL models (Gonen al., 2022; Wei al., 2022), their causal nature in language modeling re- stricts each token's attention solely to its preceding tokens. As a result, these models fail to capture the complete input-label information, thereby lim- iting their overall performance. Specifically, the pre-training objective of the current autoregressive LLMs focuses on predicting future tokens based past ones (Radford al., 2018), implemented with causal attention. While this approach works well for modeling regular sequences, it becomes less effective when applied to ICL tasks. The limi- tation of causal attention restricts ICL demonstra- tions to having only left context, which hampers the model's ' ability to fully comprehend and exploit the input-label relationship.
dupli-
Unlike tokens in a sentence that possess sequen- tial dependencies, there is no inherent sequential relationship between the demonstrations in the ICL input. Therefore, it is desirable for these demonstra- tions to interact with one another comprchensively. rather than relying solely on later demonstrations attending carlier ones, while the reverse not possible. Intuitively, if we can enable each demon- stration to attend to all the others, we can poten- tially obtain a more sophisticated context for the
1 Introduction
Large language models (LLMs) have become the backbone of various natural language processing
*Work done during an internship at Meituan.
'Corresponding authors: Dongyan Zhao (zhaody @wuoodd@ppu.ddu.cmm and Rui Yan ..n...c.

---

ICL query. However, achieving this on an LLM pre-trained with the objective of causal language modeling not straightforward. Simply removing the causal restriction and allowing the model to have access to the right context during inference is not feasible, as it would result in significant dis- parity between training and inference conditions.
between demonstrations.
We validate the feasibility of customizing causal attention during the inference stage without further training and conduct further analysis on causal attention customization. We believe this idea has great potential and sheds new light on optimizing ICL and other large model inference scenarios.
In this work, we focus on capturing full input- label mapping information from demonstrations. To achieve this target, we propose two tech- niques. The first is Repeated Demonstration. where we replicate later demonstrations and con- catenate them to the front. This allows the model to 'observe' the later information even under the causal restriction. For example, if we consider four demonstrations represented by d1 d2dad4, the input sequence after replication be- comes dzddlgc'' d d2d334. However, simply dupli- cating demonstrations brings about a new problem: we do not want to attend a demonstration twice, as this may cause the model to take shortcuts by learning to repeat the answer of its first encounter, rather than learning the input-label mapping. To address this, we propose the second technique, the Sliding Causal Attention, which customizes the original causal attention by restricting the atten- tion window so that each demonstration can only attend to all other demonstrations once. In the case of four demonstrations, attention windows are dndjd', d12,, d', d2,1,2d3,, and dj d2d3d4, re- spectively. Through experiments, we demonstrate that our proposed method (Repeated Demonstra- tions with Sliding Causal Attention, RDSCA) sig- nificantly enhances the ability learn input-label mapping from ICL. demonstrations.
We conduct experiments several text classi- fication datasets to evaluate the effectiveness of our proposed method. The experimental re- sults clearly demonstrate that our approach significantly enhances the input-label map- ping in ICL demonstrations.
2 Backgrounds
2.1
Causal Language Modeling
Most of the current decoder-only LLMs employ causal language modeling as the pre-training objec- tive (Radford et al., 2018), which aims to predict future tokens based on the past.
arg min Σ Po(u:/u0 Un u...... u4-1)
(1)
In causal language modeling, the model only at- tends to tokens that occur before (the left con- text), resulting in unidirectional attention scheme known as causal attention. This approach enables the model to process each token in the input quence in order, without accessing any information from the future (the right context).
Our proposed sliding casual attention the first attempt that customizes the causal attention in the inference stage without further training. We investi- gate a number of different designs for customizing causal attention and reach some principles for the success of ICL. For example, we find that the first <SOS> token plays an essential role. It should al- ways be available to attend to no matter where the attention window slides. Besides, the size of the attention window determines the richness of the semantic context, thus it affects the performance greatly. Our contributions are summarized as:
Causal Attention Mask Practically, a causal at- tention mask is used to implement causal atten- tion, which guarantees unidirectionality by mask- ing all right-to-left attention connections and only allowing right-to-left connections. Formally, the attention mask a binary-valued matrix M e {0, 1)mxn, where is the total sequence length. The element my in M indicates whether the j-th token in the sequence can attend to the i-th token, with a value of for yes and 0 for no. Therefore, the causal attention mask matrix is lower triangu- lar matrix where mij 0, Vi < j.
2.2 In-context Learning
To the best of our knowledge, we are the first to identify the limitation of causal language modeling in the ICL and to introduce a novel approach for enabling effective interactions
Now we formally introduce the definition and basic notations of in-context learning. We focus on clas- sification tasks and causal language models. Given

---

Regular ICL
RdSca ICL
<SOS> dj
d2 d
<SOS> d2 d 4
d
d2
0 shot
3 shot
shot
shot
2 shot
3 shot
3 shot
3 shot
4 shot
shot
attend other demos
attend to self
attend to the start of sequence
Figure I: An illustration for regular ICI. and our RoSCA ICI. Regular ICL uses vanilla causal attention, causing the uneven interaction between demonstrations. In RDSCA, each demonstration can attend all the others thanks to demonstration repetition and sliding causal attention.
K input-label pairs (I,, ) and a query zq. the objective is to predict the label of the query By by predicting the next token. Formally,
on input-label mappings to perform ICL.
Method
3.1 Defect of Traditional ICL
arg min P(yy) (21, 1i), (22,, V2), (xx,
3,c0
During few-shot ICL, we combine the labeled demonstrations with the query by concatenating them and feeding them into the LLM. The labeled demonstrations provide valuable in-context input- label information, which is included in the few-shot ICL input. As a result, few-shot ICL consistently achieves better performance compared to zero-shot ICL.
(2)
where C is the label set.
To perform ICL successfully, we transform the classification task into natural language gener- ation task by adding templates to query and demonstrations. Additionally, discrete labels are mapped to label words, such as "positive" and "neg- ative" " for sentiment classification. We denote the demonstration with templates and label words as d, T(*,, 91). The entire input to the model is formed by concatenating the demonstrations and the query q T( ,) resulting in dj d1d2...:.q.
However, few-shot ICL under the current scheme of causal language modeling has a de- fect. Due the restriction of causal attention, each demonstration only has access to half of the full context (i.e., the left context). As a result, can- not 'observe' demonstrations that follow it. For example, in 4-ohot ICL, if we consider the third demonstration as the last query (which makes sense because tokens that come after it have influence on it), then the first context while the fourth demonstration ignored, In this sense, we can regard predicting the label of the first demonstration as a zero-shot ICL task, the second as a one-shot ICL, the third as a two-shot ICL, and so on, as shown in Figure 1.
Semantically-umrelatee label ICL (SUL-ICL) In regular ICL, natural language words that are closely related to the task objective used as label words. This allows the model to utilize semantic prior knowledge. For example, when conducting sentiment classification, label words such as "posi- tive" and "negative" are highly semantically related to the sentiment labels of the input. During pre- training, the model is exposed to similar patterns and can learn to associate these label words with the corresponding sentiment labels. In this paper, we eliminate the contribution of semantic priors and perform semantically-unrelan label ICL (Wei et al., 2023). this setting, natural language la- bels are replaced with semantically-unrede bels. This approach forces the model to rely solely
demonstrations serve its
While restricting the model from accessing the right context makes when modeling regular natural language sequences prevent information leakage,
unnecessary for few-shot ICL since there no dependency between demonstrations. The causal restriction limits earlier demonstrations
la-

---

from accessing later information, resulting in only half of the available information being utilized.
repetitive context. Specifically, the attention win- dow is defined as follows:
3.2 Repeated Demonstrations
(xti-- x= d,, did2..dkg
i
K
window(())
Enabling each demonstration to 'observe" later demonstrations is not a trivial task. Simply replac- ing causal attention with full attention to allow the model to receive both left and right context is not feasible, as the model is pre- vre-trained with causal lan- guage modeling. Switching to full attention during inference would result in significant loss in per- formance, as have verified through experiments (Section 4.4). Therefore, we propose the Repeated Demonstration method to establish sufficient inter- actions among demonstrations while still adhering to the premise of causal attention. The method is based on a simple idea: duplicating all the other demonstrations except the first one and concatenat- ing them the front of the sequence. By doing so, we expand the input sequence di dıd2...dkg q to "2..d"=-d1g...ddx where dţ is the duplication of d. This operation allows each demonstration to attend to all the others by adding the later demon- strations to its left context. As a result, the model can make full use of demonstrations while still obeying the causal attention restriction.
x ==
(3)
Then we explain how sliding causal attention works. We use the window size W to represent the number of demonstrations contained within it. In our main setting, we use W K, indicating each attention window contains all the demonstrations.
As the window slides, new demonstration enters while an old one exits. This ensures that there are always consistent K different demonstrations within the context.
Additionally, we find that the <SOS> token, which represents the first token of the sequence, is crucial to the model's performance. Therefore, add <SOS> to every attention window to en- sure that each token can attend to <SOS>, We will provide further explanations on this in the experi- mental section 4.4.
Experiments
4.1 Setup
Models We conducted experiments on decoder- only causal language models. We utilized the LLAMA (Touvron et al., 2023) model family with varying scales, including 7B, 13B, 30B, and 65B.
3.3 Sliding Causal A Attention
The Repeated Demonstration method alone is not sufficient to achieve target, it introduces new problem of duplicated information in the context. Under causal attention, a token can at- tend to all tokens that precede it. When com- bined with repeated demonstrations, this can re- sult in the same information being duplicated in the context. For example, when performing 4- shot ICL. with repeated demonstrations, the in- put is dydgddd dd2ddd99. The context of d3 is dydgdd d2dy where some demonstrations appear twice. Repetitive information can cause the model learn shorteut predict the label by repeat- ing the label of the same demonstration that ap- peared for the first time, rather than learning the input-label mapping as expected. We will provide a detailed explanation of this phenomenon in the experimental section (Section 4.4).
Tasks We evaluate on classification tasks, includ- ing SST-2 (Socher et al., 2013). CB (De Marneffe et al., 2019), RTE (Dagan et al., 2005; Wang et al., 2019a), AGNews (Zhang et al., 2015), QQP (Wang et al., 2019b), and QNLI (Wang et al., 2019b). If the dataset includes validation split, we evaluate the model's performance the validation set. Oth- erwise, evaluate the test set. Datasets are obtained from Huggingface Datasets library".
Other Details For all experiments, we use K = 4 demonstrations by default. Demonstrations are uni- formly sampled from the training data. We utilize prompt templates from P Prompt.Source! (Bach et al., 2022). For each dataset, we use four different templates and select set of K training examples using different random seeds. Therefore, the re- ported results are the average of 16 different runs. We would like to emphasize that we run all meth- ods using the same random seeds, ensuring that
To tackle this issue, we propose Sliding Causal Attention, which utilizes a sliding attention win- dow to limit the range of tokens that each demon- stration can attend to. This approach effectively prevents the occurrence of repetitive information and ensures that each demonstration has a non-

---

4-shot regular ICL
4-shot Rd ICL
4-shot RdSca ICL
7-shot regular
7-shot ICL
7-shot RdSca
Random Demos, No Calibration
Demo Retrieval
Calibration
80
80
80
75
70
70
70
60
65
65
60
50
55
55
40
50
30B
13B
65B 30B 13B
7B
30B
13B
Figure 2: Averaging performance of RDSCA on LLAMA of different scales. Left: ICL with random-selected demonstrations. We report the results of both 4-shot and 7-shot ICL for this setting. Middle: We combine method with the demonstration retrieval technique. We employ KATE (Liu al., 2022) retrieve target demonstrations. Right: we combine with PROTOTYPICAL CALIBRATION (Han et al., 2022) to further improve performance. Full results can found in Appendix
the demonstrations used by each method are identi- cal. This was done to eliminate any potential bias caused by differences in the demonstrations seen by each method and ensure a fair comparison. Due to limited computing resources, we evaluate ran- dom sample of 200 examples for tasks with large validation set, instead of the entire dataset. We have found that this sample size is sufficient obtaín stable and reliable results.
on scales. Furthermore, calibration has a posi- tive influence on the ICL results, showing that our method can be combined with other techniques to further enhance the overall performance.
RDSCA improves input-label mapping We ob- serve significant performance boost over regular ICL in both 4-shot and 7-shot settings LLAMA- 30B and LLAMA-13B. Our method shows an av- erage improvement of 8.4% on LLAMA-30B and 10.5% on LLAMA-65B compared to regular ICL, indicating a significant advantage learning the input-label mapping from demonstrations. As men- tioned in previous sections, we believe that in the scheme of causal language modeling. regular ICL only utilizes half of the context (left context) for the demonstrations. In contrast, our method makes of all the available information, which enhances the representations of demonstrations and provides richer semantics for predicting the query's ' target.
4.2 Main Results
Figure 2 presents comparison between our method and regular ICL. We observe clear cor- relation between the performance of semantically unrelated label ICL and the model scale: increas- ing the model scale improves performance for all methods. In the case of LLAMA-7B, the perfor- mance is poor, and the performance gap between different methods is small, indicating that it is chal- lenging for small models learn the input-label mapping from demonstrations while the semantic meaning of labels is removed. As the model size scales up to 30B and 65B, the models begin to ex- hibit strong ICL capabilities, and the performance of different methods varies. This phenomenon is
4.3 Results on MMLU
REGULAR ICL
RDSCA
humanities
65,99
68.44
STEM social sciences
51.84
51.57
consistent with the findings of Wei et al. (2023), which suggest that larger language models learn the input-label mapping differently because the input- label mapping learning ability only emerges when the model is large enough. In comparison with ICL with random demonstrations, we find that retriev- ing demonstrations that are more relevant to the query can significantly improve the performance
72.29
73.60
other
63.34
64.65
average
62.20
63.66
Table I: LLAMA-65R 4-shot ICL results on MMLU.
We
also
validate
LLAMA-65B
on

---

<SOS> 'p' 'p dp d d2 d
Can we break the causal restriction? First, we are concerned about the limitations of causal tention. Intuitively, if we can remove the causal restriction during inference by using non-causal attention and providing the model with the full con- text, we can fully utilize the demonstration infor- mation. However, our experimental results indicate that this simple idea is not feasible. The perfor- mance of FUL.I ATTN. is better than random guessing, but why? There is simple explanation: the gap between causal attention training and non- causal attention inference too huge. Since the model is trained using causal language modeling, has no inductive bias toward data with the cor- rect context. Therefore, using full attention during inference is out-of -itto--distribution for causal model.
Figure 3: Illustration of the attention mask, which we divide into multiple regions and label with num- bers. The specific meanings are as follows: @ query-to- demonstrations attention; @ intos-demonstrattonn right- to-left attention; @ inae-deroonstraaionnsepppal right- to-left attention; 4 attention the start-of-sentence token: @ attention to redundant context; @ inter- demonstrations left-to-right attention;0) demonstrations- to-query attention:8 other left-to-right attention.
Sliding attention window matters. Based on the results, it is evident that RD has little improvement over regular ICL. Although RD provides richer interaction between demonstrations than regular ICL, the lack of attention restrictions causes the same demonstration attend to its first appear- ance. Consequently, the model takes a shortcut and repeats the answer of the first appearance in- stead learning the input-label mapping to predict the result. Therefore, the improvement brought by simply repeating the demonstrations is limited. However, the situation changes when we add the sliding causal window, which blocks repeated in- formation and forces the demonstrations attend only other unseen demonstrations, ensuring that there is no information leakage. According to the evaluation results, this approach successfully en- ables the model to better capture the input-label mapping information, expected.
MMLU (Hendrycks et al., 2021), which is more advanced benchmark for evaluating the abilities of LLMs. The results are shown in Table
4.4 Looking Deeper into Causal Attention Customization
In this section, we conduct a further investigation to look deeper into the effectiveness of sliding causal attention. How and why does sliding causal atten- tion work and are there alternatives when customiz- ing causal attention? To answer these questions, we implement the following ablations:
FULL ATTN. This uses full attention instead of causal attention so the model can attend to both the left and right context.
This ablation study further indicates another in- sight: the default causal attention may not be al- ways optimal in ICL. Previous research aimed at improving ICL has mainly focused on template construction and demonstration selection, with lit- tle attention paid to the causal attention masks. Our study shows that customizing attention masks be new technique for enhancing ICL, which is worthy of further research.
RD. This is the ablation of RDSCA that uses original causal attention without sliding causal attention.
RDSCA w/o <SOS>, In this ablation of RD- SCA, when the attention window moves for- ward, latter tokens cannot attend to the begin- ning of the sentence, iee., the <SOS> token.
Attending to the <SOS> token is essential. Next, examine the role of the first token <SOS> in causal attention customization. If not treating the <SOS> token separately, as the attention window moves, the first token slides out of the scope of the attention window. This means that the latter demon-
Figure shows how divide the whole attention matrix into various regions. The above-mentioned methods have unique access to these regions. De- tails and experimental results are shown in Table

---

Methods
Attention Regions Is causal?
LLAMA-65B
LLAMA-30B
RANDOM GUESS
13.06
43.06
REGULAR ICL
D+@
Yes
59.83
(+16.77)
52.71
(+9.65)
FULL ATTN. (v1)
@+2+6
No
45.15
(+2.09)
47.84 (+4.788)
FULL ATTN. (v2)
0+2+6+7
No
44.32
(+1.26)
47.36 (+4.30)
RD
0+2+0+00
Yes
60.18
(+17.112
54.02 (+10.96)
RDSCA w/o <SOS> @+2+0
Yes
32.20
(-10.86)
12.96 (-30.10)
RDSCA
1+
Yes
64.96 (+21.90)
56.55 (+ 13.49)
Table 2: Ablations on causal attention customization. We report the averaging performance on all tasks. We
implement two versions of the FULL ArTN., which differ in whether including left-to-right attention of the query.
strations and the query cannot attend to <SOS>, Our experiments show that in this setting the ICL ability is severely affected (0+0+G in Table 2).
Window
n-shots
LLAMA-65B
LLAMA-300 30B
REGULAR
0-1-2-3-4
69.87
63.10
W=1
040-0-0-4
60.83
52.78
However, when manually add the <SOS> to- ken to the attention window (as in the case of RD- SCA), we observe a significant improvement in performance. This comparison demonstrates that the <SOS> token is crucial for the model to per- form correctly. But why does such a seemingly insignificant token have such significant impact performance? We believe that the significance of the <SOS> token lies in allowing the model to treat the tokens under the attention window valid sequence rather than a fragment the whole sequence.
W=2
1-1-1-1-4
71.33
60.83
W=3
2-2-2-2-4
74.01
65.70
W=4
3-3-3-3-4
74.27
68.73
W=4
3-3-3-3-3
70.99
61.44
Table Average performance all tasks for differ- ent attention window sizes. We also report the num- demonstrations contained in the context each demonstration. For example, "0-1-2-3-4" means that, for regular ICL, the first demonstration has no context, the second sees one demonstration, the third sees two, and so on. In this experiment, use KATE to retrieve demonstrations instead of random-selected ones. this way, we save computational resources because there is no need to run experiments over different random seeds.
As the model is pre-trained on sequences that all start with <SOS>, it does not know how to handle sequences that do not start with <SOS>, There- fore, when customizing causal attention, make sure that the model regards the tokens in the attention window as a valid sequence starting with <SOS>. Only in this way, we can activate the model's language modeling ability correctly.
need
are K demonstrations in each window. In this case, each demonstration can attend to all other ones, so each demonstration can be considered as perform- ing (K 1 1)-shot ICL. As shown in Figure 4, we employ smaller window sizes W on RDSCA and what happens to the ICL ability. Intuitively, reducing the size of the sliding window means re- ducing the context (the number of demonstrations) that the current one is able to attend to. Thus the model's ability to learn the input-label mapping from context would be affected. Especially, when W = 1, each demonstration can only attend to itself, which is equivalent to zero-shot ICI. The results shown in Table 3. As expected, the window size is highly correlated with ICL perfor- the larger the window size, the better the performance. Surprisingly, we notice that when W 2, i.e., one-shot ICL for all demonstrations, RDSCA is already comparable with regular ICL on
W
W
W =
W 4
Figure 4: Visualization of attention masks for different
window sizes.
4.5 Ablation on Attention Window Size
mance:
this section, we discuss the impact of window size. We denote the attention window size of the original RDSCA as W K, meaning that there

---

both 30B and 65B LLAMA. This indicates that our ICL method makes more efficient exploitation of the demonstrations in ICL. The last row of Table shows when the query not seeing all 4 demonstra- tions, the performance drops.
mapping, which leads decline in ICL perfor- mance. Second, due to the repetition of demonstra- tions, the context becomes too long when adding too many demonstrations, and some studies have shown that LLMs still have difficulty modeling long sequences. Our method can effectively solve these problems by customizing attention. There- fore, we believe that our exploration of customizing causal attention highlights possible solution for tackling the long sequence modeling issue, which worth further research in the future.
4.6
What If Adding More Demonstrations?
90
4.7 Discussions on Efficiency Issues
Performance
80
As repeating demonstrations involves expanding the sequence lengths, the model is required to pro- cess more tokens when compared with regular ICL, which may lead to computational inefficiency and extra memory consumption. However, we argue that these efficiency issues can be neglected practi- cally. Thanks to the auto-regressive nature of Large Language Models LLMs, the representations of previous tokens are not dependent on later tokens. Therefore, in the inference phase, the representa- tions of demonstrations stored offline, thereby mitigating the added com- putational burden. Moreover, our adaptation the attention mask allows the LLM focus only on tokens within the sliding window for key-value (KV) caching, rather than retaining information on all previous tokens. This optimization reduces the memory requirements of RdSca. Consequently, the memory consumption of RdSca remains on par with that of standard ICL setup.
70
rdsca
60
regular
4
8
16
32
Number of demonstrations
Figure 5: Increasing the number of demonstrations for LLAMA-65B on SST-2.
be pre-computed and
In this section, we investigate how our method performs with more demonstrations. We start from K demonstrations and gradually add more. As shown in Figure 5, all methods receive significant performance gains when the number of demonstra- tions is relatively small. However, as the number of demonstrations increases, the benefits of adding more demonstrations gradually reduce. The ICL performance reaches plateau at around K 16. Before reaching this plateau, RDSCA consistently outperforms the vanilla-repeat and regular ICI., in- dicating that our method robust to the increase the number of demonstrations. However, this advantage gradually diminishes when adding more demonstrations. All methods seem to perform com- parably at around K = 16, suggesting that the ability to learn from context is close to its upper bound this point.
Related Works
5.1 In-context Learning
In-context learning (ICL) is an effective approach for adapting pre-trained LLMs downstream tasks (Brown et al., 2020). This is achieved by adding task-specific templates and demonstrations before the test query, without updating model pa- rameters. Recent works focus on enhancing ICL with various techniques. For example, Liu et al. (2022); Rubin et al. (2022); Ram et al. (2023); Luo et al. (2023) propose that choosing demonstrations more carefully results in better ICL performance. Some studies try improving ICL with more sophis- ticated prompt templates, either by hand-craft by automation (Gonen et al., 2022). Some pro- pose Mhain-of-thoughee (CoT) to elicit the reason- ing abilities of LLMs by augmenting each demon-
In addition, we observe an interesting phe- nomenon in the experiment. We find that the perfor- mance of RD sharply declines when there are too many demonstrations (decreasing by 12.02 points from K 24 to 30, and 25.24 points from K = 30 10.). while RDSCA does not suffer such huge losses. We explain this phenomenon from two aspects: First, as mentioned earlier, not customiz- ing the causal attention, the model learns short- cuts from repeated examples rather than lnput-label

---

stration with chain of reasoning steps (Wei et al.. 2022). Subsequent studies have continued build upon this approach, achieving further improve- ments (Wang et al., 2022). Some studies explore the mechanism of ICI. Min et al. (2022) shows that randomly replacing labels in the demonstrations barely hurts performance ground truth demon- strations are not required. (Wei et al., 2023) claims that the success of ICL relies on both semantic prior knowledge and input-label mappings. They find that larger models can better capture input-label mappings from demonstrations.
Limitations
Although ROSCA has demonstrated outstanding performance in capturing the full input-label map- ping from demonstrations, there are some limi- tations to our work in this section that must acknowledge. First, our method expands the sequence length by duplicating demonstrations, which leads to increased computation costs and inference latency. This may limit its practicality computation-constriined scenarios. Moreover, we have not evaluated our method on more complex tasks, nor have we determined its performance with chain-of thought scenarios. Therefore, in the fu- ture, we need to investigate how it can be extended to a broader range of task scenarios.
5.2 Attention Customization
There has been a number of work done on at- tention customization since Transformer was pro- posed (Vaswani et al., 2017). many of them fo- cus on modeling long sequences (Kitaev et al., 2020; Beltagy al., 2020) or improving efficiency through sparse attention (Tay et al., 2023). Some works have explored customizing attention to block certain dependencies. For example, Mu et al. (2023) prevents later tokens from attending to the prompt to compress context. These attention cus- tomizations can be viewed as some kind of modi- fications the model architecture, which requires additional training. However, our study is the first investigate the possibility of attention customiza- tion during inference without further training.
A Acknowledgments
This work is supported by National Key R&D Pro- gram of China (No. (FC333334))) and Na- tional Natural Science Foundation of China (NSFC Grant No. 62122089). Jingang Wang is funded by Beijing Nova Program (Grant No. 20220484098), We sincerely thank all reviewers for their valuable comments and suggestions, which are crucial for improving our work.
References
Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Al- bert Webson, Colin Raffel, Nihal V. Nayak, Ab- heesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Févry, Zaid Alyafeai. Manan Dey, An- drea Santilli, Zhiqing Sun, Srulik Ben-David, Can- wen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged Saced AlShaibani, Shanya Sharma, Ur- mish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir R. Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. Promptsource: An in- tegrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics, ACL 2022 System Demon- strutions, Dublin, Ireland, May 22-27, 2022, pages 93-104. Association for Computational Linguistics.
6 Conclusion
this study, we introduce RDSCA, a novel ICL framework that enhances the learning of the input- label mapping from demonstrations. We propose that the causal attention mechanism of decoder- only LLMs restricts the model from fully exploiting the input-label mapping from demonstrations. To address this, we suggest repeating demonstrations allow each demonstration to have full context and customizing the vanilla causal attention mecha- nism to prevent information leakage. Experimental results show that our method consistently improves the labet mapping ability of ICL on LLMs of different scales. Furthermore, we delve deeper into causal attention customization and show how different attention settings affect ICL performance. Additionally, this work the first to customize the causal attention of a pre-trained autoregressive LLM without further training, which may pave the way for further research in this direction.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric

---

Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11048-11064, Association for Computational Linguisties.
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman. 2023. Learning to compress prompts with gist to- kens. CoRR, 0os/2304.08467.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges, Eval- uating Predictive Uncertainty, Visual Object Classi- fication and Recognizing Textual Entailment, First PASCAL. Machine Learning Challenges Workshop. MLCW 2005, Southampton, UK. April 1 1-13, 2005, Revised Selected Papers, volume 3944 of Lecture Notes in Computer Science, pages 177-190. Springer.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval- retrieval-ugmeented lan- guage models. CoRR, abs/2302.00088.
Catre-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commimenntannk: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle. WA, United States, July 10-15, 2022, pages 2655-2671. Association for Computational Linguistics.
Hila Gonen. Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2022. Demystifying prompts in language models via perplexity estimation. CoRR, abs/2212.04037..
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng. and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. EMNLP 2013, 18-21 October 2013, Grand Hyast Seattle, Seattle, Washington, USA, A meeting of SIG- DAT, a Special Interest Group of the ACL, pages 1631-1642. ACL.
Zhixiong Ham, Yaru Hao, Li Dong, and Furu Wei. 2022. Prototypical calibration for shot learning of lan- guage models. CoRR, 22t/222550018.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2023. Efficient transformers: A survey, ACM Comput. Sure., 55(6): 109::
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020. OpenReview.net.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CORR. abs/2302.1397
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in io-context examples for gpt-3? In Pro- ceedings of Deep Learning Inside Out: The 3rd Work- shop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ ACL 2022, Dublin, Ireland and Online, May 27, 2022, pages 100-114, Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008.
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasu- pat, Seyed Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y. Zhao. 2023. Drich: Demonstratioon-retivvvdd in-context learning. CoRR, 14//2305.14188.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stick- ier benchmark for general-purpose purpose language under- standing systems. In Advances in Neural Information
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi. and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations:

---

Processing Systems 32: Anmual Conference on Neu- ral Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis plat- 7th In-
form for natural language understanding. ternational Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Xuezhi Wang, Jason Wei, Dale Schuurmans. Quoc Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 11171.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. 2022, Chain-of-thoone ofotthtgtt prompt- ing elicits reasoning in large language models. In NeurIPS.
Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Al- bert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang. Denny Zhou, and Tengyu Ma. 2023. Larger language models do co-context learning dif- ferently. CORR, abs/2303.03846.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems 28: Annual Conference Neural In- formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657 657.

---

A
Full results
Methods
65B
30B
13B
7B
REGULAR ICL
$9.82
55.18
47.79
49.72
Methods
65B
30B
13B
7B
59.91
49.68
49.34
RDSCA
75.76
69.32
53.03
50.92
REGULAR ICL
59.38
53.45
47.84
46.56
RD
58.27
52.28
47.77
47.63
with demonstration retrieval
RDSCA
60.51
54.49
45.33
47.29
REGULAR ICL
74.75
65.62
58.00
56.88
RD
77.13
73.63
56.38
59.25
with demonstration retrieval
RDSCA
87.38
82.13 68.88
64.63
REGULAR ICL.
63.21
60.90
51.79
52.56
62.03
65.67
51.66
53.14
RDSCA
64.43 66.15 51.27
52.98
Table 4: Full evaluation results on SST-2
Table Full evaluation results on QNLI
Methods
65B
30B
13B
7B
REGULAR ICL
76.79
58.15
44.98
50.67
RD
76.12
56.49
42.94
48.34
RDSCA
76.90
56.17
46.97
50.66
with demonstration retrieval
REGULAR ICL
78.12
64.73
45.54
57.59
RD
79.72
60.38
49.06
59.91
RDSCA
79.72 60.38 52.83
57.08
Table 5: Full evaluation results CB
Methods
65B
30B
13B
7B
REGULAR ICL
36.97
36.33
28.19
RD
37.86
36.49
30.31
27.64
Methods
65B
30B
13B
7B
RDSCA
43.85
40.09
31.12
30.89
REGULAR ICL
53.67
49.13
61.50
59.62
with demonstration retrieval
RD
53.20
53.37
62.54
57.96
REGULAR ICL
70.75
70.38
67.12
67.50
RDSCA
59.68
53.44
60.05
57.23
RD
74.63
76.12
72.00
74.12
RoSCa
75.13 76.25 70.12
70.75
with demonstration netrieval
REGULAR ICL
60.17
53.85
67.94
63.47
61.79
62.95
70.38
64.20
Table 8: Full evaluation results AG-News
RDSCA
64.87 61.54 66.79
64.36
Table 6: Full evaluation results on QQP
Methods
30B
13B
REGULAR ICL.
72.36
63.99
55.4
53.76
Ro
72.0
65.56
55.4
55.6
RoSca
73.07
65.8
57.07
53.95
with demonstration retrieval
REGULAR ICL
72.25
63.12
56.75
52.75
RD
72.39
65.27
55.7
54,66
RDSCA
74.07
65.92
56.74
53.23
Table 9: Full evaluation results on RTE


Reviews:
[{"review_id": "iBhFgEMyJF", "reviewer": "Reviewer_n9jP", "summary": "", "questions": "I'd like to ask whether this is similar to studying whether the order in which examples are listed in the ICL affects the final result.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and effort in reviewing our paper. \n\n#### **Regarding expanded sequence length**\n> The sequence length will be expanded by using repeated demonstration.\n\nThank you for your insightful question. We acknowledge that expanding the sequence length can lead to additional computational and memory expenses, especially during training. However, in the inference phase, the representations of demonstrations can be pre-computed and stored offline, taking advantage of the auto-regressive nature of Large Language Models (LLMs), thereby mitigating the added computational burden. Moreover, our adaptation of the attention mask allows the LLM to focus only on tokens within the sliding window for key-value (KV) caching, rather than retaining information on all previous tokens. This optimization reduces the memory requirements of RdSca. Consequently, the memory consumption of RdSca remains on par with that of a standard ICL setup.\n\n#### **Answer to the questions**\n> I'd like to ask whether this is similar to studying whether the order in which examples are listed in the ICL affects the final result.\n\nThank you for your great question. Yes, we agree that there is a connection between our study and the study of order sensitivity of ICL demonstrations. Their research has shown that changing the order of demonstrations can greatly impact ICL performance. We hypothesize that this high sensitivity is due to the nonsymmetric property of causal language modeling, which only attends backwardly and causes uneven emphases in all demonstrations, with later demonstrations receiving more information. Therefore, changing the order can cause significant performance perturbations.\n\nFrom this perspective, our RdSca method should be less sensitive to order, as all demonstrations receive an equal amount of context information. In the revision, we will perform an analysis of the order sensitivity and provide further insights. Thank you again for your insightful suggestions", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The limitation of causal language modeling in the ICL proposed in the paper is thought-provoking. \n2. Although the repeated demonstration with sliding causal attention is simple, the results show in Figure 2 demostrate its effectiveness.\n3. The experiments and demonstrations in this paper are relatively complete and conform to the verification of the conclusions.", "reasons_to_reject": "The sequence length will be expanded by using repeated demonstration. ", "questions_to_authors": "I'd like to ask whether this is similar to studying whether the order in which examples are listed in the ICL affects the final result.", "paper_topic_and_main_contributions": "This paper identifies the limitation of causal language modeling in the in-context learning and proposes a novel ICL method, called Repeated Demonstration with Sliding Causal Attention. Experiments on several text classification datasets shows that its method consistently improves the input-label mapping ability of ICL on LLMs of different scales.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "STmqfRDyvr", "reviewer": "Reviewer_evij", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback! We discuss your raised points as follows:\n\n#### **Regarding extra inference cost**\n> The proposed techniques inherently require a significantly higher cost to conduct in-context learning, yet the paper does not address this inefficiency, concentrating only on (potentially marginal) improvements in the final benchmark scores.\n\nThank you for the question. We acknowledge that expanding the sequence length may introduce extra computation and memory costs. However, we'd like to highlight how these inefficiencies are mitigated during inference:\n- In real-world, large-scale ICL inference, the representations of demonstrations can be pre-computed and stored offline, leveraging the auto-regressive nature of LLMs. Consequently, when performing ICL for a specific query, there is no need to re-compute the key-value (KV) state for previous tokens. This approach ensures that inference latency remains as low as in regular ICL.\n- From a memory consumption standpoint, 4-shot RdSca is as efficient as 4-shot regular ICL, and significantly more memory-efficient than 7-shot regular ICL. This is due to RdSca's ability to track the status of a maximum of 4 demonstrations for KV-caching, thanks to the sliding window mechanism.\n\nWe will expand upon the points mentioned above and provide additional results of training and inference time during revision. Thank you again for the great suggestion.\n\n\n#### **Regarding semantically-unrelated label ICL**\n> Additionally, the authors focused their testing on the setting of \"semantically-unrelated label ICL\", which is uncommon in practice. As a result, the practical impact of the proposed method in the real world becomes questionable.\n\nThanks for this insightful feedback. We’d like to provide some clarifications here. Our choice to evaluate our method in the semantically-unrelated label ICL setting was not driven by the performance of our method in the standard ICL setting. Instead, it was motivated by our goal of enhancing the input-label mapping within ICL. Assessing our approach in the semantically-unrelated label ICL setting aligns with this motivation as it relies solely on the input-label mapping to conduct ICL, without the assistance of semantic priors, as explained in section 2.2.\n\nTo prevent any potential misunderstandings, we have also conducted experiments in the standard ICL setting on the original MMLU dataset, and the advantages of RdSca over regular ICL are clearly evident.\n\n| 4-shot ICL | average | humanities | STEM  | social sciences | other |\n|-|-|-|-|-|-|\n| LLAMA-65B (Regular) | 63.47   | 68.34      | 54.29 | 72.44           | 64.58 |\n| LLAMA-65B (RdSca)  | 69.36   | 76.18      | 54.67 | 75.35           | 68.20 |\n| LLAMA-30B (Regular) | 58.59   | 60.46      | 48.90 | 67.59           | 62.67 |\n| LLAMA-30B (RdSca)  | 64.75   | 74.32      | 53.00 | 65.66           | 65.00 |\n\nWe will supplement the above evaluation results on the standard ICL setting with more specific clarifications in the revision.\n\n#### **Regarding why a specific combination of mask snippets is particularly effective**\n> Finally, the paper would benefit from a clear explanation as to why a specific combination of mask snippets is particularly effective, while most similar ones are not.\n\nWe ablate the design of the attention mask in section 4.3, from which we draw some conclusions on why the current design works while its ablations fail. Generally, we attribute the success of sliding causal attention to three factors: preventing information leakage with the sliding window, maintaining the causal constraint of auto-regressively-trained LLMs, and paying additional attention to the <SOS> token while the window slides away. We will provide a clearer explanation of these factors and our findings during revision.\n\nThanks again for your careful review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed techniques are simple, easy to implement, and intuitive.\n- Attempted to evaluate different variations of attention masks to investigate the role of the masks for in-context learning.\n", "reasons_to_reject": "- The proposed techniques inherently require a significantly higher cost to conduct in-context learning, yet the paper does not address this inefficiency, concentrating only on (potentially marginal) improvements in the final benchmark scores. \n- Additionally, the authors focused their testing on the setting of \"semantically-unrelated label ICL,\" which is uncommon in practice. As a result, the practical impact of the proposed method in the real world becomes questionable. \n- Finally, the paper would benefit from a clear explanation as to why a specific combination of mask snippets is particularly effective, while most similar ones are not.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper emphasizes the need for proper contextualization of demonstrations in in-context learning, a task that was originally limited by the causal nature of autoregressive language models. To accomplish this goal, the authors propose two methods: \"Repeated Demonstrations\" and \"Sliding Causal Attention.\" The first approach involves duplicating demonstrations twice, enabling the former demonstrations to reflect the content of the latter ones. Additionally, the second technique modifies attention masks to prevent an over-reliance on duplicated demonstrations. In their research, the authors explored various attention mask variations, providing insight into how these masks inherently function to facilitate in-context learning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "exZrWVRKCr", "reviewer": "Reviewer_Zcd4", "summary": "", "questions": "A: Is it possible to provide additional explanations regarding the specific setup of experiments, such as the verbalizer?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and dedication in reviewing our paper. In the following, we carefully address each of your concerns point by point.\n\n#### **Answer to the questions**\n> A: Is it possible to provide additional explanations regarding the specific setup of experiments, such as the verbalizer?\n\nThanks for your suggestion. For the verbalizer, we use semantically-unrelated label words such as 'A', 'B', 'C', and 'D'. For more specific information, please refer to the supplementary materials where we have uploaded the detailed implementation code. We will also include more setup details in the appendix as suggested.\n\n#### **Regarding providing standard deviations would provide more insight**\n> Additionally, providing standard deviations would provide more insight.\n\nThank you for the great suggestion. We agree that more detailed experiments are helpful and we present the standard deviations for the main experiment below. These results are averaged across six tasks, with the standard deviation computed across 16 runs for each task.\"\n| std         | 7b   | 13b  | 30b  | 65b  |\n|-------------|------|------|------|------|\n| regular ICL | 1.75 | 0.71 | 2.54 | 3.28 |\n| Rd ICL      | 1.08 | 1.19 | 2.29 | 3.74 |\n| RdSca ICL   | 1.17 | 0.93 | 2.10 | 3.84 |\n\nWe will supplement these results in our revision.\n\n#### **Regarding reducing redundant content and providing more analytical experiment**\n> Overall, there is quite a bit of repetition. Reducing redundant content and incorporating more engaging and analytical experimental results would greatly improve the paper.\n\nWe appreciate your suggestions and will revise accordingly. Specifically, we will conduct a comprehensive review of our paper to eliminate any redundant content. Furthermore, we plan to incorporate additional analytical experimental results, including evaluations on more advanced benchmarks like MMLU, and an in-depth analysis of sensitivity to demonstration order. If you have any more specific suggestions or recommendations, we would greatly appreciate your input to further enhance the quality of our revision.\n\n#### **Regarding more explanation for the last row of T2**\n> Providing a bit more explanation for the last row of T2 would be beneficial. While it is not entirely incomprehensible, it could be made more intuitive for better understanding.\n\nThanks for pointing it out. The difference between the last two rows of Table 2 is the number of demonstrations seen by the query. \"3-3-3-3-4\" means that the query sees all 4 demonstrations while \"3-3-3-3-4\" means that the query sees the last three. We will provide more explanations and supplement an illustration in Figure 4 to improve the representation in the revision.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The main motivation of this paper is to overcome the structural limitations of CLMs in ICL.\n- To overcome the unidirectional constraints of CLMs, the paper propose RDSCA, which duplicates later demonstrations and concatenates them to the front. Moreover, apply sliding causal attention to prevent repetitive information from occurring.\n- The effectiveness of the proposed methodology is demonstrated through various experiments.\n- Additionally, the paper performs a thorough analysis of the method's specific elements, such as the importance of the sos token and W-size.", "reasons_to_reject": "- The coherence between the hypothesis and methodology is positive.\n- There is a logical analysis of the hypothesis and its relation to the methodology.\n- The well-structured figures significantly aid in understanding the proposed method.", "questions_to_authors": "A: Is it possible to provide additional explanations regarding the specific setup of experiments, such as the verbalizer?", "paper_topic_and_main_contributions": "- The main motivation of this paper is to overcome the structural limitations of CLMs in ICL.\n- To overcome the unidirectional constraints of CLMs, the paper propose RDSCA, which duplicates later demonstrations and concatenates them to the front. Moreover, apply sliding causal attention to prevent repetitive information from occurring.\n- The effectiveness of the proposed methodology is demonstrated through various experiments.\n- Additionally, the paper performs a thorough analysis of the method's specific elements, such as the importance of the sos token and W-size.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It would be helpful if there were detailed experiment specifics in the appendix. Additionally, providing standard deviations would provide more insight.\n- Overall, there is quite a bit of repetition. Reducing redundant content and incorporating more engaging and analytical experimental results would greatly improve the paper.\n- Providing a bit more explanation for the last row of T2 would be beneficial. While it is not entirely incomprehensible, it could be made more intuitive for better understanding."}}, {"review_id": "OPkvCwyWTH", "reviewer": "Reviewer_6837", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions. They are exceedingly helpful in improving our paper.\n\n#### **Regarding \"4-shots RdSca ICL cannot outperform regular 7-shots, which suggests that using regular ICL might still be the best choice with the same amount of computation budget\"**\n\nThanks for this insightful question. We agree with the reviewer that regular ICL with more demonstrations can outperform RdSca with fewer demonstrations. In the following, we provide several explanations to further clarify this.\n\n- It's a common observation that ICL tends to exhibit improved performance as more demonstrations are added, whether in the case of regular ICL or RdSca ICL. However, the primary advantage of RdSca lies in its ability to maximize the utility of a fixed number of ICL demonstrations. This capability becomes particularly valuable when dealing with situations where data is extremely limited. MMLU serves as an illustrative example, given that it contains only 5 demonstrations in the development set. Consequently, attempting a 7-shot ICL approach on MMLU is impractical.\n- Regarding the additional computational resources required by RdSca, we assert that these demands are negligible since LLM inference tends to be limited by memory-IO rather than compute power [1]. Furthermore, in practical, large-scale ICL inference scenarios, it's possible to pre-compute and store the representations of demonstrations offline, leveraging the auto-regressive nature of LLMs. Therefore, the incremental computation overhead is not a significant concern.\n- From a memory consumption perspective, 4-shot RdSca is as efficient as 4-shot regular ICL, and far less consumptive than 7-shot regular ICL because RdSca only needs to track the status of at most 4 demonstrations for KV-caching thanks to the sliding window mechanism.\n\nWe will provide further clarification during revision.\n\n[1] Shazeer N. Fast transformer decoding: One write-head is all you need[J]. arXiv preprint arXiv:1911.02150, 2019.\n\n#### **Regarding more advanced benchmark**\n> The selected benchmark seems a bit outdated. Just like people use MMLU/BigBench to evaluate LLM instead of SST/CB, Such LLM-based ICL approaches need to be validated on MMLU/BigBench to be more convincing.\n\nWe appreciate your suggestion of evaluating LLM-based ICL approaches on a more convincing benchmark. We present evaluation results on MMLU below.\n\n| 4-shot ICL         | average | humanities | STEM  | social sciences | other |\n|--------------------|---------|------------|-------|-----------------|-------|\n| LLAMA-65B (Regular) | 63.47   | 68.34      | 54.29 | 72.44           | 64.58 |\n| LLAMA-65B (RdSca)  | 69.36   | 76.18      | 54.67 | 75.35           | 68.20 |\n| LLAMA-30B (Regular) | 58.59   | 60.46      | 48.90 | 67.59           | 62.67 |\n| LLAMA-30B (RdSca)  | 64.75   | 74.32      | 53.00 | 65.66           | 65.00 |\n\nThe evaluation results on MMLU clearly demonstrate the advantage of our method over the regular ICL setting. In the revision, we will provide a more comprehensive comparison of different LLMs, numbers of demonstrations, and masking strategies on MMLU/BigBench.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear, and the proposed approach is intuitive.\n2. Significant improvement is demonstrated on used benchmarks.", "reasons_to_reject": "1. On the one hand, the work shows an interesting result: you can boost performance by duplicating 4-shots and manipulating the attention. On the other hand, 4-shots RdSca ICL cannot outperform regular 7-shots, according to Figure 2, which suggests that using regular ICL might still be the best choice with the same amount of computation budget.\n2. The selected benchmark seems a bit outdated. Just like people use MMLU/BigBench to evaluate LLM instead of SST/CB, Such LLM-based ICL approaches need to be validated on MMLU/BigBench to be more convincing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "After rebuttal: The authors effectively addressed my comments. \n\n------------------\n\nThis paper proposes a new mechanism to improve in-context learning (ICL) by adding repeat demonstrations and sliding window attention, to enable each demonstration to attend to all the others and eliminate sequential dependencies.\nThis work uses LLaMA (7B ~ 65B) models and validates the proposed approach on some \"classic\" text classification datasets (SST-2, CB, RTE...).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper argues that causal attention within in-context demos of limits the ICL ability of language models, and the authors propose a simple technique to prepend the demos to the original demos (i.e. demonstration replay) so that each demo example can observe the information of later ones. A sliding window approach is applied to avoid duplicated information in the attention. Experiments on several traditional classification datasets demonstrate the effectiveness of the proposed approach. The pros and cons summarized by the reviewers and the AC are listed below.

### Pros
1. "The limitation of causal language modeling in the ICL proposed in the paper is thought-provoking."
2. "The proposed techniques are simple, easy to implement, and intuitive."
3. "The effectiveness of the proposed methodology is demonstrated through various experiments."

### Cons
> 1. "The sequence length will be expanded by using repeated demonstration."  

This point has been raised by multiple reviewers. While the authors have responded in the rebuttal that the KV cache could be pre-computed, storing these precomputed caches still incurs extra cost (maybe relatively minor). I agree with the authors in the rebuttal that the time cost is not increasing with a sliding window approach given that the KV cache is precomputed.  

> 2. "On the other hand, 4-shots RdSca ICL cannot outperform regular 7-shots, according to Figure 2, which suggests that using regular ICL might still be the best choice with the same amount of computation budget."

This point is concerning. I do agree with the authors that 4-shot RdSca ICL should be more efficient than regular 7-shots given precomputation. However, a detailed analysis of memory and time efficiency of RdSca ICL is necessary since multiple reviewers are worried about its efficiency -- the 4-shot RdSca ICL is not equivalent on efficiency to regular 4-shots because the first demo of RdSca still attends the other 3 demos while the regular one does not.

> 3. "The selected benchmark seems a bit outdated. Just like people use MMLU/BigBench to evaluate LLM instead of SST/CB, Such LLM-based ICL approaches need to be validated on MMLU/BigBench to be more convincing."

In summary, all the reviewers agree that this paper is sound, but they did not find this paper exciting enough with some concerns as above and results on slightly outdated benchmarks.
Decision: Accept-Findings