StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses
Authors: Jia-Nan Li
Quan Tu
Cunli Mao
Zhengtao Yu
Ji-Rong Wen
Rui Yan
Venue: NeurIPS 2024 poster (2024)
Keywords: dialogue compression
conversational attention sinks
memory
Primary area: natural_language_processing
TL;DR: StreamingDialogue efficiently compresses dialogue history into conversational attention sinks with minimal losses, enhancing the model's long-term memory and facilitating prolonged streaming conversations.

Abstract: Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our observation, dialogue contexts are highly structured, and the special token of End-of-Utterance (EoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as ``conversational attention sinks'' (conv-attn sinks). Accordingly, we introduce StreamingDialogue, which compresses long dialogue history into conv-attn sinks with minimal losses, and thus reduces computational complexity quadratically with the number of sinks (i.e., the number of utterances). Current LLMs already demonstrate the ability to handle long context window, e.g., a window size of 200K or more. To this end, by compressing utterances into EoUs, our method has the potential to handle more than 200K of utterances, resulting in a prolonged dialogue learning. In order to minimize information losses from reconstruction after compression, we design two learning strategies of short-memory reconstruction (SMR) and long-memory reactivation (LMR). Our method outperforms strong baselines in dialogue tasks and achieves a 4 $\times$ speedup while reducing memory usage by 18 $\times$ compared to dense attention recomputation.

StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses
Jia-Nan Li
Quan Tu
Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan
Gaoling School of Artificial Intelligence, Renmin University of China 2 Kunming University of Science and Technology flijianan, quantu, jrwen, ruiyan)&ruc. edu.. en maocun1i0163. com, ztyu@hotmail. com
Abstract
Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our obser- vation, dialogue contexts are highly structured, and the special token of End-of- Unterance (BoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as "conversational attention sinks" (conv-attn sinks). Accord- ingly, we introduce StreamingDDallguee into conv-attn sinks with minimal losses, and thus reduces computational com- plexity quadratically with the number of sinks (i.e., the number of utterances). Current LLMs already demonstrate the ability to handle long context window, e.g., window size of 200K more. To this end, by compressing utterances into EoUs, our method has the potential to handle more than 200K of utterances, resulting in prolonged dialogue learning. In order to minimize information losses from reconstruction after compression, we design two learning strategies of short- memory reconstruction (SMR) and long-memon reactivation (LMR). Our method outperforms strong baselines in dialogue tasks and achieves a 4 x speedup while reducing memory usage by x compared to dense attention recomputation
which compresses long dialogue history
Introduction
Layer Head
Laysr ta
Layse <ho
Layer
Laper
A4
Llama-2-7B
(a)
Lllme-2-7B-CC
Stroomingi. L.M
(b)
StrsminggDolegee (ours)
Attention map visualization. (a) Llama-2--2- M-ama-2--B/Chal with "
" and "In" as EoU ("
" counts as one token, "'n" two). (b) StreamingI versus StreamingDDaloguee attention on Llama- 2-7B with "
" as EoU.
Figure 1:
Large Language Models (LLMs)) 115
rapidly advancing. However, their performance is constrained by context size during pre-training For example, with a context size 4,096, the inference capability of LLaMA2 6 sharply drops when the context length exceeds the preset limit. Moreover, the attention mechanism leads to quadratic growth in computational complexity with
'The first two authors contributed equally. "Corresponding authors: Rui Yan (nuiyan (riiyun@mcc.duucc)) and Zhengtao Yu (ztyu@ hotmail.com).
1 'Code: https://githhbe http://tttt..ccoo/aaaaeeee//eaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

text length, increasing GPU memory usage and slowing generation speed. As LLMs find widespread use in various conversational applications [8-TO, these limitations become particularly severe for dialogue tasks [[ rendering standard LLMSs infeasible for supporting prolonged dialogues with long conversation histories.
In order to support conversations with long contexts, natural solution is to reduce the computation of inter-token correlations by modifying the implementation of attention. Beltagy et al. [14] proposed local attention, which confines attention within a k window size, reducing computational complexity to linear. However, when the text length exceeds k, the generation performance substantially declines. StreamingLLM [TS] enhanced long context streaming by introducing the concept of "attention
sinks." This approach builds on local attention, allowing initial tokens be consistently attended to, which supports stable long-term interactions and efficient generation. However, Streaming continuously updates the cached information within the fixed-size window, excluding initial tokens. These initial tokens are blind to subsequent tokens during the auto-regressive generation process. Consequently, as the dialogue context lengthens, historical information is progressively lost, which is detrimental to dialogue consistency and severely impacts the user experience.
We introduce StreamingDialogue, a method designed for efficient conversations with enhanced long- term memory capabilities. In dialogue contexts, we observe an interesting phenomenon: tokens used to separate utterances (namely Of-tteracee EoU). such as "
" and "An," tend to aggregate more attention than other tokens (for details, refer to Figure I (a) and see $4.11 for further analysis). We refer to these separator tokens as "conversational attention sinks" (conv-attn sinks). Figure D (b) demonstrates that, in contrast to the highly dispersed attention pattern of StreamingI.IMM. StreamingDialogue maintains focus on critical positions like conv-attn sinks, thereby utilizing them to aggregate utterance information, compressing lengthy dialogues to only require caching conv-attn sinks' key-values to improve efficiency and reduce memory consumption.
Specifically, in the long-term generation, we preserve conv-attn sinks to memorize historical dialogues for retrieval. Additionally, caching both the first token and the previous and current utterances is crucial to ensure stable output beyond a certain inference length and to facilitate the smooth generation of consecutive replies. Beyond these measures, we introduce two self-learning strategies to better characterize the conv-attn sinks: (I) we devise a reconstruction task, where the reconstruction process can only attend to the conv-attn sink of the target utterance, thereby encouraging the conv-attn sink to restore information from the target sentence, namely short-memory reconstruction (SMR); (2) we propose a recall task, treating the final utterance as a query and attending solely to conv-attn sinks in the dialogue history to retrieve the matching response, thus prompting the model to reactivate information from lengthy dialogues, named as long-memory reactivation (LMR). These two tasks will be jointly optimized before dialogue learning.
Experiments on widely-used dialogue datasets demonstrate that our proposed method outperforms other sparse attention and memory- enhancceennannementt methods (in terms of evaluation metrics of Perplexity.
BLEU, ROUGE, Distinet, USL-H, and Dial-M). In terms of efficiency, our method achieves a 4 x speedup and an 18 x reduction in memory usage compared to dense attention with recomputation. In particular, currently some LLMs support handling long contexts, such as Claude 2. f. with a 200K context window. In this way, leveraging our method with such long context LLMs enables the completion of numerous utterances within the conversation session, which indicates one big step towards prolonged dialogue learning with long contexts. In summary, our main contributions are as follows:
(1) We discover that EoU tokens have the potential to aggregate utterance information. By defining these separator tokens as "conv-attn sinks." " we propose StreamingDialogue, which efficiently handles long context by only caching the first token, conv-attn sinks, and tokens from the most recent two utterances.
(2) We propose two learning strategies: short-memory reconstruction (SMR) and mnngmmeoory reactivation (LMR), enhancing the capability of conv-attn sinks to aggregate information and the ability to store historical information.
(3) We demonstrate that StreamingDialoguu significantly reduces computational complexity experi- mentally, ensuring the efficiency of streaming conversations.
http:://www.aaurrrppp..coooo//// 2-1

---

2 Related work
StrlooingDialogue efficiently handles long context, improving the model's long-term memory for conversation history. Existing methods for processing long context in transformer-ben based models broadly fall into three categories: efficient transformer design, long-term memory enhancement, and length extrapolation techniques.
2.1 Efficient transformers
Due to attention's computational bottleneck in transformers, methods aim explore efficient attention mechanisms. Solutions include trading accuracy for speed, e.g., Longformer [14] employs sliding window attention, expanding the receptive field with dilated sliding pattern and optionally integrating global attention. BP-Transformer []6] balances complexity and capacity with fine-to- coarse attention across multiple scales using binary partitioning. Linformer [17] approximates self- attention with a low-rank matrix, simplifying operations to linear ones. LongLoRA [18] uses block- wise attention and token shifting enhance communication between blocks. Another solution lies in system-level optimizations, e.g., FlashAttention [19 20 optimizes memory access by perceptually reading and writing, improving efficiency without sacrificing aceuracy. However, these methods don't ' preserve dialogue history expand the context window sufficiently for prolonged dialogue with long-term memory.
2.2 Long-term memory
Some methods enhance models' long-term memory to improve long-text modeling. One approach is introducing recurrent mechanisms into attention, enabling the model to maintain information over long sequences. For example, Transformer- XL [2I] introduces segment-level recurrence, reusing previous time step hidden states to model long dependencies. co-former [22 employs continuous-
space attention for arbitrary context modeling with fixed computational cost. Another approach is utilizing existing models interfaces to external knowledge bases, enhancing contextual input and long-term memory through reading and writing to these bases during inference [23]. e.g., MemGPT [24] employs hierarchical memory for LLMs, optimizing information transfer between context windows and external storage. However, they require retraining LLMs from scratch or additional information retrieval, lacking efficiency.
2.3 Length extrapolation
Length extrapolation in models refers to their ability to maintain good performance beyond the training length during inference. A mainstream solution based on position encoding. LLMs 27 employ rotary position embedding (RoPE) [28] for length extrapolation without fine- tuning. Initially introduced by Chen et al. 32, position interpolation proportionally extends the inference length by reducing rotation angles. NTK-award and NTK-by-parte parkbby-patemimmerppllaaaio interpolations balance high and low-frequency information to optimize performance. YaRN (3) combines -TK--by-prrts interpolation with an attention distribution correction strategy, reducing rotation angles for low frequencies and adjusting attention distribution. Additionally, randomized position encoding [34] extends context exposure by decoupling pre-traine pre-training length from inference length, utilizing random positions during training for broader context coverage. Due to current methods' inability for infinite length extrapolation, they're 'e unsuitable for prolonged dialogue in streaming applications.
3 StreamingDialogue
3.1 Empirical observation
StreamingI.LM 15 focuses initial tokens attention sinks, i.e., initial tokens attract significant amount of attention. After visualizing the attention maps of all layers and heads for both Llama-2-7B and -lama-2-7B--ha.n
observe similar phenomenon in structured texts such multi-turn
reddit.
allows, 11ama 1llamammdeela_..mmmmmmm wwtaaaamoeeele__ahaaaeeee ttt_etteeeeaaaeee

---

Predict tokens
Allow to attend
SMR&LMR
Inital
Short-memory Reconstructon
Follow-ap
Supervised Learing
Inference
Sole HaM 192
Convensation
Conseraation
Hiistory
Hilstoyy
rasd, sttodd, steed, aassed A880
Meckay
Thailing of SE000) Siee
What Goalis unorkactt are 50
unorkosb
an you tixe dio
Whall kiidd Insekoe
63.70
51 11
Fiitadly ait curtooudd dhorgghtnttillppss
Convensaiion
Long memory Reactivation
Conversation History
Hllstrry
Setal How dbset rynng moan axtinatiio
What
What 2ou
5OU
A fltor maltple sectioes
Paocskly RIEE Df Curti ueal dewgghhhhhhoooooo ttkrttig
Trisaely canto and sirogg toitnng
A dappting to Downstream
tows a innnt ensia
clallal slacing fii-So
Task Dialogue
20, ogp How'ss your cartio
andiategfi tr ryg-skes
Frotals
Neclond And pouil
swe.gh
Figure 2: StreamingDialoguee framework. SMR & LMR strategies co-train the model by adjusting attention mechanisms. In supervised learning, the SMR & LMR-trained model is fine-tuned with dialogue datasets. During inference, only specific tokens are cached, with critical historical dialogue information in bold italics for clarity.
dialogue, where LLMs tend to attend more to tokens used to separate dialogue when speakers switch (i.e., the end symbol "
," newline symbol "In," or other symbols, known as End-of-Utteranee and their attention aggregation is even greater than that of initial tokens (shown in Figure []. Based on the attention map, we suggest that each EoU captures the information of the current utterance, and EoUs visible to subsequent utterances. These EoUs imply the potential for information aggregation, which we refer to as "conversational attention sinks" (conv-attn sinks).
According to the observation, rather than caching entire utterances to retain information done in dense attention, we cache conv-attn sinks as replacement. Let T represent the number of utterances, and L denote the average length of each utterance. By caching only the corresponding conv-attn sinks, the space complexity reduces from O(TL) in dense attention to O(T). and the time complexity from O 1.²) to O). Moreover, given the infrastructure of LLMs based long context modeling, our method is capable of efficiently handling dialogues with prolonged conversational histories. To this end, conv-attn sinks matter because they memorize the context information not only effectively, but also efficiently.
3.2 Framework overview
3.2.1 Fine-tuning LLMs with conv-attn sinks
We compress the content of each utterance into the subsequent conv-attn sink and recall historical information during the dialogue by attending to conv-attn sinks. To achieve this, we adjust the attention pattern A E {0, NN.1,,,,, where N represents the conversation length and 0 indicates masked attention values, i.e., specifying the specific keys and values that query can attend to. Each token within an utterance can focus on the first token to uphold stable output during extended conversations, preceding conv-attn sinks to extract historical context, and tokens from both the previous and current utterances to ensure continuity with the preceding utterance. Formally, we denote an utterance as u = d
, where d represents the dialogue content and 
 denotes the EoU token, a.k.a., conv-attn sink. Thus, a conversation can be organized as D 
u U2... My where t is the number of utterances. Attention mask matrix A is defined as:
1, j = kl < (ke E N), 0 0si<N
1, 1 isisisi
Ai
1, h N). (1') 22
0, otherwise,

---

where denotes the average length of each utterance, N represents a non non-negative integer, and [-] represents the ceiling function. During fine-tuning, all tokens are treated predicted tokens to participate in the loss calculation.
While this method excels in managing long contexts compared to sparse attention methods like Streaming LLM, it falls short in characterizing short-term memories. To learn towards more robust model with balanced memory capacity for both long and short memories, propose two learning strategies to co-train the model and address these issues: short-memory reconstruction (SMR) and Long-memory reactivation (LMR). The final version of StreamingDialogue conducted three stages, including SMR & LMR, supervised learning, and inference, as illustrated in Figure 2
3.2.2 Short-memory reconstruction
We propose learning strategy to guide model behavior, enabling conv-attn sinks to consciously aggregate information through short-memory reconstruction (SMR). We reorganize data formats, modify the attention pattern, and adjust the loss function. More specifically, training samples are organized as D <S>w1 u/ u2u6. ... ..,, where U1, Ug..... U, are randomly selected from the original dataset, and represents the number of randomly selected utterances. Each "un"" pair can be regarded as reconstruction task, where tokens in u can attend to 
 and tokens that appear before the token in the current utterance. u' can additionally attend to the conv-attn sink in u. The task objective is to reconstruct u in u', encouraging the conv-attn sink in to aggregate information from for utterance reconstruction. The attention pattern A in SMR is defined as:
1, j
0
2k (k (
1,
Aij
1, 2k + (k (,, N,
0,
otherwise,
where N represents positive integer.
Since the goal is to reconstruct the contents of u/, u2...., , u', into u1, 02, Uss the loss calculation is defined as:
lul
LsMR
Σ
log (Po (ye x. v))))
(2.8) 1=1
where - (x,,, Mi).e. denotes the set of (u, u') pairs. denotes the target utterance and represents the reconstructed utterance u'. The model learns to aggregate information into conv-attn sinks during SMR with minimal training.
3.2.3 Long-memory reactivation
The model is required to both aggregate dialogue information into conv-attn sinks and retrieve information from them. To ensure consistency in multi-turn dialogue, our proposed model must efficiently extract long context information during dialogue generation. Therefore, we introduce long-memory reactivation (LMR) to enhance its long-term memory capability.
Each pair of utterances in the dialogue dataset, denoted as qr, represents a query-response pair with and from distinct roles. We organize training samples as D = 
(1'1 ... 9=T,- ... ....r///'''.. with ( denoting the number of training pairs. Each pair g,r", at the end of the sample is randomly selected from historical dialogues.
We design a response recall task where the goal is to recall / from the historical context GrTx given query d'g Concurrently, we adjust A so that each utterance can only attend to the first token, all conv-attn sinks, and itself. Moreover, each response in a training pair can attend the corresponding query, while the conv-attn sink of the response is restricted to attending only to the response itself,
5

---

ensuring that the conv-atin sink aggregates information solely from its associated utterance. i.c.,
1,
j j=0 0
1, [1-=2((EE)),))112)2kj555
2k (K (*)
Aij =
1, j= Id or 1>111111, -
0, otherwise.
Since the objective is to evoke historical dialogue within the response of the last training pair in the sample, the loss function of LMR is defined as:
In]
LLMR = Σ
Σ
log (Po (ne m, m,n))
(mL.) t=1
where m represents <S>IITI... QaTa .... ...'r' and n denotes r'z.
The model now effectively utilizes historical information through LMR. We co-train the model using SMR and LMR, fully harnessing the information aggregation potential of conv-atin sinks and enhancing both short and long-temn term memory capability of the proposed model.
Following SMR & LMR, the model requires additional refinement through fine-tuning dialogues using the methods outlined in Section 3.2.1
Table 1: Main results on the PersonaChat and MSC datasets. + indicates lower values are better, while t indicates the opposite. The best result for each metric is presented in bold, while the second-best one is underlined. * indicates significance (p < 0.05) via pairwise t-test compared to other methods. "PC" denotes PersonaChat and "StrLLM" represents StreamingLLM.
Data
Mathod
L B-avg
B-1
(%)
B-21
R-11
ROUGE (%)
R-2 1
R-L
D-11
Distinet D-21
D-3
USL-H
1
4
Donse
8.41
13.15
49.30*
20.05*
13.98
3.07
13.44
16.37
41.61
63.36
14.21
2.38
Local
11.59*
13.01
50.78
20.13*
13.83
2.69
13.29
12.49
32.17*
51.12"
17.35
2,07
Bird
13.78
13.33
$2.17
16.95*
Str.LM
8,96
13.16
50.15
20.68
13.94
2.73
13.36
12.00
32.64
52.36*
17.63
2.30
MemBART
13.15
11.18
46.63
17.65
13.11
2.56
12.78
12.86
30.87
48.86
12.23
2.49
Ours
8.71
13.63
$1.27
20.77
13.96
3.05
13.43
14.43
37.23
58.02
17.96
2.10
Dense
7.58
19.47
52.22
28.41
16.93
2.92
15.48
12.85
37.75*
$7.51
90.11"
1.94
Local
7,89
22.71%
MSC
Big Bird
8.42
16.54
46.63"
24.77
15.32
2.34
14.15%
8.72"
25.81*
40.34%
85.30
L.72
StiLLM
8,38
16.76*
47.54*
25.08*
15.25*
2.44
14.21
9.18
26.93*
41.62*
86.91
1.71
MemBART
13.73*
17.11*
49.78*
25.82*
14.93*
2.61
13.76
10.86*
30.55*
47.37*
85.
1.77
7.99
19.33
28.12
17.18
15.86
90.48
4 Experiments
4.1 Experimental setup
Datasets & baselines We conduct experiments on PersonaChat [35], Multi-Session Chat (MSC) (6). Topical-Chat (37) and MultiWOZ (38) datasets. MSC, known for its extended conversational context, differs from PersonaChat, which is single-session. MSC's training set includes up to sessions, and the test set comprises 5 sessions. Please refer Appendix A for detailed information on the datasets. We use only the dialogue portions of all datasets and explore the impact of additional knowledge from various datasets in Appendix F
Baseline methods include dense attention 7 for capturing all information, local attention |4] with a
fixed window size restriction, Big Bird 39 combining sliding window, global, and random attention, and StreamingLL.M [15] attending to attention sinks alongside the recent fixed window. Additionally, compare method with several methods that support long contexts, i.e., MemBART [40),
memory-augmentte Transformer encoder-decoder model, and HRED [41] and VHRED [42] which utilize hierarchical encoders capable of encoding conversations of arbitrary lengt
Appendix B provides attention features and implementation details for both the baselines and StreamingDia-
logue

---

Evaluation metrics We evaluate model performance on the dialogue generation task using several metries: BLEU (B-avg/ B-1 B-2, where B-avg the average of BLEU-1 BLEU-4 scores) 4 ROUGE (R-1 / R-2/ / R-L.) [4], and Distinct (D-1 / D-2 D-3) 45 Additionally, we utilize two reference-free metrics specifically designed for dialogue quality assessment: USL-H [46] and Dial-M
[77]. Perplexity (PPL) is also computed. Furthermore, we report the C scores [8] for various models on PersonaChat to assess the consistency the generated dialogues.
4.2 Main results
Table shows evaluation results for each method on the test sets (partial results related to baselines HRED and VHRED, metric score, and datasets Topical-Chat and MultWOO, are available in Appendix C For generation, use the last utterance from each test set episode as the target for the model to generate. We also calculate the overall PPL for the entire test sets.
Our method outperforms sparse attention and memory-augmennte baselines, achieving higher scores in BLEU, ROUGE, Distinct, and USL-H, while maintaining lower PPL and Dial-M metrics. For example, in MSC, StreamingDialoguee demonstrates significant improvements over the second-best baseline, StreamingL with B-avg increasing from 16.76% 19.33% and R-L. rising from 14.21 to 15.86%. In PersonaChat, specifically in D-2, Streaming increases from 32.64% to 37. 37.23% compared to StreamingI.LMM. The notable superiority Streaming can attributed to its focus conv-attn sinks, which compress historical information into them and cache them to enhance long-term memory, unlike baselines that rely on local attention windows and cannot handle extended dialogues effectively.
Furthermore, StrlamingDialogue exhibits comparable performance dense attention, e.g., in Per- sonaChat, the difference in ROUGE is less than 0.02%. It also outperforms dense attention in terms of and R-L in MSC and achieves better BLEU scores on PersonaChat. This validation highlights the more accurate information conveyance capabilities of text generated by our method.
Win
Tie
Loss
Fluency Coherence Consistency
3-%
35%
31%
41%
22%
37%
44%
27%
29%
Figure 3: Fluency, coherence, and consistency in human evaluations: ours vs StreamingL
4.3 Human evaluation
In human evaluation, generate dialogues from 100 randomly selected episodes of the MSC test set. Four crowdsource evaluators compare our method with Streaming LLM in fluency, coherence, and consistency, categorizing the outcome win, tie, loss. Figure [ demonstrates our method's superiority across all metrics, particularly in consistency, showcasing StreamingDialogue' 's superior long-term memory capacity. We apply Fleiss' kappa 49 measure the agreement among four annotators, yielding a result of 52.51 %. This indicates that the annor-an agreement is moderate (n E [0.4, 0.6)). More details on human evaluation are in Appendix D
Table 2: Ablation results MSC with different learning strategies. "Base" denotes the model fine-tuned without SMR and LMR learning.
Model
PPL
BLEU-avg
ROUGE-L
Distinct-3
Ours
7.99
19.33
15.86
50.27
Base
8.21
17.32
10.25
46.15
LMR
8.01
18.87
15.66
49.44
SMR
8.40
18.25
15.24
48.57
4.4 Ablation results
We conduct an experiment to test the effectiveness of SMR and LMR. Results, shown in Table 2 highlight significant decline in model performance when either strategy is ablated, indicating the
7

---

importance of both strategies. The absence of SMR results in prominent declines in BLEU and ROUGE scores, indicating inadequate information aggregation conv-atin sinks. Consequently, the model struggles to extract valuable information from conv-attn sinks during lengthy conversations, resulting in reduced text quality.
Similarly, without LMR, the model's performance declines significantly, indicating that relying solely on SMR leads to excessive guidance, limiting the model's ability in extended conversations. Thus, both SMR and LMR are crucial for enhancing information gathering and text extraction across conversations of long contexts.
LL
StraamingDaaogue
Streaming LLM Streaming Dialinege
Dense Dense with Re-compurattion
Llama-2-718
Llama-2-71
125
2,5
100
looPPL
Latency
2.3
0.15
25
2.1
0.10
A.2
512
Input Length
Input Length
Figure 4: Average perplexity and BLEU for StreamingLLM and StreamingDialogue on the MSC test set across varying utterance counts.
Figure 5: Per-token latency and memory usage by method on MSC for varying input lengths, with memory reported as total minus fixed.
4.5 Performance on different context length
We evaluate the model performance across different conversation lengths in terms of perplexity and BLEU under varying context length (i.e., the number of utterances in the dialogue context) during inference, as shown in Figure 4 As dialogue length increases, StreamingDialoguue exhibits greater superiority over StreamingLLMM with perplexity stabilizing and nearing convergence, and BLEU improving. Furthermore, StreamingDialogue maintains stable perplexity even with prolonged conversations over 25K tokens in inference (see Figure 6). This highlights our method's s stability in handling long dialogues and emphasizes the importance of conv-attn sinks in enhancing long-term memory.
logPPL
Llama-2-7B
5.0 2.5
200
300
400
500
600
700
800
Number of Utterances
Figure 6: The perplexity for StreamingDDallogue under the concatenated MSC test set, evaluating approximately 25K tokens.
Table 3: Results under the non-training setting on the MSC test set.
Model
Method
BLEU-avg
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
StreamingL
20.16
51.18
29.99
15.90
1.92
14.26
Llama-2-7B-Chn
Ours
20.19
51.55
30.03
16.46
2.11
15.00
StreamingL
16.48
39.68
24.63
16.88
1.93
15.47
Llama-3-8B-Insiee
Ours
16.77
40.10
24.88
17.11
2.01
15.85
StreamingI.I.MM
12.75
42.86
19.99
12.58
1.83
11.73
Mistral-7B
Ours
13.33
44.08
20.65
13.40
1.98
12.58
4.6 Performance under the non-training setting
We validate the performance StreamingI.LMM and our method under non-training setting on the MSC test set using Llama-2-7 2-7B-Chai Mamma-3-8B-Instruee (50) and Mistral-7B [51]. As the table

---

Base
SMR & LMR
Base
SMR & LMR
Base
SMR & &MRR
12-
22-
12-
12-
16-
20
20
20
20
201
4
12
16
20
-08
20
12
16
20
20
8
16
20
-00
20
Layer Head
Layer 11 Head
1
Layer 12 Head 8
Figure 7: Comparison of attention maps before and after learning. "Base" denotes Llama-2-7B, while "SMR& & LMR" represents the model obtained post co-training with SMR and LMR on Llama-2-7B The "
 positions in the encoded sentences are: 3, 6, 13, and 21.
shows, StreamingDialogue, thanks the conv-attn sinks, retains more complete historical information. Consequently, consistently outperforms StreamingLLM. This demonstrates that applying conv-attn sinks for modeling long contexts remains effective even without any training.
Table 4: Dialogue reconstruction performance.
BLEU-avg
BLEU-1
BLEU-2
ROUGE-1
ROUGE-L
68.02
89.19
76.83
76.79
72.94
4.7 Information preservation
To assess our method's information compression loss, we use EMRRtrained models to reconstruct dialogue content from the MSC test set, leveraging only the conv-attn sink of each utterance. Randomly selecting 6,000 utterances from the MSC test set, we present the average results Table Our method achieves a BLEU-1 score of 89,19, signifying effective com- pression of dialogue information with minimal losses.
Comeesation History
11 utternnoe
Do low fax duse? 2 Towe cdlfiardo.
L war the greete sansd so for fethwyyes lowos like mancee thas cansee, bet hzard tamt supposed be some groat stactt this wed. and laganel shoald le you laasn das thene cay be suzer good tean.
do like zeriss quie ktt but dact gand suaccs at Atter
en grod sanade wlame live
inne, yoafl
evcetng
omaage vtole, It's stontigg.
4.8 Speedup for inference
utteronos
Figure 5 depicts the average per-token latency and memory usage during dialogue generation with NVIDIA A100 GPU using various methods. As in- put lengths increase, StreamingDialogue shows min- imal growth in memory usage for caching conv-attn sinks, with latency exhibiting linear growth. This sug- gests that as dialogue length increases, StreamingDi- alogue's advantage becomes more promising. At an input length of 2,048, our method demonstrates 6 x improvement in memory usage compared to dense tention and an 18 improvement compared to dense attention with re-computation. In terms of per-token latency, our method shows 4 improvement com- pared to dense attention with re-computation. More- over, our method maintains similar latency and memory usage StreamingLLM as context length varies.
eerjyy foodh lior Spon moars, and faencio night iodes park. The way they pregen the moken it bette good
Generale
Iveanser haurd of azs of tham daias, but I'a sare Lil able is fied then en the more.
Stenamiogg. L.M
T'e heard lone then. the evobed bry new trrnge I'm sam 111 allo fiad sume Blowadas food ia Caliiionia
dlibes,
Figure 8: The generated dialogues by StreamingLLM and StreaningDialogue for the same input dialogue history from MSC episode, with an average utterance length of L 32 tokens. Bold italic indicates key in- formation the dialogue.
4.9 Impacts of SMR LMR learning
Since the motivation of SMR and LMR learning is to improve conv-attn sinks aggregation capability, we examine the attention maps after SMR and LMR co-training, comparing them with the base model to confirm enhancement. Results are illustrated in Figure 7 Guided by SMR and LMR, the model's attention patterns transform into maps that sharply concentrate on conv-attn sinks, showcasing our effective enhancement of their information aggregation ability.

---

4.10 Case study: memory capacity
To validate ArraminngDaaoguee effectiveness in enhancing long-term m we conduct case study comparing it with StreamingLLMM Figure g illustrates content generated by both methods. StreamingLLM responds solely based on recent utterances, lacking connection distant context and coherence, thus reaffirming its unsuitability for open-domain dialogue. In contrast, StreamingDiar logue effectively recalls distant historical information (e.g., 44 utterances ago), demonstrating the model's is enhanced ability to remember long conversations through SMR and LMR.
4.11 Generality of conv-attn sinks
To establish the generality of the conv-attn sink phenomenon, where separators attract more attention than other tokens within dialogues, we conduct both qualitative and quantitative analyses. Qual- itatively, we visualize attention patterns across different training methods, attention mechanisms, and types of dataset constructions, demonstrating that this phenomenon persists regardless of these variables. Quantitative measures further support these findings, with a set threshold indicating signifi- cantly higher attention on separators than on other tokens. Detailed results, including visualizations and statistical data, are provided in Appendix E This comprehensive analysis confirms the robustness of the conv-attn sink phenomenon across different settings and models.
4.12 Analysis of EoU tokens' information aggregation capability
To assess the effectiveness of EoU tokens in capturing dialogue information, we conduct experiments using an untrained Llama-2-7B-CC model. The model focuses solely on EoU tokens and the last complete utterance. a case study with the conversation: "Did you have caramel macchiato today?
 Yest kind of coffee did you have today?
," the model successfully identifies the key information, responding with, "I had delicious caramel macchiato this morning."
We replicate this experiment across 10 different prompt formats, each containing 20 samples. The results indicate that 68% of the model's responses accurately include essential information. Further details on these formats are available in Appendix
4.13 Sapea--prr sensitives
We investigate the impact of two Syper-parameters in our method: the number of utterances in SMR samples (8) and the number of query-response pairs in LMR samples (l), both ranging from {8, 12, 16, 20, 24, 28, 32). We maintain constant total number of utterances in the training set, with examples like
PPL
R-L
D-3
Norallzzd Sevees
1.0
Nornalizd Scoret
1.0
0.3
P.
0.5
0.5
0.4
x samples 24, 000 and
= 12 x samples = 16, 000. Results shown in Figure 9 reveal that StreamingDialogue performs best with higher values of and I, optimally at {28, 32} and {20, 24, 28). This suggests that longer texts enhance the model's 's learning.
Figure 9: Normalized performance scores (PPL., B-avg, R-L,, and D-3) on MSC for var- ious with s fixed at 28 and various s with / fixed at 24.
5 Conclusion
Generating high-quality open-domain dialogues with prolonged contexts is quite challenging. Existing solutions, like dense attention, have efficiency issues. While StreamingLLM supports efficient language modeling, struggles to preserve historical information, leading to low-quality generation in prolonged conversations. In this paper, introduce StreamingDialogue, framework capable of facilitating efficient and prolonged dialogue. By identifying separator tokens FoU as "conv-attn sinks" and compressing dialogue information into them with minimal losses, StreamingDDialogue conserves memory, enhances efficiency, and augments long-term memory capabilities. Additionally, we propose learning strategies enhance conv-attn sink aggregation and memory reactivation. Our method shows better performance compared to strong baselines. In the future, we will explore extending StreaningDialogue towards never-ending dialogue in the context of lifelong learning.
10

---

Acknowledgments and Disclosure of Funding
We sincerely thank Songhao Wu and Ang Lv for their valuable discussion and feedback on the manuscript.
This work supported by the National Natural Science Foundation of China (NSFC Grant No. 62122089, U21B2027), Beijing Outstanding Young Scientist Program NO. BJIW2YJH01201110000000
and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double- First Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.
References
[1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21()):5485-5511, 2020.
[2] Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.103000 2023.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (xXiv:2303.08744 2023.
[4] Yuhan Liu, Xiuying Chen, Xiaoqing Zhang, Xing Gao, Ji Zhang, and Rui Yan. From skep- ticism to acceptance: Simulating the attitude dynamics toward fake news. arXiv preprint wrXiv:2403.09499. 2024.
[5] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[6] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arX.h:2307.099366 2023.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[8] Liu Yuhan, Chen Xiuying, and Yan Rui. Unleashing the power of large models: Exploring human human-machine conversations. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), pages 16-29, 2023.
[9] Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Dialogue state tracking with a language model using schema-driven prompting- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference Methods in Natural Language Processing, pages 4937-4949, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.118861110000 org/
Empirical
emnlp- m04n.404. URL
www..aiii.... 404
[10] Songhao Wu, Quan Tu, Hong Liu, Jia Xu, Zhongyi Liu, Guannan Zhang, Ran Wang, Xiuying Chen, and Rui Yan. Unify graph learning with text: Unleashing 11m potentials for session search. In Proceedings of the ACM on Web Conference 2024, pages 1509-1518, 2024.
[11] Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji-Rong Wen, and Rui Yan. MISC: A mixed strategy-awe aware model integrating COMET for emotional support conversation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 308-319, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/202..
URL https: http://alaaaatttoaaa/ea org/ 2022. acl acl-long....... 25

---

[12] Björ Bebensee and Haejun Lee. Span-selective linear attention transformers for effective and robust schema-guided dialogue state tracking. Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 78-91, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v acll2222...loooo..... URL htttps://aclantholpgs .arpegg...gg//gg org/ acl -
[13] Ang Lv, Jinpeng Li, Yuhan Chen, Gao Xing, Ji Zhang, and Rui Yan. DialoGPS: Dialogue path sampling continuous semantic space for data augmentation in multi-turn conversations. In Anna Rogers, Jordan
Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1267-1280, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. aact.///oo........oo..o....
URL htttt:://ccaaaaatait
2023/2023.cg acl -oog..n..... 70
[14] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- long-document transformer. arXiv preprint urXiv:2004.05/50. 2020.
[15] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309./740 /7453, 2023.
[16] Zihao Ye, Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. Bp-transformer: Modelling long-range context via binary partitioning. arXiv preprint arXiv: 2019.
[17] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, arXiv preprint arXiv:2006.04766 04768, 2020.
[18] Yukang Chen, Shengju Qian, Haotian Tang. Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12300 2023.
[19] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
[20] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08661, 2023.
[21] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, 2019.
[22] Pedro Henrique Martins, Zita Marinho, and Andre Martins. oc-former: Infinite memory trans- former. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pa- pers), pages 5468-586 5485, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/1 1/t033////222..arm acl- 1ong.355.
wwcpllgggoo....gg 2022 acl acl-long
375
[23] Yunpeng Huang, Jingwei Xu, Zixu Jiang, Junyu Lai, Zenan Li, Yuan Yao, Taolue Chen, Lijuan Yang, Zhou Xin, and Xiaoxing Ma. Advancing transformer architecture in long-contex. large language models: A comprehensive survey. arXiv preprint arXiv:2311./233 2023.
[24] Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards Ilms as operating systems. arXiv preprint arXiv:2310.08560, 2023.
[25] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint ar0iv:2308. 12950, 2023.
[26) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint XrXiv:2309.16600 2023.
12

---

[27] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[28] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Nenrocomputing. 568:127003, 2024.
(29) Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455, 2023.
[30] Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05200, 2023.
[31] Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, and Rui Yan. Mixture of in-context experts enhance Ilms" long context awareness. arXiv preprint arXiv:2406. 19598, 2024.
[32] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window large language models via positional interpolation. arXiv preprint orXiv:2306.15599 2023.
[33] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint wrXiv22309.0007/ 2023.
[34] Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length gen- eralization of transformers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, ed- itors, Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 1 1889-1903. Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v1/2023 acw.:......llll.c.... acl-shorr.161. URL https
/ //aclanthology. org/ 202//2223..22 acl- 161
[35] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia, URL
July 2018. Association for Computational Linguistics. doi: 10.
https://aclaathoology htwphhooooyyooog/oP.. oww/ 1205
[36] Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open- domain conversation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, ed- itors, Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 5180-5197, Dublin, Ireland, May 2022. Associ- ation for Computational Linguistics. doi: 10. hatp://ww//2222allloooggg... 7/ //aclanthology.orgt ww.oorg//20022 wwl.22cclllo....... 356
URL https
[37] Karthik Gopalakrishnan. Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür, Topical-Chat Towards Knowledge- Grounded Open- Conversations. In Proc. Interspeech 2019, pages 1891-1893 2019. doi: 100211447//terspeecch211111 2019-399. URL //tttt:///xxx doi org/ 10. 21437 htt.:///.......///eeeeeeeeee 2079-3079
[38] Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. Multiwoz-a large-scale multi-domaii wizard-of-or dataset for task-oriented dialogue modelling. arXiv preprint 1rXiv:1810.00275 2018.
[39] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 17283-17297, 2020.
13

---

[40] Qingyang Wu and Zhou Yu. Stateful augmente transformers for dialogue modeling. arXiv preprint 220::22000,0664,, 2022.
[41] Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian- Yun Nic. A hierarchical recurrent encoder-decoder for generative context-awadd aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM 11, page 553-562, New York, NY. USA, 2015. Association for Computing Machinery. ISBN 978 97814393379 htttss///oi..o 10. 1145/2806116 2884444
doi: 10. 00.1145/28004112280664999 URL
[42] Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[43] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational URL http::ttps/// / ac waaeholnntoggyyy org/PO2-1040
Linguistics. doi: 10.31 15115/50030110033303
[44] Chin-Yev Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- fion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL U https:tt/////ct htt.t::://acllanthoooooo org/ htrr//004-1033
[45] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promotiin objective function for neural conversation models. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, pages 11, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/ v1/N16- 1014. URL https: http:::://alaaahhhooocoo .rg.lloggoorggg//ee www.oog............... 1014
[46] Vitou Phy, Yang Zhao, and Akiko Aizawa, Deconstruct to reconstruct a configurable evaluation metric for open-domain dialogue systems. arXiv preprint arXiv:2011.0044 2020.
[47] Suvodip Dey and Maunendra Sankar Desarkar. Dial-m: A masking- based framework for dialogue evaluation. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 77-84, 2023.
[48] Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. Personalizing dialogue agents via meta-learning. In Anna Korhonen, David Traum, and Lluis Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 5454-5459, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/1//11--1544 URL https: /tops://acclaee htt////accaathoooggyyoo org/ 1542
[49] Joseph I, Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
[50] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[51] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[52] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 320-335, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. URL https: http///aaaaantttoaaaoaao org/ wtg//022....02oooo 2022. acl acl-long...... 26
14

---

[53] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-41880 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/11//1191 1423. URL https: acnthoooogy org org/N19-1423
15

---

Dense attention
Local attention
Big Bird
StreamingL.MM
StreamingDilogue (ours)
cathed
iquoeed cached tokoss
Innored Tokens
Jquoeed "ukoms
cached "tokens"
Iynorvl tokinss
cached tokens
Imuored Tokans
Takens
Igpored 21 cached
Takens
com-aatte
sink
(a)
(b)
(c)
(d)
(e)
Dialogue:
T utterances
tokens
tokens
Figure 10: Attention maps' visualization of StreamingDialogue and various other methods. In dialogue with T utterances, each averaging tokens, dense attention caches TL tokens, local attention caches R tokens (where R is the window size), Big Bird caches global size + random size + tokens, StreamingL.LM caches R tokens, and StreamingDDiloggue requires caching up to 1 1+T + 2L tokens.
A Dataset details
Refer to Table 5 for specific details regarding the PersonaChat, MSC, Topical- Chat and MultiWOZ datasets.
Table 5: Details of dialogue datasets. We present the number of utterances (Utts.) and the average length per utterance (Avg. for each session in the training and test sets.
Train
Test
Data
Data Type
Utts.
Avg. I.
Utts.
Avg. I.
PersonaChat
Total
122499
13.59
14602
13.85
Session
59894
14.16
6572
15.47
Session
46420
31.44
5939
30.86
Session
47259
32.90
5924
32.94
MSC
Session
11870
32.25
5940
34.67
Session 5
5945
36,43
Total
165443
25.66
30320
29.77
Topical-Chal
Total
188378
26.76
11760
26.98
MultWOZZ
Total
113552
18.92
14744
19.23
B Attention patterns & implementation details
We visualize the attention maps of StreamingDialouee and several other baselines focusing different attention patterns, as shown Figure 10 Below are the details of the experiments for each method.
During the SMR & LMR phase, we construct short-memory reconstruction dataset comprising n, = 6857 samples, each containing random selection of 28 utterances from dialogue datasets. Simultaneously, for the long-memory reactivation dataset, we generate n, = 8000 samples, each consisting of random selection of I 24 query-response pairs from the full set of query-response pairs dialogue datasets, with one additional pair randomly chosen from the 7 pairs appended the end of each sample. The SMR and LMR datasets merged and shuffled for co-training Llama-2-7B 2-BB.
16

---

We only train the attention layer for epoch, with the learning rate set to 5e-5, utilizing cosine annealing for adjusting the learning rate, and setting the warm-up step The SMR LMR phase requires about 2 hours two A A100-40G GPUs.
In the supervised learning phase, StreamingDialoge undergoes fine-tuning based the model trained with SMR & LMR, while baselines focusing on different attention patterns are fine-tuned on Llama-2-7B. All models fine-tune only the attention layer for 2 epochs, with the learning rate to 5e-5, utilizing cosine annealing for adjusting the learning rate, and setting the warm-up step to O, This phase demands only approximately hour on two A100-40G GPUs. MemBART, HRED and VHRED can be fine-tuned directly on dialogue datasets.
In inference, set the maximum generation length to 120 and report the average results of all episodes. Dialogue generation takes only about 15 minutes on a single A A100-40G GPU.
C Additional results for main evaluation
Table presents the experimental results incorporating baselines HRED and VHRED. Our method achieves favorable results compared to any baseline across all metrics.
Table 6: Results on the MSC dataset. indicates lower values are better, while indicates the opposite. The highest- result for each metric is highlighted in bold, and the second-highest underlined.
Method
B-avg t
R-1 f
R-2
D-1 f
D-2
USL-H
Dial-M 4
StreamingLLM
16.76
15.25
2.44
26.93
86.91
1.71
HRED
15.72
14.75
1.85
7.37
20.91
58.70
2.13
VHRED
17.02
15.16
1.48
5.28
14.72
59.31
2.35
Ours
19.33
17.18
2.77
11.54
32.58
90.48
1.76
Table 7 reports the results of the C score on PersonaChat, Our StreamingDialogue still achieves the best results among the baselines, except for dense attention. As an efficient algorithm, our method can significantly improve the speed compared to dense attention while maintaining the contextual and character consistency of long conversations.
Table 7: Results of the C score on the PersonaChat dataset. t indicates higher values are better.
Method
Dense
Local
Big Bird
StreamingL.LM
MemBART
Ours
C (%)
3.10
-3.40
-4.00
-4.70
0.77
2.70
Table 8 presents the results for the Topical-Chat and MultiWOZ datasets, where our method outper- forms all strong baselines by retaining more complete historical information.
D Human evaluation details
D.1 Information about evaluators
We engage four crowdsource evaluators to assess the performance of our method. These evaluators exhibit outstanding English proficiency, enabling them to accurately discern subtle nuances and meanings in language expressions. Moreover, they possess comprehensive understanding of the distinctions among fluency, coherence, and consistency, and are adept at determining which response is superior. In terms of human evaluation, pass the review by relevant institutions, and anonymize all evaluators' responses. After a reasonable assessment of the workload, we pay each evaluator $1.70 per 10 samples.
D.2 Task description
Table T presents the detailed task description provided to evaluators, where responses generated by different models are randomized in each sample.
17

---

Table 8: Results on the Topical-Chas and MultiWOZ datasets. + indicates lower values are better, while f indicates the opposite. The highext-performing result for each metric highlighted in bold, and the second-highest is underlined.
Data
Method
PPL 4
ROUGE-1 f
ROUGE-2 t ROUGE-L1 1
Dial-M
Dense
9.49
15.70
3.65
14.88
3.09
Local
27.55
12.60
2,09
10.37
7.02
Big Bird
10.36
14.21
3.55
11.79
3,01
Topical-Chat
Streamingl
10.34
14.25
3.55
11.84
3.05
MemBART
12.54
13.86
2.98
13.18
2.83
Ours
9.80
15.46
3,99
14.37
2.66
Dense
4.51
24.79
13.93
24.67
2.27
Local
5.38
24.26
13.47
24.15
2.45
Big Bird
4.79
24.38
13.26
24.30
2.51
MultiWOZ
StreamingLLM
4.76
23.66
13.09
23.41
2.47
MemBART
5.36
20.05
12.41
19.94
2.37
Ours
4.34
25.26
14.27
25.20
2.25
Table 9: Task description provided evaluators in human evaluation.
Task description
We aim to evaluate the quality of dialogues generated by various models, specifically focusing fluency, coherence, and consistency. You will be presented with dialogue history followed by responses generated by different models the latest utterance. consistency please identify the superior response. Assign a "1" the first response is better, a "2" if the second response is better, and "0" if both responses are of equal quality, Ensure that your selections clearly reflect which response better meets each metric.
each evaluation metric fluency, coherence, and
The specific descriptions of the three metrics are as follows:
(1) Fluency assesses whether the response itself is well-written and grammatically correct.
(2) Coherence refers the response being relevant to the content the historical dialogue.
(3) Consistency requires response to remain consistent from the historical dialogue.
persona information and objective facts
Details on the generality of conv-attn sinks
To demonstrate the generality of the conv-attn sink phenomenon, i.e., the model aggregates more attention separators than other words and tokens, we provide both qualitative and quantitative analysis.
For qualitative analysis, visualize the attentions using various training methods, attention mechanisms, and dataset constructions to observe the behavior of attention patterns.
(1) Regarding the independency on the training methods, we compare Llama2 with GLM 52 Llama2 pretrained by the next-token prediction objective, while GLM is pretrained on an autoregressive blank infilling objective.
(2) Regarding the independency the attention mechanisms, compare Llama2 with BERT 553 Llama2 uses a unidirectional causal attention mechanism, while BERT uses a bidirectional attention mechanism.
(3) Reagarding the independency on the dataset constructions, we compare two-person dialogues with multi-party dialogues. Otherwise, we also have validated that the different separators for orgnizing the dialogue also exhibit the same phenomenon, as illustrated Figure
The visualization results under different settings are shown in Figure II which demonstrates the consistent conclusion that separators will attract more attention than other tokens the dialogue. For the two-person dialogue, the data we use is coffee Fater>k Starbucks?<FOSS>" * and for the multi-party dialogue, the data we use is *<BOS>Alice: How your weckend Good, youns?<ES--hartie Great,
grab
18

---

BERT Layer 4 Head 4
BERT Layer 4 Head 4
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
EOUEOU
EOU
EOU
EOU
EOU Multi-party
EOU EOU
Two- party
GLM
Laver 1 Head 8
GLM Layer Head 8
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
EOUEOU
EOU
EOU
EOU EOU EOU EOU Multi-party
Two-party
Llama2 Layer 1 Head 30
Llama2 Layer 1 Head 30
1.0
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
EOU BOU
EOU Two-party
EOU
EOU
EOU EOU EOU Multi-party
Figure 11: Attention maps under different settings.
thanks<EOS>Alies Awesome<EOS>." "<BOS>" denotes the start symbol, and "<EOS>" denotes the end symbol.
For quantitative analysis, we set threshold indicating the occurrence of the conv-attn sink phe- nomenon in an attention head. This threshold defined the average attention aggregated by conv-attn sinks being three times more than that aggregated by normal tokens. We report the
19

---

Table 10: Proportion of attention heads exhibiting the conv-attn sink phenomenon across models.
Model
Llama2
GLM
BERT
Proportion (%)
14.36
16.32
79.86
proportion of these conv-attn heads among all attention heads using three heterogeneous models, as depicted in the Table This illustrates the universality of the conv-attn sink. Furthermore, we observe stronger tendency of the conv-attn sink in the BERT model, warranting further investigation in our future work.
F
Impact of additional knowledge
The original datasets used for both training and testing contain some additional knowledge: Persona- Chat includes personas, Topical-Chat includes grounding knowledge, and MultiWOZ. includes belief states. Our main experiments utilize only the dialogue portions of these datasets, without leveraging the additional knowledge. To make our results more compelling, explore the impact of incorporating this extra knowledge into our experiments.
For MultiWOZ, we add the belief states before each corresponding utterance. The results are shown in the Table I1
Table 11: The results of integrating the belief states on the MultiWOZ dataset.
Method
PPL
BLEU-avg
BLEU-1
BLEU-2
Distinct-1
Distinct-2
Distinct-3
Dense
1.92
25.56
48.33
29.14
3.74
6.86
8.89
StreamingLLM
2.19
25.70
47.53
29.21
4,48
9.09
12.60
Ours
1.98
25.77
48.58
29.38
5.30
10.03
13.60
Since our method retains historical information by compressing each utterance's information into conv-attn sinks, only the conv-attn sinks from the previous utterances will be attended to in subsequent utterances. Therefore, for Topical-Chie and Chrsona-Chat, we consider two settings:
(1) We treat each sentence of the grounding knowledge/personaa profiles as an utterance, and the subsequent utterances can only attend to their conv-attn sinks. The results are shown the Table 12
Table 12: Results on the Topical-Cii -Chat and Persona Chat datasets under the setting treating each sentence of the grounding knowledge/persona profiles utterance.
Data
Method
PPI.
Distinct-2
Distinct-3
Dial-M
Dense
7.19
43.56
66.27
2.53
PersonaChat
StreamingI.LM
8.36
33.17
53.58
2.47
Ours
7.60
39.16
61.06
2.36
Dense
3.24
39.07
57.64
4.32
Topical- Chaat
StreamingLLM
8.31
16.87
23.56
3.72
Ours
3.20
31.47
49.10
2.57
(2) We use the grounding knowledgc/personaa profiles as a prompt: "The conversation will be based on the following knowledge: <knowledge> detailed knowledge <conversation>" in Topical-Chat and "The conversation will be based on the following persona profile: <persona> detailed persona profiles <conversation>" in Persona-Chat allowing the subsequent utterances to fully attend to it. The results are shown the Table 13
In setting that includes grounding knowledge, our method consistently retains the memory of both grounding knowledge and historical dialogue, As a result, our method still outperforms the baseline, except for dense attention. As an efficient algorithm, our method can significantly improve speed compared to dense attention while maintaining the contextual and character consistency of long conversations.
20

---

Table 13: Results on the Topical-Chat and Persona-Chie datasets under the setting of treating the grounding knowledge/persona profiles as a prompt.
Data
Method
PPL
Distinct-2
Distinct-3
Dial-M
Dense
7.93
44.26
66.63
2,48
PersonaChat
StreamingI.IMM
7.99
36.40
57.44
2.91
Ours
7.67
37.82
58.93
2.57
Dense
11.64
36,98
54.96
4.60
Topical-Chi Chat
StreamingLLM
30.37
26.07
34.26
3.61
Ours
10.21
32.16
50.41
2.97
G
Examples of prompt formats
Examples of prompt formats are as follows, where the "keywords" will be replaced with specific content.
1. "template": "A and went to PLACE today.c/s>They had a great faane.//s>>Wit did A PLACE with today?
",
"keywords": "A": "person", "B": "person", "PLACE"; "place", "answer key": "B"
2. "template"; "B made A's favorite food, FOOD, today.
A was delighted. 
tattt food did B make for today?
", "keywords": "A": "person", "B": "person", "FOOD": "food", "answer key": "FOOD"
3. "template": "A doing ACTIVITY when B called.
A had stop and answer the call.
What was A doing when B called?
", "keywords": "A": "person". "B": "person", "ACTIVITY": "activity"" "answer key": "ACTIVITY"
4. "template": "A bought a new ITEM today.
B was impressed by A's purchasc.
 What item did A buy today?
", "keywords": "A": "person", "B": "person", "ITEM": "item",
"answer key": "ITEM"
5. "template": "A participated in an EVENT today.
B cheered them on. on.
What event did A participate in?
", "keywords": "A": "person", "B": "person", "EVENT": "event",
"answer key": "EVENT"
H Limitations
Dialogue significantly reduces space and time complexity during the inference stage. Additionally, we can outperform the baseline under the non-training setting without additional cost. To optimize LLMs for the conv-attn sinks mode, we implement two learning strategies: short-memory reconstruction and long- mong-memory reactivation. Consequently, this inevitably increases computational costs under the training setting, with the SMR and LMR phases requiring about two hours on two A100-40G GPUs.
While StreamingDialogue effectively enables prolonged conversations with long-term memory, there's bere's merit in exploring selective caching of conv-attn sinks, focusing only on those aggregating key information. This will further enhance inference speed and reduce memory usage. Additionally, our utilization of dialogue structure is somewhat limited, and we aim to leverage conv-attn sinks to explore more intricate dialogue features in the future. Furthermore, evaluating our method across wider range of structured texts will offer more comprehensive assessment.
21

---

I Broader impacts and safety issues
This paper identifies the conversational attention sink (conv-attn sink) phenomenon and proposes two learning strategies to naturally compress extended dialogue history into conv-attn sinks and effectively retrieve memories in subsequent conversations, thereby enabling prolonged streaming dialogue. Additionally, the datasets utilized, including PersonaChat and MSC, as well as the pre- trained models Llama-2-7B and Llama-2-7B-CC through official open-source channels. Our enhancements are purely architectural and we will not release any new models or datasets. Nonetheless, the capability to efficiently compress and retrieve long dialogues may allow this technology to be used for monitoring conversations over extended periods, thus raising privacy concerns and the potential for surveillance without proper consent or transparency.
were sourced from their respective publishers
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we claim that StreamingDialogue efficiently compresses dialogue history into conversational attention sinks with minimal losses, enhancing the model's long-term memory and facilitating prolonged streaming conversations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We create a separate "Limitations" section, i.e., Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification:
Our paper does not introduce theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have shared URL that contains executable code, enabling easy reproduc- tion of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have open- sourced our code and shared the link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
publit//uiiees/CoeeSttuiii Cosooooeesmmiissiooooooooryy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttpe//ggu..eeeoooouuuummbibbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4 and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We present the statistical significance of the experiments in Table 1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on the computing resources required to reproduce the experiments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have conformed the NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section ||
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
See Section
!|
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have correctly cited all the data, scripts, and models used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have included README document with our code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "mFax3zkQDt", "reviewer": "Reviewer_7W1D", "summary": "The paper introduces a novel approach for encoding long conversations. Motivated by the results of StreamingLLM, the authors observe that the end-of-utterance (EOU) or separator tokens aggregate more attention than other tokens in a dialogue generation task. The authors refer to the EOU tokens as conv-att sinks (conversational attention sinks). Based on this observation, the authors propose to attend and cache only the conv-att-sinks of the past utterances to represent the dialogue history, thereby making the space complexity of the attention mechanism linear to the number of turns in a conversation. To learn quality embeddings for conv-attn sinks, the authors propose two auxiliary tasks, SMR (a response reconstruction task) and LMR (a response recall task). The proposed StreamingDialogue encoding strategy achieves comparable performance to dense attention (attention on all previous tokens) and outperforms memory-efficient baselines on Persona-Chat and MSC datasets. The method also exhibits 4× speedup while reducing memory usage by 18× compared to the dense attention strategy.", "questions": "1. Why did the authors not use USL-H and Dial-M for Table 1?\n2. Did the authors use the persona profiles to generate the responses? If yes, how was it included in the context? If not, is it fair to compare BigBird and StreamingLLM with StreamingDialogue?\n3. Explain the inconsistent use of the BLEU metric.\n4. Why are Layer 0 and Layer 1 shown in Fig. 1a, whereas Layer 28 is shown in Fig. 1b?\n5. Did the authors analyze the conv-attn sinks for the example in Fig. 7?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand. The proposed method is well-motivated and addresses an important problem of encoding long dialogue contexts.\n2. The proposed attention strategy of utilizing only the conv-attn sinks is simple and effective. StreamingDialogue shows better performance than memory-efficient baselines on both automated and human evaluation. The method also shows performance comparable to that of the dense attention strategy.\n3. Results suggest that the SMR and LMR help to learn a rich representation of the conv-attn sinks. The authors also show evidence (Fig. 7) that the model can recollect/generate past information from the previous conv-attn sinks.\n4. The method is cost-effective and can achieve significant speed-up compared to the dense attention strategy.", "weaknesses": "1. Although the results shown in Table 1 are positive, the metrics are not well-suited for the open-domain dialogue generation task. The authors have shown their results on two additional metrics (USL-H and Dial-M) in Table 6 for the MSC dataset. USL-H and Dial-M have been shown to be better metrics than BLEU, ROUGE, Distinct, and perplexity, especially for persona-grounded datasets like Persona-Chat. However, Table 1 does not show the results with USL-H and Dial-M. Also, in the case of MSC, StreamingDialogue performs better than StreamingLLM on the USL-H metric but not on the Dial-M metric. This is why I think that although the method is appealing, the results are not strong enough to support it.\n2. There is no information about inter-annotator agreement for the human evaluation.\n3. The experimental setup with the Persona-Chat and MSC dataset is not clear. The authors have not mentioned whether they used the persona profiles to generate the responses for the result of Table 1.\n4. The use of the BLEU metric is not consistent. The authors use average BLEU in Table 1 and Table 2. However, BLEU-1 and BLEU-2 are shown in Table 3, whereas only BLEU-1 is shown in Table 4. It would be better to show all three variations of BLEU in all the result tables.\n5. The results are shown only on two dialogue datasets. There are other datasets with long dialogue contexts like Topical-chat. MultiWOZ is another dataset for task-oriented dialogue systems, which includes lots of conversations where the user utterance directly refers to past dialogue history.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7W1D,\n\nWe sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.\n  \n> Table 1 does not show the results with USL-H and Dial-M. Also, in the case of MSC, StreamingDialogue performs better than StreamingLLM on the USL-H metric but not on the Dial-M metric. \n\nThank you very much for your valuable suggestion. The experimental results using USL-H and Dial-M for Table 1 are shown in the below table:\n\n|MSC|Dense|Local|BigBird|StreamingLLM|MemBART|Ours|  \n|:-|:-|:-|:-|:-|:-|:-|\n|USL-H&uarr;|90.11*|76.68*|85.30*|86.91*|85.13*|90.48|  \n|Dial-M&darr;|1.94*|2.15*|1.72^|1.71^|1.97*|1.76|  \n|**PersonaChat**|**Dense**|**Local**|**BigBird**| **StreamingLLM**|**MemBART**|**Ours**|  \n|USL-H&uarr;|14.21*|17.35*|16.95*|17.63*|12.23*|17.96|  \n|Dial-M&darr;|2.38*|2.07^|2.37*|2.30*| 2.49*|2.10|\n\n*&uarr; indicates the higher score is better, while &darr; indicates the lower score is better. * indicates significance and ^ indicates insignificance.*\n\nOur method significantly outperforms all baselines across several metrics, including USL-H, PPL, BLEU, ROUGE, and Distinct. It also exceeds most baselines on Dial-M, but BigBird and StreamingLLM show only non-significant improvements over our method in Dial-M on MSC. Additionally, our method demonstrates significant improvemnet over all baselines in Dial-M when evaluated on two newly added datasets: Topical-Chat and MultiWOZ. These results confirm that our method is indeed better than baselines.\n\n> There is no information about inter-annotator agreement for the human evaluation.  \n  \nWe apply Fleiss' kappa [1] to measure the agreement among four annotators, yielding a result of 52.51%. This indicates that the inter-annotator agreement is moderate ($\\kappa \\in [0.4, 0.6]$).  \n  \n> Did the authors use the persona profiles to generate the responses? If yes, how was it included in the context? If not, is it fair to compare BigBird and StreamingLLM with StreamingDialogue?  \n  \nAll of the baselines and our method did not use persona profiles to generate the responses. Therefore, our experimental comparison is fair.  \n  \n> The use of the BLEU metric is not consistent. It would be better to show BLEU,  BLEU-1 and BLEU-2 in all the result tables.  \n  \nThank you for your suggestion. We have reported BLEU,  BLEU-1 and BLEU-2 in all the result tables as follows:  \n  \n| **Data** | **Method** | **BLEU** | **BLEU-1** | **BLEU-2** |  \n|-|-|-|-|-|  \n| PersonaChat | Dense | 13.15 | 49.30 | 20.05 |  \n|| Local | 13.01 | 50.78 | 20.13 |  \n|| Big Bird | 12.93 | 50.00 | 20.52 |  \n|| StreamingLLM | 13.16 | 50.15 | 20.68 |  \n|| MemBART | 11.18 | 46.63| 17.65 |  \n|| Ours| 13.63 | 51.27| 20.77 |  \n|MSC|Dense|19.47|52.22 | 28.41 |  \n||Local| 13.34 | 41.14| 20.44 |  \n|| Big Bird| 16.54 | 46.63| 24.77 |  \n|| StreamingLLM|16.76 | 47.54 | 25.08 |  \n|| MemBART|17.11 |49.78 | 25.82 |  \n|| Ours|19.33 | 51.49 | 28.12 |  \n  \n*Table 1: Main results on the PersonaChat and MSC datasets.*  \n  \n|**Model**|**BLEU**|**BLEU-1**|**BLEU-2**|  \n|-|-|-|-|  \n|Ours|19.33|51.49|28.12|  \n|Base|17.32|47.41|25.61|  \n|LMR|18.87|50.83|27.76|  \n|SMR|18.25|49.45|26.84|  \n  \n*Table 2: Ablation results on MSC with different learning strategies.*  \n  \n|**Method**|**BLEU**|**BLEU-1**|**BLEU-2**|  \n|-|-|-|-|  \n|StreamingLLM|20.16|51.18|29.99|  \n|Ours|20.19|51.55|30.03|  \n*Table 3: Results under the non-training setting on the MSC test set.*  \n  \n|**BLEU**|**BLEU-1**|**BLEU-2**|  \n|-|-|-|  \n|68.02|89.19|76.83|\n*Table 4: Dialogue reconstruction performance.*  \n\nWe will include these results in the revision. Thank you for the great advice!\n  \n> More datasets are needed (Weakness 5)\n\nThank you for the constructive feedback.\n\nWe have conducted experiments on Topical-Chat and MultiWOZ. The results are shown in the table below.\n|**Data**|**Method**|**PPL**|**ROUGE-1**|**ROUGE-2**|**ROUGE-L**|**Dial-M**|\n|-|-|-|-|-|-|-|\n| Topical-Chat | Dense |9.49|15.70|3.65|14.88|3.09|\n|| Local |27.55|12.60|2.09|10.37|7.02|\n|| Big Bird | 10.36 |14.21|3.55|11.79|3.01|\n|| StreamingLLM | 10.34 | 14.25|3.55|11.84|3.05|\n|| MemBART |12.54|13.86|2.98|13.18|2.83|\n|| Ours|9.80|15.46|3.99|14.37|2.66|\n|MultiWOZ|Dense|4.51|24.79|13.93|24.67|2.27|\n||Local| 5.38 | 24.26|13.47|24.15|2.45|\n|| Big Bird| 4.79 | 24.38|13.26|24.30|2.51|\n|| StreamingLLM|4.76 | 23.66|13.09|23.41|2.47|\n|| MemBART|5.36|20.05|12.41|19.94|2.37|\n|| Ours|4.34|25.26|14.27|25.20|2.25|\n\nOur method outperforms all strong baselines due to its ability to retain more complete historical information.\n  \nWe will include these results in the revision.  \n  \n> Why are Layer 0 and Layer 1 shown in Fig. 1a, whereas Layer 28 is shown in Fig. 1b?  \n\nWe have added the Layer 0 and Layer 1 attention maps for Fig. 1b, as detailed in Figs. (a)-(d) of the PDF in *global response*.\n  \n> Did the authors analyze the conv-attn sinks for the example in Fig. 7?  \n\nIn *global response*, Figs (e)-(f) of the PDF illustrate the model's attention to conv-attn sinks for the example shown in Fig. 7. During the inference stage, only the key-values of the conv-attn sinks are retained. In the case study of Fig. 7, the generated response requires the critical information \"California\" from the 12th and 14th utterances. We observed that more attention is allocated to the conv-attn sinks of the 12th and 14th utterances, indicating that during inference, the model focuses more on the conv-attn sinks that are useful for the inference.\n  \nThank you once again for your valuable time and constructive feedback to improve our paper. We are eager to address any additional questions or concerns that you may have.  \n\nIf your concerns have been addressed, we would be very grateful if you could raise the score.\n  \n**References**  \n[1] [Measuring nominal scale agreement among many raters]([https://psycnet.apa.org/record/1972-05083-001) (Fleiss, J. L., Psychological Bulletin 1971)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Adl2B4KdWb", "reviewer": "Reviewer_4bTd", "summary": "This paper tackles the challenge of long context dependencies of LLMs in dialogue settings. The authors first posit that end of utterance tokens like \"\\n\" and 
 could conceivably summarize the information in the utterance, and propose to attend to such sinks rather than entire utterances to allow LLMs to perform long-context tasks with lower computational costs. Two learning strategies, SMR and LMR are introduced to encourage EoU tokens to carry key information from preceding utterances, and remember key information from previous EoU tokens. The authors evaluate on multiple dialog datasets, and demonstrate that the proposed approach can significantly improve computational complexity and make it possible to operate on longer dialogs.", "questions": "1. Line 111: Yes, position interpolation based approaches may not provide for infinitely long sequences, but is that required for dialogue tasks? Perhaps this could be rephrased to make the authors point. \n\n2. Line 119 makes a leap in logic that is perhaps not fully intuitive. The authors claim that higher attention on EoU tokens suggests that these aggregate information. Did the authors test whether the responses of the LLM to conversations implied that high attentions on EoU tokens could possibly capture information from the preceding utterances ?\n\n3. Line 160: Did the authors consider restricting the attention for u' tokens to just the sink token, rather than sink tokens and previous tokens? The current phrasing seems to indicate the former over the latter.", "limitations": "The authors are requested to add notes on limitations and potential negative social impacts. The additional training methods proposed may incur computational cost, and I encourage the authors to comment on this.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem that the paper is tackling is important for the community at large, and the proposed approach seems intuitive and logical. \n2. The paper is original, well structured and written, and experiments appear apt to substantiate the authors claims.", "weaknesses": "1. The authors do not compare against plausible alternative approaches like infinity former or the compressive transformer, which are also memory based approaches. Comparisons to position interpolation based approaches with RoPE would also be interesting to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4bTd,\n\nWe sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.\n\n> More baselines are needed (Weakness 1).\n\nYour kind advice has inspired us to conduct more comprehensive experiments by incorporating the recommended baselines: infinity former and two position interpolation-based approaches with RoPE, which are YaRN [1] and Dynamic NTK-RoPE [2]. Our method achieves favorable results compared to infinity former across all metrics and outperforms YaRN and Dynamic NTK-RoPE on some metrics. Since the lengths of the MSC and PersonaChat datasets are within the training length of Llama, it is reasonable that our method only shows advantages in some metrics relative to the full-attention RoPE position interpolation methods.\n|**Data**|**Method**|**PPL**|**BLEU**|**R-1**|**R-2**|**R-L**|**D-2**|\n|:-|:-|:-|:-|:-|:-|:-|:-|\n|MSC|Dense|7.58|19.47|16.93|2.92|15.48|37.75|\n||YaRN|7.84|19.68|14.99|2.37|11.18|42.29|\n||Dynaminc NTK|7.58|19.95|15.61|2.62|11.66|39.39|\n||Infinity former|16.61|6.90|8.21|0.28|7.97|7.92|\n||Ours|7.99|19.33|**17.18**|**2.77**|**15.86**|32.58|\n|PersonaChat|Dense|8.41|13.15|13.98|3.07|13.44|41.61|\n||YaRN|8.25|13.28|13.09|2.85|12.15|41.89|\n||Dynaminc NTK|8.24|13.40|13.21|3.00|12.30|41.99|\n||Infinity former|15.83|9.27|10.06|0.73|9.76|26.30|\n||Ours|8.71|**13.63**|**13.96**|**3.05**|**13.43**|37.23|\n\n> Line 111: Yes, position interpolation based approaches may not provide for infinitely long sequences, but is that required for dialogue tasks? Perhaps this could be rephrased to make the authors point.\n\nIdeally, our objective is to develop a lifelong dialogue system capable of continuous conversation while retaining memory of all past dialogues. The term *infinitely long* refers to the cumulative length of each and every utterance, not the length of a single utterance. Position interpolation-based approaches lack this capability.\n\n> Line 119 makes a leap in logic that is perhaps not fully intuitive. The authors claim that higher attention on EoU tokens suggests that these aggregate information. Did the authors test whether the responses of the LLM to conversations implied that high attentions on EoU tokens could possibly capture information from the preceding utterances ?\n\nWe tested and confirmed that high attention to EoU tokens effectively captures information from previous dialogues. The answer is yes, and we illustrate this with the following case study.\n\nUsing an untrained Llama-2-7B-Chat model, we restrict it during inference to focus only on EoU tokens from previous utterances and the last complete utterance. Given the input \"Did you have a caramel macchiato today?&lt;/s&gt;Yes!&lt;/s&gt;What kind of coffee did you have today?&lt;/s&gt;,\" the model responds with \"I'm glad you asked! I had a delicious caramel macchiato this morning.\" This shows that the EoU tokens successfully capture the key information \"caramel macchiato\" from the first utterance.\n\n> Line 160: Did the authors consider restricting the attention for u' tokens to just the sink token, rather than sink tokens and previous tokens? The current phrasing seems to indicate the former over the latter.\n\nThere may be some misunderstandings due to unclear expressions. To clarify, in short-memory reconstruction, $u'$ can indeed only see the conv-attn sink token of $u$ to reconstruct $u$, and the conv-attn sink of $u$ can attend to $u$ to compress $u$'s information onto itself.\n\n> The authors are requested to add notes on limitations and potential negative social impacts. The additional training methods proposed may incur computational cost, and I encourage the authors to comment on this.\n\nThank you very much for your valuable suggestion. StreamingDialogue significantly reduces space and time complexity during the inference stage. Additionally, we can outperform the baseline under the non-training setting without additional cost.\n\nTo optimize LLMs for the conv-attn sinks mode, we implement two learning strategies: short-memory reconstruction and long-memory reactivation. Consequently, this inevitably increases computational costs under the training setting, with the SMR and LMR phases requiring about two hours on two A100-40G GPUs. We will include additional details on computational costs in the limitations section of the revision.\n\nThank you for your instant feedback and valuable revision advice. The points you raised are worthy to ponder upon. We are so glad to make a further discussion on those issues. Hope our response will address your concern.\n\n**References**  \n[1] [YaRN: Efficient Context Window Extension of Large Language Models](https://arxiv.org/pdf/2309.00071)\n\n[2] https://github.com/jquesnelle/yarn/pull/1", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjQh25zVYL", "reviewer": "Reviewer_wRZP", "summary": "Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. This paper finds that the structure of the dialogue context is consistent, and special tokens may aggregate information. Therefore, this paper aims to use special tokens to encode dialogue history information to reduce inference costs and enhance the ability of LLMs to handle long dialogues. To achieve this, it proposes two information reconstruction functions to improve the information aggregation capability of special tokens. Experiments have shown that the article has achieved its intended purpose.", "questions": "1. In line 127-128, the authors mention that by caching only the corresponding conv-attn sinks, the time complexity of attention computation can be reduced from $O(T^2L^2)$ to $O(T^2)$. However, have the authors considered that if each utterance contains $L$ tokens, the time complexity should be $\\mathbf{O(T^2L)}$?\n\n2. For the same sample, it would be best for the authors to use the same notation on line 157 and line 175 to avoid confusion.", "limitations": "The authors adequately address the limitations and, if applicable, potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper finds that the separator tokens, namely conversational attention sinks, generally aggregates more attention than other words and tokens. Therefore, this paper proposes StreamingDialogue, which utilizes there special tokens to enhance the capability of LLMs to handle long-context dialogue.\n2. StreamingDialogue has achieved good results on multiple dialogue datasets. Analysis experiments also demonstrate that this method is capable of handling long-context dialogues and can reduce inference latency and memory usage.", "weaknesses": "1. The training method introduced by StreamingDialogue is overly complex, resulting in the model performing multiple forward passes on the same sample during training, leading to more than three times the training cost. \n2. The comparison of different methods is unfair. In the main experiment, StreamingDialogue is fine-tuned on specific datasets, making it evident that it can surpass the non-training method StreamingLLM [1]. Although the authors also explored the non-training setting of the proposed method in Section 4.6, they only conducted experiments on Llama2-7B-chat and evaluated it using only 1-gram and 2-gram metrics. Therefore, the effectiveness and generalizability of the method are not verified.\n\n\n[1] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wRZP,  \n  \nWe sincerely thank you for your constructive suggestions and valuable feedback! We hope our response can help resolve your concerns.\n\n> The training method is complex, resulting in higher training costs.\n  \nOur method outperforms baselines in both training and non-training settings and the non-training setting incurs no additional costs. The training method we introduced is intended to better adapt the model to the conv-attn sink mode. This results in a trade-off between significantly improved performance and increased training costs. However, regardless of the choice, our method consistently performs better than baselines.\n\nAdditionally, during the inference stage, our method can significantly reduce space and time complexity by compressing historical dialogues into conv-attn sinks. Below is a comparison of our method with StreamingLLM in both training and non-training settings.\n|**Method**|**BLEU**|**BLEU-1**|**BLEU-2**|**ROUGE-1**|**ROUGE-2**|**ROUGE-L**|\n|-|-|-|-|-|-|-|\n|StreamingLLM (non-training)|20.16|51.18|29.99|15.90|1.92|14.26|\n|Ours (non-training)|20.19|51.55|30.03|16.46|2.11|15.00|\n|StreamingLLM (training)|16.76|47.54|25.08|15.25|2.44|14.21|\n|Ours (training)|19.33|51.49|28.12|17.18|2.77|15.86|\n\n> (1) The comparison of different methods is unfair; (2) More base models and metrics are needed for the non-training setting.\n\n(1) We have made every effort to ensure all comparisons are conducted fairly. There may be some misunderstandings: in the main experiment, all methods were maintained with the same training settings and were fine-tuned on specific datasets, including StreamingLLM. We will further clarify the experimental settings in the revision. Thank you for the suggestion.\n  \n(2) Thank you for the great advice! We have added tests on the Llama-3-8B-Instruct and Mistral-7B [1] under the non-training setting and included additional metrics: BLEU and ROUGE-L.\n|**Model**|**Method**|**BLEU**|**BLEU-1**|**BLEU-2**|**ROUGE-1**|**ROUGE-2**|**ROUGE-L**|\n|-|-|-|-|-|-|-|-|\n|Llama-2-7B-Chat|StreamingLLM|20.16|51.18|29.99|15.90|1.92|14.26|\n||Ours|20.19|51.55|30.03|16.46|2.11|15.00|\n|Llama-3-8B-Instruct|StreamingLLM|16.48|39.68|24.63|16.88|1.93|15.47|\n||Ours|16.77|40.10|24.88|17.11|2.01|15.85|\n|Mistral-7B|StreamingLLM|12.75|42.86|19.99|12.58|1.83|11.73|\n||Ours|13.33|44.08|20.65|13.40|1.98|12.58|\n\n> In line 127-128, the authors mention that by caching only the corresponding conv-attn sinks, the time complexity of attention computation can be reduced from $O(T^2L^2)$ to $O(T^2)$. However, have the authors considered that if each utterance contains $L$ tokens, the time complexity should be $O(T^2L)$?\n\nAfter our thorough verification and calculations, the time complexity is indeed $O(T^2L)$. We sincerely appreciate your suggestion and will revise the complexity in the revision.\n\n> For the same sample, it would be best for the authors to use the same notation on line 157 and line 175 to avoid confusion.\n\nThank you for your suggestion. We will use the same notation in the revision for lines 157 and 175.\n\nOnce again, we appreciate your efforts and valuable suggestions to improve our paper. If you have further questions, please leave more comments in the OpenReview system. We would appreciate it very much if you could kindly raise your score if your concerns are addressed!\n\n**References**  \n[1] [Mistral 7B](https://arxiv.org/pdf/2310.06825)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We greatly appreciate the time and effort all reviewers devoted to reviewing our paper and providing detailed, constructive feedback. The reviewers' insights and queries have played a crucial role in helping us refine our research. We have thoughtfully considered feedback from all reviewers and hope these responses address their concerns.

We have included a PDF in the global response to address the concern raised by Reviewer 7W1D.

Meta-review: This paper builds on an interesting observation regarding the model's attention to the end-of-utterance (EoU) units and proposes compressing dialogue turns into EoUs. Experiments demonstrate the benefit of the proposed method on two dialogue datasets, which are extended to two other dialogue data with similar results, based on reviewer suggestions.
Decision: Accept (poster)