SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Results of translation task on JRC-Acquis measured by BLEU. Models denoted by the same symbol ( $\star$  and  $\dagger$ ) have the same parameters and only differ in memory as input. The bolded numbers show the SOTA performance and the underlined numbers show the second-best result. * denotes the system is significantly better than baselines with  $p$ -value  $< 0.05$  tested by [37].
System	Es→En	En→Es	De→En	En→De
Dev	Test	Dev	Test	Dev	Test	Dev	Test
None Memory
RNNsearch [3]	55.02	59.34	50.54	50.48	50.20	49.74	44.94	43.98
Transformer [84]	64.08	64.63	62.02	61.80	60.18	60.16	54.65	55.43
Retrieval Memory
SEG-NMT [28]	60.28	59.34	57.62	57.27	55.63	55.33	49.26	48.80
NMT-pieces [101]	63.97	64.30	61.50	61.56	60.10	60.26	55.54	55.14
G-TFM [92]	66.37	66.21	62.50	62.76	61.85	61.72	57.43	56.88
MonoNMT [8]	67.73	67.42	64.18	63.86	64.48	64.62	58.77	58.42
CMM [17]	67.48	67.76	63.84	64.04	64.22	64.33	58.94	58.69
Transformerdual★	66.87	67.12	63.14	63.54	64.09	63.36	58.69	58.06
Transformeruni†	67.74	67.32	63.93	64.12	64.50	64.40	58.16	58.58
Self-Memory
Transformerdual★	68.63*	69.20*	64.12*	64.67*	65.06*	64.98*	59.26*	59.49*
Transformeruni†	68.26*	68.80*	66.07*	65.94*	65.32*	65.65*	59.88*	60.11*
Table 3: Comparison between retrieval memory and self-memory. The quality of memory and hypothesis is measured by the n-gram overlap with reference (BLEU). All experiments are conducted with Transformer<sub>joint</sub> on JRC-Acquis.
Retrieval	Self
memory	hypothesis	memory	hypothesis
En-De	→	38.89	58.58	57.92	60.11
←	42.56	64.40	64.32	65.65
En-Es	→	40.67	64.12	63.57	65.94
←	43.05	67.32	67.78	68.80
The dual problem is revealed in Table 3. Self-memory, which essentially represents the model's own output, exhibits greater similarity with the ground truth and serves as a more effective memory for generating the final output. This observation highlights a key distinction between Selfmem and previous reranking works [39, 68]. Reranking aims to select candidates of higher quality than the beam output, whereas in Selfmem, the chosen candidates serve as memory for the retrieval-augmented generator and do not necessarily need to surpass the quality of the beam hypotheses.
Table 4: Evaluation results of in-context learning with self-memory.
XGLM-1.7B	XGLM-4.5B	XGLM-7.5B
Random	kNN	Self	Random	kNN	Self	Random	kNN	Self
En-De	→	11.51	37.87	40.94	17.51	37.60	38.25	18.48	47.82	48.32
←	27.42	51.00	51.88	30.62	48.12	48.36	33.03	55.65	55.12
En-Es	→	23.87	46.20	48.56	31.83	48.37	49.17	29.97	53.86	54.32
←	25.29	51.55	53.13	32.16	48.55	49.22	35.22	57.25	57.56
In Table 4, we present the results of LLM with self-memory. We employ XGLM [48] as our backbone generator, with three different sizes ranging from 1.7B to 7.5B. We utilize the recommended prompt as described in [48]. We select three in-context learning examples and report the average scores from three separate runs, taking into account the sensitivity of example selection in ICL [49]. From the table, we first observe a general trend where few-shot translation performance improves as the
size of the model increases. Furthermore, we find that more similar translation demonstrations significantly enhance performance across all model sizes (from random, kNN to Self). This suggests that demonstration examples in in-context learning not only act as triggers for model ability but also adhere to the primal problem, where better demonstration example leads to better generation. Also, by comparing the results in Table 2 and Table 4, we can conclude that the cross-lingual LLM with designed examples still falls short of the supervised baselines in this task.
5.2 Summarization
In this paper, we compare the performance of our trainable model with those of REINA [87], PEGASUS [100], and BART [40]. The results are presented in Table5. Initially, it can be observed that memory has varying impacts on different datasets. The enhancement brought by memory in the BigPatent dataset is significantly larger than that in the XSum dataset. This can be attributed to the inherent characteristics of the BigPatent dataset, which consists of official patent documents that exhibit considerable similarity. Consequently, this greatly improves the summarization quality in accordance with the primal problem. Furthermore, we discovered that self-memory substantially enhances the performance of both BRIO (+1.2 R1) and BART (+18.5 R1), achieving state-of-the-art results on both datasets. We selected these baselines for a fair comparison, as they share the same base generator. Due to space constraints, additional comparisons and the confidence region of the SOTA model can be found in the Appendix E.
Table 5: Results of summarization task on XSum and BigPatent measured by ROUGE.
System	Memory	R-1	R-2	R-L
XSum
PEGASUS	None	47.2	24.6	39.3
BRIO	None	49.1	25.6	40.4
REINA (PG)	Retrieval	48.2	26.0	40.2
REINA (B)	Retrieval	43.2	21.0	35.5
REINA (L)	Retrieval	46.5	24.1	38.6
\( BRIO_{dual} \star \)	Retrieval	48.6	26.1	40.6
\( BRIO_{joint} \dagger \)	Retrieval	49.5	26.5	41.2
\( BRIO_{dual} \star \)	Self	49.2	26.2	40.8
\( BRIO_{joint} \dagger \)	Self	50.3	26.7	41.6
System	Memory	R-1	R-2	R-L
BigPatent
PEGASUS	None	53.6	33.2	43.2
BART	None	44.4	21.3	31.0
REINA (B)	Retrieval	59.5	42.6	50.6
REINA (L)	Retrieval	60.7	43.3	51.3
REINA (PG)	Retrieval	44.6	21.5	33.3
BARTdual★	Retrieval	57.4	43.3	49.7
BARTjoint†	Retrieval	59.6	43.4	51.0
BARTdual★	Self	61.2	44.6	52.3
BARTjoint†	Self	62.9	48.1	59.6
5.3 Dialogue Generation
As demonstrated in Table 6, the self-memory significantly enhances the performance of the retrieval-augmented generator for dialogue generation tasks. By optimizing memory using BLEU as  $\Delta (\cdot ,\cdot)$ , the self-memory improves the B-1,2 score over retrieved memory by 3.08 B-1 and 0.6 B-2 on  $\mathrm{BART}_{\mathrm{joint}}$ . Intriguingly, although Selfmem surpasses the baselines in terms of B-1/2, it falls behind in D-1 and D-2, which can be attributed to the trade-off between BLEU score and Distinct score when evaluating a dialogue system [104]. To address this issue, we opt for D-1,2 as  $\Delta (\cdot ,\cdot)$  when optimizing  $S_{\theta}$ , denoted as  $\mathrm{BART}_{\mathrm{joint}}\dagger (\mathrm{D})$ . The results in Table 6 highlight the remarkable flexibility of Selfmem by directly optimizing memory to achieve the desired attributes for diverse and informative dialogue.
6 Further Analysis
To gain a deeper insight into Selfmem, we first examine the impact of each key component, namely  $G_{\xi}$  and  $S_{\theta}$ . Subsequently, we perform a detailed token-level analysis of the generated output concerning their frequency in the training set. Experiments are conducted on the JRC-Acquis En→De dataset. We also include latency analysis and human evaluation on Appendix F and G.
Tuning  $S_{\theta}$  We explored various  $S_{\theta}$  by direct selection from the candidate pool based on gold rankings. As shown in Figure 3a, both architectures with enhanced  $S_{\theta}$  significantly outperform the current SOTA performance (60.11 BLEU). Moreover, we assessed the candidate pool quality during this iterative process using an oracle  $S_{\theta}$ , as displayed in Figure 3b. A clear pattern emerges
Table 6: Results of dialogue generation task on DailyDialog measured by B-1/2 and D-1/2.  $\mathrm{BART}_{\mathrm{joint}}$  (D) denotes the metric  $\Delta (\cdot ,\cdot)$  for  $S_{\theta}$  is the average of D-1 and D-2.
System	Memory	B-1	B-2	D-1	D-2
NCM [86]	None	33.60	26.80	3.00	12.80
iVAE [25]	None	30.90	24.90	2.90	25.00
PLATO-2 [5]	None	34.80	25.12	3.54	25.11
DiaLoFlow [45]	None	36.17	27.67	4.56	27.12
BART	None	20.72	11.36	3.92	19.44
BARTdual★	Retrieval	29.50	21.89	4.74	26.01
BARTjoint†	Retrieval	36.72	31.55	6.13	35.65
BARTdual★	Self	33.43	22.85	4.66	26.16
BARTjoint†	Self	39.80	32.15	5.84	32.16
BARTjoint† (D)	Self	36.92	32.09	9.12	37.05
(a) Hypothesis
Figure 3: (a) shows generation quality in the iteration process with different  $S_{\theta}$  in both trainable generator architectures. (b) shows candidates quality in the iteration process with an oracle  $S_{\theta}$ .
(b) Candidates
in this boxplot, revealing improvements in the oracle, quartile, average, and minimum scores of the candidate pool. These two experiments jointly clarify the Selfmem's underlying intuition: a retrieval-augmented generator profits from superior memory, which can be chosen from its own unbounded output, and subsequently, the generator with improved memory produces a higher-quality candidate pool for the next selection round. Consequently, the model lifts itself up.
Tuning  $G_{\xi}$  As discussed in §3.1, we demonstrated that a trained retrieval-augmented generator, with fixed parameters, possesses the ability to distinguish between "good" and "bad" memory. This observation not only justifies our decision to maintain a fixed generator within our framework but also implies that the  $G_{\xi}$  is not the current bottleneck of the Selfmem.
Figure 4: 1-gram F1 score sorted by training corpus frequency.
Frequency Analysis We conduct a comprehensive token-level analysis by computing the 1-gram F1 scores for generated translations and subsequently categorizing the tokens based on their frequency in the training set. The results are depicted in Figure 4. A noticeable pattern emerges, suggesting that the more frequently a model encounters a token during training,
the higher the accuracy of the generated output [102]. Moreover, our findings indicate that retrieval-augmented models, particularly those incorporating self-memory augmentation, exhibit superior performance in handling long-tail inputs which are challenges for parametric models [67, 57].
For the first time, we investigate the fundamental limitation of bounded memory in the current retrieval-augmented literature. We combine the primal and dual problems together and propose Selfmem, a general framework for retrieval-augmented text generation by uplifting generation model with its own output. We conduct comprehensive experiments across various text generation tasks and different generation paradigms, including trainable small model and few-shot prompted LLM. We surpass strong baselines and improve the state-of-the-art performance in serval datasets. We also meticulously investigate each crucial component and pinpoint the existing system bottleneck to guide future research endeavors.
We discuss the limitations of our framework as follows:
(1) Although Selfmem greatly improves the generation quality compared with other retrieval-augmented generation models, it requires more computational resources with respect to the memory selection process. For large dataset with long context (e.g., BigPatent), it would become a more crucial problem considering the quadratic time complexity of transformer architecture.
(2) This paper proposes a general idea for the retrieval-augmented generation. But we only experiment with transformer-based architecture for both generator and memory selector and the architecture of generator and memory selector keeps the same across all text generation tasks. We believe the task-specific design for the model architecture, training objective and generation methods in different text generation scenarios would further improve the performance.
This work was supported by the National Key Research and Development Program of China (No.2021YFC3340304) and National Natural Science Foundation of China (NSFC Grant No.62122089). We appreciate the anonymous reviewers for their helpful comments. Dongyan Zhao and Rui Yan are the corresponding authors.
[1] Armen Aghajanyan, Anchit Gupta, Akshit Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proc. of EMNLP, 2021.
[2] Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. In-context examples selection for machine translation. CoRR, 2022.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR, 2015.
[4] Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. PLATO: pre-trained dialogue generation model with discrete latent variable. In Proc. of ACL, 2020.
[5] Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. PLATO-2: Towards building an open-domain chatbot via curriculum learning. In Proc. of ACL Findings, 2021.
[6] Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. Skeleton-to-response: Dialogue generation guided by retrieval memory. In Proc. of NAACL, 2019.
[7] Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, and Shuming Shi. Retrieval-guided dialogue response generation via a matching-to-generation framework. In Proc. of EMNLP, 2019.
[61] Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alexander M. Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir R. Radev. A smorgasbord of features for statistical machine translation. In Proc. of NAACL, 2004.
[62] OpenAI. GPT-4 technical report. CoRR, 2023.
[63] Md. Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. In Proc. of EMNLP Findings, 2021.
[64] Hao Peng, Ankur P. Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. Text generation with exemplar-based adaptive decoding. In Proc. of NAACL, 2019.
[65] Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. On extractive and abstractive neural document summarization with transformer language models. In Proc. of EMNLP, 2020.
[66] Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - November 1, 2018, 2018.
[67] Vikas Raunak, Siddharth Dalmia, Vivek Gupta, and Florian Metze. On long-tailed phenomena in neural machine translation. In Proc. of EMNLP Findings, 2020.
[68] Mathieu Ravaut, Shafiq R. Joty, and Nancy F. Chen. Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In Proc. of ACL, 2022.
[69] Mathieu Ravaut, Shafiq R. Joty, and Nancy F. Chen. Towards summary candidates fusion. CoRR, 2022.
[70] Stephen E. Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 2009.
[71] Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proc. of ACL, 2020.
[72] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.
[73] Eva Sharma, Chen Li, and Lu Wang. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proc. of ACL, 2019.
[74] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Proc. of ICML, 2018.
[75] Libin Shen, Anoop Sarkar, and Franz Josef Och. Discriminative reranking for machine translation. In Proc. of NAACL, 2004.
[76] Weizhou Shen, Yeyun Gong, Yelong Shen, Song Wang, Xiaojun Quan, Nan Duan, and Weizhu Chen. Joint generator-ranker learning for natural language generation. CoRR, 2022.
[77] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models, 2023.
[78] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: retrieval-augmented black-box language models. CoRR, 2023.
[79] Suzanne Sia and Kevin Duh. In-context learning as maintaining coherency: A study of on-the-fly machine translation using large language models. CoRR, 2023.
[98] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In Proc. of NeurIPS, 2020.
[99] Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, and Fei Sha. Generate-and-retrieve: Use your predictions to improve retrieval for semantic parsing. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4946-4951, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
[100] Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proc. of ICML, 2020.
[101] Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Graham Neubig, and Satoshi Nakamura. Guiding neural machine translation with retrieved translation pieces. In Proc. of NAACL, 2018.
[102] Tong Zhang, Wei Ye, Baosong Yang, Long Zhang, Xingzhang Ren, Dayiheng Liu, Jinan Sun, Shikun Zhang, Haibo Zhang, and Wen Zhao. Frequency-aware contrastive learning for neural machine translation. In Proc. of AAAI, 2022.
[103] Xueliang Zhao, Lemao Liu, Tingchen Fu, Shuming Shi, Dongyan Zhao, and Rui Yan. Towards efficient dialogue pre-training with transferable and interpretable latent structure. CoRR, 2022.
[104] Yinhe Zheng, Zikai Chen, Rongsheng Zhang, Shilei Huang, Xiaoxi Mao, and Minlie Huang. Stylized dialogue response generation using stylized unpaired texts. In Proc. of AAAI, 2021.
[105] Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. CoRR, 2022.
Table 7: Dataset statistics for three tasks.
Task	Dataset	#Train	#Dev	#Test
Translation	JRC (en ↔ de)	663,487	2,454	2,483
JRC (en ↔ es)	653,127	2,533	2,596
Summarization	BigPatent	1,207,222	67,068	67,072
XSum	204,045	11,332	11,334
Dialogue	DailyDialog	87,170	8,069	7,740
B Self Memory Details
For machine translation tasks, following [93, 8, 17] we use randomly initialize  $\mathrm{Transformer}_{\mathrm{base}}$  architecture [84] as  $G_{\xi}$ . We use the joint-bpe algorithm [72] and share the parameters between the memory encoder and source encoder for dual encoder architecture. The hyper-parameter setting follows [17] with dropout 0.1, label smoothing 0.1, gradient clipping 1.0, Adafactor [74], warm-up steps 4000, maximum learning rate 4.4e-2 and training epochs 30 for total. The evaluation metrics are BLEU, TER and  $\mathrm{chrF++}$  from SACREBLEU [66]. The backbone of memory selector  $S_{\theta}$  is XLM- $\mathbf{R}_{\mathrm{base}}$  [22] with BLEU as  $\Delta (\cdot ,\cdot)$ . The hyper-parameter setting for  $S_{\theta}$  follows [39] with  $\tau 0.5$ , minmax normalization for candidates ranking, Adam optimizer with max learning rate 5e-5 and polynomial decay scheduler, and classifier dropout 0.2.
For Summarization, we init the  $G_{\xi}$  with BART<sub>base</sub> [40] for BigPatent following [87] and state-of-the-art BRIO [55] for XSum. Optimization is based on Adafactor with a maximum learning rate of 5e-3, warm-up steps 10000 and gradient clipping value 1.0. The maximum input length is 512 for XSum and 1024 for BigPatent. The evaluation metric is Rouge (R-1/2/L) [47].
For Dialogue Generation, we use  $\mathrm{BART}_{\mathrm{base}}$  as the backbone for  $G_{\xi}$  on DailyDialog. We tune the hyper-parameters from learning rate  $\{5\mathrm{e} - 3,1\mathrm{e} - 3,4\mathrm{e} - 4\}$  and set dropout 0.1, batch size 64, label smoothing factor 0.1, maximum input length 120 for DailyDialog. Following [4, 13], we evaluate our dialogue system with BLEU (B-1/2) and Distinct (D-1,2) [43]. For both Summarization and Dialogue Generation task, we follow [54, 26] and adopt RoBERTa<sub>base</sub> [52] as the backbone for  $S_{\theta}$ . We choose the linear combination of B-1/2 as  $\Delta(\cdot, \cdot)$  for Dialogue Generation and R-1/2/L for Summarization following [76]. We tune the hyper-parameters  $\tau$  from  $\{0.08,0.2,0.5,0.8\}$ , learning rate from  $\{5\mathrm{e} - 5,7\mathrm{e} - 5,2\mathrm{e} - 4\}$ . The maximum input length for  $S_{\theta}$  is 512 and we truncate tokens from the longer input of source and candidate.
Machine Translation We evaluate our MT system with BLEU, TER and  $\mathrm{chrF}++$  from SACRE-BLEU $^3$  [66]. The signatures for BLEU, TER and  $\mathrm{chrF}++$  are shown in Table 8.
Table 8: Signature from SACREBLEU.
[c]Signature
nrefs:1|case:mixedleff:noltok:13alsmooth:explversion:2.0.0
nrefs:1|case:lcltok:tercomlnorm:nolpunct:yeslasiannolversion:2.0.0
nrefs:1|case:mixedleff:yeslnc:6lnw:2|space:nolversion:2.0.0
<sup>3</sup>https://github.com/mjpost/sacrebleu.git
Summarization We evaluate our Summarization system with standard ROUGE [47] Perl package for evaluation. Following [55], we use PTB tokenizer<sup>5</sup> for tokenization. And the parameters for ROUGE are "-c 95 -r 1000 -n 2 -m".
Dialogue Generation Following [27], we evaluate our dialogue system with NLTK BLEU  $^6$  with space as tokenizer and smoothing method1. The Distinction score is from [42].
D More results on translation tasks
Table 9: Evaluation results on JRC-Acquis En→De measured by BLEU, TER and chrF++.
System	Memory	BLEU ↑	chrF++ ↑	TER↓
Transformer	None	55.43	70.31	36.35
\(Transformer_{dual}\)	Retrieval	58.06	71.58	35.41
\(Transformer_{joint}\)	Retrieval	58.58	72.22	34.39
\(Transformer_{dual}\)	Self	59.49	72.62	34.04
\(Transformer_{joint}\)	Self	60.11	73.25	32.62
E More Summarization Baselines
In this Table 10, we include more baselines on the benchmark dataset XSum and BigPatent. We also report the confidence region of SOTA model for XSum and BigPatent as shown in Table 11.
Table 10: More baselines on XSum and BigPatent.
System	R-1	R-2	R-L
XSum
[51]	38.8	16.5	31.3
[40]	45.1	22.3	37.3
[100]	47.2	24.6	39.3
[54]	47.6	24.6	39.4
[55]	49.1	25.6	40.4
[87](PG)	48.2	26.0	40.2
[87](B)	43.1	21.0	35.5
[87](L)	46.5	24.1	38.6
[68]	48.1	25.0	40.0
[69]	47.1	24.1	38.8
[16]	47.8	25.0	39.7
Selfmem	50.3	26.7	41.6
System	R-1	R-2	R-L
BigPatent
[100]	53.6	33.1	42.3
[40]	44.4	21.3	31.0
[98]	60.6	42.5	50.0
[65]	38.7	12.3	34.1
[90]	45.0	20.3	39.2
[1]	52.3	33.5	42.8
[87] (B)	59.5	42.6	50.6
[87] (L)	60.7	43.3	51.3
[87] (PG)	44.6	21.5	33.3
Selfmem	62.9	48.1	59.6
F Empirical analysis of latency
In Table 12, we present empirical results of Selfmem latency, measured in seconds. We compare Selfmem with a retrieval-augmented baseline model across various datasets and computational platforms, including CPU and CUDA. The number of iterations for Selfmem is set to one. All experiments are conducted on the same device, equipped with one NVIDIA A100 GPU and one AMD EPYC 7V13 64-Core Processor.
4https://github.com/summanlp/evaluation/tree/master/ROUGE-RELEASE-1.5.5
<https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/process/PTBTokenizer.html>
$^{6}$ https://www.nltk.org/_modules/nltk/translate/bleu_score.html
Table 11: Confidence region for SOTA model in XSum and BigPatent.
System	ROUGE-1/2/L	95%-conf.int
XSum
BRIOjoint	50.3	0.49986 - 0.50602
26.7	0.26300 - 0.26989
41.6	0.41231 - 0.41900
BigPatent
BARTjoint	62.9	0.62664 - 0.63080
48.1	0.47783 - 0.48333
59.6	0.59401 - 0.59847
Table 12: Generation Latency analysis.
NMT	XSum	BigPatent	DailyDialog
Average Input Length	87	512	1024	71
Average Output Length	44	75	127	16
CPU
Retrieval-augmented Baseline	0.97	1.79	3.16	0.32
Selfmem	Candidate Generation	3.20	7.50	15.00	1.02
Memory Selection	0.50	0.52	0.95	0.14
Hypothesis Generation	0.97	1.79	3.00	0.32
×4.80	×5.47	×6.04	×4.63
CUDA
Retrieval-augmented Baseline	0.29	0.44	0.75	0.10
Selfmem	Candidate Generation	0.51	1.00	1.72	0.18
Memory Selection	0.01	0.01	0.01	0.01
Hypothesis Generation	0.29	0.44	0.75	0.10
×2.76	×2.99	×3.35	×2.91
G Human and GPT-4 Evaluation
We employ both human annotators and GPT-4 (gpt-4-0314) annotators to perform pairwise ranking of the output generated by Selfmem and baseline systems. For GPT-4 annotators, we utilize the prompt from Alpaca Eval  $^{7}$ . We randomly select 50 samples for translation tasks and 20 samples for summarization and dialogue tasks. The win rate of Selfmem versus retrieval-augmented baselines is depicted in Figure 1.
Human and GPT-4 Evaluation Results
Figure 5: Human and GPT-4 evaluation results.
$^{7}$ https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evaluatorsconfigs/alpaca_eval_gpt4/alpaca_eval.txt