RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling
Authors: Jingcheng Deng
Liang Pang
Huawei Shen
Xueqi Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: Retrieval-Augmented Language Model
Hallucination
Variational Auto-Encoder

Abstract: Retrieval-augmented language models show promise in addressing issues like outdated information and hallucinations in language models (LMs). However, current research faces two main problems: 1) determining what information to retrieve, and 2) effectively combining retrieved information during generation. We argue that valuable retrieved information should not only be related to the current source text but also consider the future target text, given the nature of LMs that model future tokens. Moreover, we propose that aggregation using latent variables derived from a compact latent space is more efficient than utilizing explicit raw text, which is limited by context length and susceptible to noise. Therefore, we introduce RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE). It encodes the text corpus into a latent space, capturing current and future information from both source and target text. Additionally, we leverage the VAE to initialize the latent space and adopt the probabilistic form of the retrieval generation paradigm by expanding the Gaussian prior distribution into a Gaussian mixture distribution. Theoretical analysis provides an optimizable upper bound for RegaVAE. Experimental results on various datasets demonstrate significant improvements in text generation quality and hallucination removal.

Rega VAE: A Retrieval-Augmennte Gaussian Mixture Variational Auto-Encoder for Language Modeling
Jingcheng Dengl,2 Liang Pang",, Huawei Shen! Xueqi Cheng
IInstitute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences [dengjinggchegg23, pangliang, shenhuawei, cxq}eict.ac. ac. cn
Abstract
Future Info. in
Aggreg- ation
Model
Query
X
Key
Value
Retrieval-augen augeented language models show promise addressing issues outdated infor- mation and ballucinations in language models (LMs). However, current research faces two main problems: determining what informa- tion to retrieve, and 2) effectively combining retrieved information during generation. We ar- gue that valuable retrieved information should not only be related to the current source text but also consider the future target text, given the nature of LMs that model future tokens. More- over, we propose that aggregation using latent variables derived from a compact latent space is more efficient than utilizing explicit raw text, which is limited by context length and suscep- tible to noise. Therefore, we introduce Re- gaV,,, a retrieval-augmented language model built upon the variational auto-encoder (VAE). It encodes the text corpus into latent space, capturing current and future information from both source and target text. Additionally, we leverage the VAE to initialize the latent space and adopt the probabilistic form of the retrieval generation paradigm by expanding the Gaus- sian prior distribution into Gaussian mixture distribution. Theoretical analysis provides an optimizable upper bound for RegaVAB Ex- perimental results on various datasets demon- strate significant improvements text gener- ation quality and hallucination removal. Our codes is released in the link'.
KNN-LM
X
Explicit
RAG
X
X
X
Explicit
REALM
X
Explicit
XXX
SPALM
XXXXXXXXX
Explicit
FiD
Implicit
EMDR
Implicit
EPR
Explicit
Re2G
Implicit
RETRO
Implicit
Rega VAE
Implicit
Table Differences between Rega VAE and existing representative models. Query, Key and Value respec- tively indicate whether future information is contained in query, key and value parts. Aggregation represents the aggregation method of retrieved documents and source text.
et al., 2020), which results in model parameters having to grow exponentially order to gain more world knowledge; 2) difficulty in adjusting for time- sensitive knowledge (Lewis et al., 2020); 3) may produce "fact hallucination" problem (Guu et al., 2020; Marcus, 2020).
Recently, the advent of -ubieval- text generation has emerged at addressing these pertinent issues (Borgeaud al., 2022: et al., 2022: Shi et al., 2023). Compared to generative-only models, this paradigm not only explicitly exploits similar texts to generate more fluent sentences but also leverages expertise to gen- erate difficult responses. Nonetheless, we contend that there two primary challenges associated with current retrieval-augmenner language models. Firstly, not only current semantic information, but also future semantic information need to be consid- ered during retrieval. Previous studies (Khandelwal et al., 2020: Guu et al., 2020; Lewis et al., 2020)
novel paradigm aimed
1 Introduction
Language models (LMs) have achieved state-of- the-art performance on many NLP tasks (Zhu et al., 2021; Pang et al., 2021), which reveals that they store large amount of world knowledge as im- plicit parameters. While this development ex- citing, LMs still suffer from some problems (Li et al., 2022): 1) performance and model parame- ter size follow power law relationship (Kaplan
*Corresponding Author
'https://githun htaptehhhub.oo/uuuuuueeee http:/ahhwtb..cmmmtuteeett/t////

---

either directly use the entire text as key and value parts the same time, and then use cosine similar- ity (Xu et al., 2023), TF-IDF and other indicators to search, which leads to the value part is only similar to the source text (query), and does not necessarily serve the best for generator. Another way is to di- vide piece of text into two parts, where the first part and the second part are regarded as current in- formation and future information, such RETRO (Borgeaud al., 2022). However, RETRO adds future information to value part, but ignores the future information in query and key, which leads to the fact that candidate documents with high simi- larity do not necessarily contain future information that can help the generator. Secondly, explicitly aggregating retrieved documents and source texts is limited by the length of the model input and introduces too much noise. Implicit aggregation is inefficient in irregular embedding spaces, and retrieval vectors are not generalizable.
gate retrieval information into the generation process. Furthermore, we derive an upper bound on the optimization of this framework.
Experiments have shown that Rega VAB is competitive in generating quality, generating diversity, and eliminating hallucinations.
Related Work
We classify related studies into two categories, ex- plicit aggregation and implicit aggregation, accord- ing to the way the retrieved documents and source text are aggregated. Explicit aggregation refers to concatenating retrieved documents directly into source text to construct augmented input. Implicit aggregation refers to adding retrieved documents the generator the form of vectors or distributions.
Explicit Aggregation Guu et al. (2020) proposed an end-to-end framework REALM that achieves state-of-the performance on three open-domain QA. A representative work is RAG (Lewis et al., 2020), which first uses DPR (Karpukhin et al., 2020) to retrieve relevant documents, and then links relevant documents with the source text for gequence-to-sqquee
To address the above challenges, we design Re- gAE,,, a Retrieval-augmenned language model based on gaussian mixture Variational Auto- Encoder. Unlike previous methods that directly encode unlabeled corpora (Karpukhin et al., 2020; Lewis al., 2020) or only adding future informa- tion to the value part (Borgeaud al., 2022), as shown in Tab. 1, our model considers future in- formation through a latent space, given an x, we decode it into a y using a conditional VAE, which ensures that the latent variables contain information from both source and target data. In addition, in or- der to implicitly aggregate the retrieved documents and source texts, also use the probabilistic form of the retrieval generation paradigm theoretically extend the prior Gaussian distribution a Gaussian mixture distribution. This allows the latent space to satisfy continuity and uniformity, and the latent vector after aggregating retrieved documents and source text has better representation ability. Tab. I summarizes the differences between Rega and existing representative methods. Overall, our con- tributions are as follows:
generation. Different from RAG and REALM, Rubin et al. (2022) proposed EPR, which is a method for retrieving prompts and can improve the effect of prompts. Re2G (Glass et al., 2022) is an enhanced version of RAG, which improves the quality of retrieved documents by integrating multiple retrieval methods. Explicit ag- gregation simple and effective, but suffers from the limitation of the input length of the language model and cannot fully utilize the large number of retrieved documents. In addition, it is easy to introduce noise, making the model performance un- stable. Unlike these methods, our model implicitly aggregates retrieved documents into the generation process.
Implicit Aggregation FiD (Izacard and Grave, 2021) uses a DPR to retrieve candidate documents, and then splices and encodes the candidate docu- ments with the source text, and inputs them into the generator in the form of vectors. EMDR² (Sachan et al., 2021) is similar to FiD, and it provides an end-to-end framework to train both the retriever and the generator. However, the query. key and value parts of FiD and EMDR do not contain fu- ture information, which will cause the value part to be similar to the query part and not conducive to the generation of future tokens. RETRO (Borgeaud
We propose a retrieval method that implic- itly combines current and future information, which introduces future information into the query, key, and value parts at the same time, so that the higher the document similarity, the more helpful is for the generator.
We integrate the VAE and retrieval generation probabilistic framework to efficiently aggre-

---

Source Data
Step1: Build a Compact Space
Target
VAB Encoder
VAE Decoder
2-N(0,1)
Step2: Build Retrieval Database
R
Step3: Aggregate Retrieved Information
S Corpus Data
Corpus Embeddings
Zi
Retrieve
VAE Encoder
4
2
Key: Valuer 2
23
z z G (TT)
Source Data
Query
g
Target Data
VAE Encoder
z - (0,1)
VAE Decoder
Figure I: Architecture of RegaVAE. Based the training data, first train VAE construct compact latent space, which ensures that the latent variable 2 contains both current and future information (see 3.1). We then build retrieval database and then aggregate the retrieved information into the generator (see 3.2). VAE Encoder and Decoder parameters the same in all steps. In order to ensure faireess, the Corpus data and the Source data in the training set are the same. G represents the Gaussian mixture distribution, and ı is the corresponding parameter.
al., 2022) and KNN-LM (Khandelwal et al., 2020) set key and value parts piece of text, and added the continuation and the next token of this text in value part, respectively. However, they only calculate the similarity between the query and key while ignoring future information in value part, resulting in high similarity documents containing future information that may not necessarily help the generator. Our model sets both key and value parts as latent variables of piece of text and its future continuation, and the query encoded by the VAE encoder also contains future information, so future information also taken into account when calculating the similarity between query and key, making up for the shortcomings of previous stud- ies.
Commonly used retrieval indexes D include cosine similarity, TF-ID,, etc. This paradigm also be expressed in probabilistic form:
p(u|=) -
Σ rEtop-K(p(-le))
p r)((r))) (1)
Next, the framework of RegaVAB introduced, which consists of three steps. Firstly, in order construct compact space, we introduce the VAE structure. Since transformers based on VAE all suffer from the posterior collapse (Fu et al., 2019), we follow a previous study (Hu et al., 2022) which combines low-rank tensor products for latent vari- ables and decoders (see 3.1 and step in Fig, 1), Secondly, to introduce retrieval information into the latent space, we first introduce how the retrieval library is constructed (see step 2 in Fig. 1), and then replace the prior Gaussian distribution the original VAE with a Gaussian mixture distribution to derive Rega VAE (see step in Fig. I). This allows for deep aggregation of retrieved and input documents and simultaneously incorporates future information into query, key and value parts, which helps generate more fluent sentences (see 3.2) Finally, train RegaVALs able upper bound on the loss function for unclosed solutions (see $ 3.3). Fig. 1 shows the whole frame- work diagram.
3 Methodology
Most text generation tasks can be formulated as a mapping from source text y to target text y fx), while rerrieeal-augmeeneed text gen- eration can be further formulated as: = f(x, r), where r is the relevant document retrieved based Specifically, this approach generally encom- passes the utilization of retriever denoted as R and generator denoted G. The retriever R ob- tains r from the retrieval source by the retrieval metric and z. Then r and r are fed into 9 to obtain y through a predefined integration method
derive an optimiz-

---

3.1 Introduce Retrieval Information into Latent Space
each training step, resulting in changes in the latent space. Considering that it is impractical to update the retrieval database in real-time, and previous work (Guu et al., 2020) has shown the practicality of updating the index intermittently during training, we follow this approach and update the index of retrieval database every fixed number of training steps.
We consider using the VAE structure to make the space compact and continuous. As kind of gen- erative model, VAE estimates the intractable data distribution p(x: by deriving and maximizing its Evidence Lower BOund (ELBO) as:
log (x)) 2 BBLBO
(2)
A ggregate Retrieved Information Inspired by the retrieval-generaiee text generation paradigm, we assume y is influenced by latent variables z and æ", To obtain the ELBO of RegaVAE, we first model log p(v) as:
E ( 2) [lo [[o]p](<]=]]
where 2 is the latent variable. p(z) and (() is the prior and posterior distribution of z. respec- tively. (=(2(r) and po(r]2) represent Encoder and Decoder. 0 and @ are corresponding parameters.
log p(y) = log // ply, (002",=")dd dd)dz'd= de"
Due to the power of the decoder, transformers based on VAË usually have the problem of posterior collapse. According Hu et al. (2022), we use a low-rank tensor product in the 1-th layer of the model:
// log log 2², d=22") d2" d2"
// loga(z", 221x0)
logply, 2², 25,2") 2²)
log (2, ,0,,2" ,- dz
dzdd=" dz
"Byt"" =1|))) log/ p(y, 2", 2²)
will)))). 2)).
(3)
((2", 2
(5) From the first step, the Jensen inequality can be used to transform the second step, and then the expression of the desired form can be obtained. According to Bayes formula:
where z and (( represent latent variable and hid- den variable of (-th layer respectively. u!") repre- sents the hidden vector of the i-th token in /-th layer. r is parameeer,,,,rr, and o means element-wise multiplication. Wo and W2 are learnable param- eters which are shared across all positions (i) but not shared with 1-th layer.
ply, 2", 2²*) p(yl=", , >²)p("", (6)
Substituting Eq. 6 into Eqq 5:
Ee 2*))) log ply, p(y, 2*)² 00 23 [x)
In order not to introduce additional data, we use the training set as the data for training VAE. By optimizing ELBO, each sample is encoded into the latent space and then restored by the decoder to obtain a compact latent space.
(x( logl plyl , pp"xp("" q(2", 2((x)
(7)
( [lo p (1
KL(((<", , ==]x)) 1p2=pp(2_ 2²))
3.2 Build the Rega VAE Model
Build Retrieval Database With a compact latent space, we use an encoder to encode r and from S into the latent space. The latent variables of and are denoted by z" and 2", respectively. Then we store z" key and value parts in the retrieval database. Given a query z², we compute the inner product and to obtain the similarity.
where KL stands for calculating the KL divergence between two distributions. Eq. 7 is the ELBO of RegaVAE. At this point, Eq. and Eq. 2 have the same form, but the latent variable = is replaced by z" and z² Since each of follows a Gaussian distribution, we consider using Gaussian mixture distribution to combine nn and 2" So q(2", "==",="000 * can be expressed as:
D(z", 24) = cos(2") 2²,,))
(4)
q(=", wog(>"1x) + Σ w.g(<[[a)) (8)
where zf N(u, of) represents the latent vector of the i-th retrieved sample in S. μ4 and of are the corresponding mean and standard deviation, respectively. Since our framework is trained end- to-end, the parameters of the encoder change with
where n represents the number of retrieved docu- ments.
w/ = (oft
(9)

---

where Ei= wi 1 makes q(=",="|a1 satisfy the
Eq. 15 can be regarded an optimizable upper bound of Eq. 8. When given a dataset, we first en- code the source text to obtain retrieval database. The top-k documents are then retrieved for each x separately. Then the corresponding latent variables z" and z" are aggregated in the form of Gaussian mixture distribution and then input into 9 to ob- tain the output. Finally, use Eq. 15 to train RegaVAE.
requirement of Gaussian mixture distribution. So far, we have obtained the theoretical framework for introducing retrieval information in latent space.
3.3 Training Rega VAE
We can optimize Rega by optimizing Eq. 7. In the KL divergence term of Eq. 7, the closed-form solution cannot be obtained because the two distri- butions are mixed Gaussian distributions. There- fore, we continue to use previous research (Dilok- thanakul et al., 2016), that is, optimize its upper bound. First we assume two Gaussian mixture dis- tributions as:
Experiment
This section provides the experimental datasets, experimental settings, and experimental results.
9i
p
R...
(10)
4.1
Datasets
The KL divergence between them can be ex- pressed as:
For experiments, we employ three datasets, namely Yelp (Yang et al., 2017), Yahoo (He et al., 2019) and WritingPrompts (WP) (Fan et al., 2018). As in previous studies (Hu al., 2022), due to the limitation of computing resources, we adopt the methodology established in previous research and sample 100,000 data instances from the training set of Yelp and Yahoo for model training. This consistent approach ensures fair and equitable basis for comparison across the evaluated models.
KL(pl1)) (D) / T. log
Σ, Tigi
Σ;
Tigi Tath log nigi
(11)
Σ Tyy log Σ,
log gh
= [T](T Ty (9:ll:))))
Σ,
In the variational distribution g(2", 2 [z), the trainable parameter is only w; and (=*1xx) And the prior distribution p(2", 2²) is defined as:
4.2 Metrics
Generation Quality In the context of the text generation task, present the evaluation metrics of perplexity (PPL), Self-BLEU (Zhu et al., 2018), Dist2 (Li et al., 2016). and Activation Units (AU) (Burda et al., 2016). For the WritingPrompts, in ad- dition to PPL., we also report the metrics of BLEU (Papineni et al., 2002), Rouge-1, Rouge-2, Rouge- L (Mithun et al., 2012), and BERTScore (Zhang et al., 2020).
p(z",=)) =) uopp(= + Σn (xp((2), (12)
where n NO, 1) and 27 (O, 1). So the upper bound for the KL term in Eq. become:
=1=((=",=\==)]l(<= (2)) Σ "
KL KL(w:llkb
i
+KL(((>")))) I)) +
(13) where C constant that has nothing to do with model parameter updates. We do not update the retrieval library in real time, but regularly update according to the number of training steps. In this setup, wi is constant, so Eq. 13 becomes:
Hallucination We use SeliCheckGPT (Manakul et al., 2023) to detect hallucinations produced by the model. There are four indicators in total, namely SBERTS SQ^. S and Sm The higher their value, the more likely the model is hallucinating.
x )))) < <KL(g<>")ar) [=)] c
4.3
Experiment Settings
(14)
We have chosen two distinct categories of models as our baselines. The first category comprises trans- formers based on VAE, and the second category consists of retrieval-generale models. These base- lines provide us with a comprehensive framework for evaluating and contrasting different approaches.
Substituting Eq 14 into Eq we can get the final optimized loss function:
L = Ey(er,==1y) Dog x"")
(15)
=KL((("|=))]

---

Yelp
Yahoo
Model
Cost
PPL1
Self-BLEUt
Dist2
AUT
PPL 24.17
Self-BLEU
Dist21
AUT
GPT2
22.13
65.90
17.96
54.06
21.07
Retrieval-augmented Language Model
KNN-LM
39.95
62.30
8
FiD 14.08
42.26
24.45
14.71
42.84
26.49
66
RETRO
16.53
46.65
23.23
13.27
38.64
28.83
44
RAG
20.68
58.53
28.16
17.62
48.91
24.95
58
Transformers based on VAE
Optimus 22.79
23.11
Embed 19.98
65.27
15.59
22.18
54.15
20.80
3
Memory 19.95
63.90
16.91
11
22.03
54.59
21.87
18
Softmax
20.14
64.26
16.51
13
22.35
54.49
21.65
19
ADAVAE 15.49
49.80
32
14.23
32
DELLA 12.35
60.02
17.63
23
11.49
48.53
21.88
21
Rega VAE
VAE
8.62
36.10
28.83
52
6.99
30.74
33.03
56
60
Table Results for the Yelp and Yahoo. For transformers based on VAE, results of Optimus are directly copied from the original paper with À 0.5. The activation threshold of AU is 0.2. For retrieval- -augevel-augmenneed language models, RETRO, FiD and RAG are reproduced by ourselves under the same parameter size. KNN-LM employs the training set data as the retrieval corpus. addition, ensure fairness, all retrieval sources are training sets. The Cost column provides an indication of the temporal investment(h) required for training the respective model on A40-48G GPU.
Transformers based on VAE For comprehen- sive comparison, choose Optimus (Li et al., 2020) and ADAVAE (Tu al., 2022) the base- line models, along with four distinct paradigms: Embed (Li et al., 2020), Memory (Fang et al., 2021), Softmax (Wang and Wan, 2019) and DELLA (Hu et al., 2022). Optimus is a large-scale model based on VAE that utilizes pre-trained BERT model as its encoder and a pre-trained GPT- 2 model as its decoder. In order to ensure the fair- ness of the evaluation, Rega VAE uses the same pre-trained language model Embed, Memory, Softmax and DELLA. This selection facilitates rigorous and unbiased comparative analysis across these models.
the hidden variable is set to 32, and KL annealing (Fu et al., 2019) is implemented to mitigate the is- of KI. term disappearance. The learning rate is fixed at 5x 10 ensure stable training. Our train- ing procedure entails an initial 10 epoch training phase on the original DELLA model to establish a robust initial VAE space. Subsequently, we con- duct approximately fifteen epochs of training on the RegaVAE model until it achieves convergence. To make the training process more efficient, we precomputed document embeddings for the train- ing dataset and created FAISS index (Johnson et al., 2021) for fast similarity searches. We use the bert_score library to calculate the BERTScore for our models and baselines.
Retrieval-augmeened Language Model Ac- cording to the division method of related work, we select representative works from different cate- gories of retrieval- -uggeented language models as baselines. Specifically, RAG, FiD, and RETRO rep- resent models with explicit aggregation, implicit aggregation without future information, and im- plicit aggregation with only future information in value part, respectively.
4.4 Automatic Evaluation
Text Generation Tab. 2 presents the results at- tained by Rega model on text generation datasets. Compared to the three baseline mod- els for retrieval augmentation, our model achieves substantial improvements in all metrics, and per- forms particularly well in generating quality met- rics. The enhanced PPL, SeIf-BLEUU, and Dist2 scores demonstrate that latent variables, which con- tain both source and target information, combined
Our Model Consistent with prior research, we adopt the GPT2 model as the underlying backbone network for our experimentation. The dimension of
Phttps: http:///ttthtt.co//iiiiirrrrrttt

---

Model
PPL.
BLEU
R11
R21
RL
BERTScore"
Self-BLEU,
Dist21
GPT2
27.89
27.72
7.96
14.30
78.12
53.78
22.99
Embed
39.67
36.17
7.96
15.78
81.64
64.55
14.31
Memory
40.79
36.13
8.04
16.16
81.68
67.56
12.90
Softmax
41.04
36.14
8.12
16.30
81.75
67.02
13.08
DELLA 2.16
41.39
35.46
8.78
17.20
81.77
56.28
20.91
RegaVAE 1.18
43.83
32.21
9.62
30.57
84.31
52.70
23.28
Table 3: Results for the WritingPrompts. RI, R2 and RI. represent Rouge-1, Rouge-2 and Rouge-I, respectively.
The results for GPT2, EMbed, Memory and softmax are from the DELLA paper.
Model
BBRT
S
S
+
Hallucination Evaluation We evaluate halluci- nations of RegaVAE on the Yelp dataset. Specif- ically, we sample the text generated by the same latent variable three times, and then feed the sam- pling results into SelfCheckGPT obtain evalu- ation scores. The results are shown in the Tab, 4. From the experimental results, can seen that the text generated by Rega VAE is the least halluci- natory compared with other models.
FiD
8.27
37.99
4.96
6.31
RETRO
7.94
39.84
4.89
5.78
RAG
8.52
38.78
5.04
5.76
DELLA
8.41
40.01
5.21
5.30
RegaVAr VAE
8.01
37.82
4.42
4.89
Table 4:
Hallucination evaluation results the Yelp
dataset.
Model
Flu.t
Coh.1
Div.
Hal.
4.5
Human Evaluation
FiD
3.64
2.96
3.17
3.83
In addition to automated evaluation, we conducted human evaluation to assess and compare the per- formance of baseline models against our proposed method. Five professionals with expertise in the domain were enlisted participate in the manual evaluation process. Each evaluator was tasked with rating the attributes of fluency (Flu.), coherence (Coh.), diversity (Div.), and hallucination (Hal.) on a scale ranging from to 5. A rating of denoted very low performance, while rating of indicated very high performance. A total of 50 test samples were randomly selected and evaluated across dif- ferent models. The final human evaluation result was obtained by averaging the scores provided by the evaluators.
RETRO
3.33
3.11
3.32
4.01
RAG
3.15
2.73
3.25
3.99
DELLA
3.67
3.31
3.15
3.90
RegaVAF 3.78
3.21
3.47
4.11
Table 5: Human evaluation results on the Yelp dataset.
with the extension to Gaussian mixture priors, ef- fectively enhances the fluency and diversity of the generated text. This empirical validation corrobo- rates the theoretical strength of our model.
Notably, in comparison to the transformer-based VAE model, Rega VAE with retrieval demonstrates superior performance in terms of both generative diversity and quality. This enhancement can be at- tributed to the utilization of a Gaussian mixture dis- tribution, which offers the ability to capture multi- modal distributions more effectively than a single Gaussian distribution. Leveraging the source data, RegaVAE retrieves auxiliary latent variables that facilitate the generation of the target data, thereby yielding improved text generation outcomes. Fur- thermore, the significant improvement in the AU value indicates that the aggregation we employ pos- itively contributes the generative process de-
Tab. presents the outcomes of human evalu- ation conducted on the Yelp dataset. RegaVAB outperforms the baseline models in almost all di- mensions, demonstrating superior performance comparison. To further establish the correlation between human and automated evaluation results, we calculated the Pearson correlation coefficient and presented the corresponding values in Tab. 6. The results obtained from human evaluation align closely with those derived from partially automated evaluation metrics. For example, the correlation between the human evaluation metries (Flu., Coh.) associated with PPL and PPL itself is nearly identi- cal.
coder. This alleviates the problem of collapsing at the rear of the model to a considerable extent.

---

9.5
Flu.
Coh.
Div.
Hal.
-0-
Yelp
1.8
Yahoo
PPL
94.02
86p.07
10o.s1
260.68
WP
Self-BLEU
51o.37
190.76
660.23
350.57
8.5
Dist2
22a.73
570.31
670.21
580.30
Table Correlation between human and automated evaluation results. The order in which the model results were used to calculate the correlation coefficient is: FiD, RETRO, RAG, DELLA, and RegaVAR The correlation coefficients have been processed with absolute value and amplified by a factor of one hundred from their original values.
144
7.5
Numbel of neighbors
Figure Performance of RegaVAE on test sets as a function of the number of retrieved neighbors. The brown broken line corresponds to the scale on the right.
Model
PPL1
Self-BLEU
Dist21
Ours
8.62
36.10
28.83
Aggregation Method
Input
Generated Text
w/o VAE & R
17.95
53.24
16.46
Source: Our Damsel in Dincss, upon secing her knight in shining
Targge: Liveryone stood the doer with wide open eyes.
*CURSED "Mey mimered, polnting up and down the path
w/o VAE
20.13
62.48
17.04
leading into their nesigned balls What happened?? He
made his through bushes Surests
hall. He
Retrieval Method
had weering armour proudly
whole walk...
appreciates the effort but decides to just get the job date herselt.
fact. the light made his hair stand out staridy, Finally, he discovered the curse. "I still full, daes Eliaben"" be
Mase+BM2.
13.49
55.37
20.72
walkted back through the bushes to the door.
base+DPR 12.58
58.46
20.54
Retrieved Top-1
Wonds exerte with uraan teare.
are
nameless sooyy, duiing
Whhat happens acat? serites of bushes and small trees before lupporing upon a sie that horriffed him. Rick scnamblud to his fert and back the brush his dugout....
moved through
Table Results of ablation experiments on the Yelp dataset. w/o VAE represents the removal of VAE space, and w/o VAE & R represents the removal of VAE space and retrieval operations. base represents the Rega VAE that removes the retrieval.
minion
batle between good va bad you
accidonnalyy killed the protagonast antagonis.
Span
Seca Sin the doonder
Spanii-1
Unees Se the docoder
Figure 3: Generation examples Rega VAE from test set of WritingPrompts, Green font and brown font rep- resent the future information of the retrieved document and the relevant text generated by RegaVAE, respec- tively.
5.2 Ablation Experiment
To evaluate the effectiveness of the model struc- ture, conducted ablation experiments involving retrieval and aggregation, as depicted in Tab. 7.
5 Analysis
When we excluded the VAE structure, there was a notable decline in the performance of Rega VAE. In- terestingly, we observed that the model augmented with retrieval performed even worse than the model without retrieval when the VAE structure was ab- sent. We speculate that the retrieved variables in this particular scenario reside in space that fails meet the requirements of uniformity and continuity. As a result, the model struggled to generate valid samples based on cosine similarity, introducing unwanted noise instead.
To further analyze RegaVAE. we explore the im- pact of the number of retrieved neighbors, different model structures on the model performance. We also give case to verify the model performance.
5.1 Number of Retrieved Neighbors
Fig .2 depicts the performance trends in relation to the number of retrieved neighbors. Notably, as the number of retrieved neighbors increases from 5 to 100, we observed a reduction in PPL by 0.64 on the Yelp dataset and 0.69 on the Yahoo dataset, and PPL on the WP datasct is reduced by 0.59. This upward trend proves that implicit aggregation meth- ods can effectively filter noise compared explicit aggregation methods, and moreover, aggregations using Gaussian mixture distributions are effective for retrieving documents and source texts.
Compared with other retrieval methods, it can be seen that the performance of traditional retrieval methods is obviously insufficient. This discrepancy can be attributed our approach incorporating future information into key, value, and query parts simultaneously, thus taking future information into account in both retrieval and generation phases, further validating our motivation.

---

5.3
Case Study
Acknowledgement
We present a compelling example to examine the quality of BegaVAF-generated text and explore the integration of retrieval information into the gener- ated content, as illustrated in Fig. 3.
This work was supported by the National Key R&D Program of China (2022YF33103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.
Through our observations, we have noted that the text produced by RegaVAE demonstrates a re- markable ability to establish coherent connection with the source text while being vivid and specific. Moreover, despite encoding only the retrieved doc- ument into the latent space and subsequently inte- grating it into the generation process, it is evident that Rega -eE-geneaatd text effectively incorpo- rates future information from the retrieved docu- ment.
References
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang. Loren Maggiore. Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals. Simon Osindero, Karen Si- monyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR.
6 Conclusion
In this paper, we summarize two major challenges of existing ettrieval-augmentee language model methods, and propose RegaVAE to address them. We find that RegaVAs outperforms traditional re- trieval generative models in terms of both gener- ative quality and reduce hallucinations. In addi- tion, ablation experiments and three analysis exper- iments verify the correctness of the model motiva- tion. In future work, will consider migrating Rega VAE large language models.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In 4th In- ternational Conference on Learning Representations, ICLR 2016. San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Nat Dilokthanakul, Pedro M. Mediano, Marta Gar- nelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. 2016. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. CoRR,
Limitations
At present, almost all large language models are pre-trained on large-scale corpus, and due to the limitation of computing resources, cannot pre- train Rega VAE on large-scale corpus, which will lead to performance degradation.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018 Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. pages 889-898. Association for Computational Lin- guistics.
Furthermore, the model is not stable to train due to the posterior collapse problem. Even if we adopt low-rank tensor product, this problem still cannot be completely solved.
Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. 2021. Transformer- based conditional variational autoencoder for controllable story generation. arXiv preprint arXiv:2101.00822.
Ethics Statement
We honor and support the EMNLP code of Ethics. This paper mainly studies the use of retrieval gen- cration to eliminate the illusion in the language model and make the generated text more fluent. Our method can introduce canonical text to make language models more reliable. addition, the data sets used in this article are all open source and do not involve any privacy or ethical issues.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cycli- cal annealing schedule: simple approach to mit- igating KL vanishing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HIT 2019, Min- neapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 240-250 Association for Computational Linguisties.

---

Michael R. Glass, Gaetano Rossiello, Md. Faisal Mah- bub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2g: Retrieve, rerank, gen- crate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACI. 2022, Seattle, WA, United States, July 10-15, 2022, pages 2701-2715 Association for Computational Linguistics.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. eetrieval-augmente chowledgeinntensiig NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 12, 2020, virtual.
generation for
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang- 2020. REALM: retrieval- augmented language model pre-training, CoRR. abs/2002.08909.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng. Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Process- ing, EMNLP 2020, Online, November 16-20, 2020, pages 4678-4699 Association for Computational Linguistics.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Kirn-KKiikatiikk.. 2019. Lagging inference networks and posterior collapse in variational autoen- coders. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022. A survey on cetrivval-augmente text generation. CoRR, abs/2202.01110.
Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, and Xing Xie. 2022. Fuse it more deeply! variational transformer with layer-wise latent variable inference for text generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL 2022, Seattle, WA, United States, July 10- 10-15, 2022, pages 697-716. As- sociation for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A
ob- jective function for neural conversation models. NAACI DAACL.HI..2 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Humun Language Technologies, San Diego California, USA, June 12-17, 2016, pages 110- The Association for Computational Lin- guistics.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 874- 880. Association for Computational Linguisties.
Potsawee Manakul, Adian Liusie, and Mark J. F. Gales, 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models.
Gary Marcus. 2020. The next decade in Al: four steps towards robust artificial intelligence. CoRR, abs/2002.00177.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with gpus. JEEE Trans. Big Data, 733):335-547.
Shamima Mithun, Leila Kosseim, and Prasad Perera. 2012. Discrepancy between automatic and manual evaluation of summaries. In Proceedings of Work- shop on Evaluation Metrics and System Comparison for Automatic Sunmarization @ NACCL-HLT 2012, Montrèal, Canada, June 2012, 2012, pages 44-52. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish. Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361.
Vladimir Karpukhin, Barlas Oguz. Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781, Associa- tion for Computational Linguistics.
I Liang Pang, Yanyan Lan, and Xueqi Cheng. 2021. Match-ignition: Plugging pagerank into transformer for long-form text matching. In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 5, 2021, pages 1396 1405. ACM.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6- 6-12, 2002, Philadelphia, PA, USA, pages 31 311-318. ACL.

---

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2655-2671. Association for Computational Linguistics.
Research & Development Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 12, 2018, pages 1097-1100. ACM.
Yunchang Zhu, Liang Pang, Yanyan Lan, Huawei Shen, and Xueqi Cheng. 2021. Adaptive information seek- ing for open-domain question answering. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event Punta Cana, Dominican Republic. 7-11 November, 2021, pages 3615-3626. Association for Computational Linguistics.
Devendra Singh Sachan, Siva Reddy, William L. Hamil- ton, Chris Dyer, and Dani Yogatama. 2021. End-to- end training of multi-document reader and retriever for open-domaie question answering. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual. pages 29968-25981.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. REPLUG: retrieval-augmentee black-box language models. CoRR, abs/2301.12652.
Haoqin Tu, Zhongliang Yang, Jinshuai Yang, and Yongfeng Huang. 2022. Adavae: Exploring adap- tive gpt-2s in variational auto-encoders for language modeling. arXiv preprint arXiv:2205.05860
Tianming Wang and Xiaojun Wan. 2019. T-CVAE:
transformer-bas basedd conditioned variational autoen- coder for story completion. In Proceedings of the Twenty-Fighth International Joint Conference on Ar- rificial Intelligence, IJCAI 2019, Macao, China, Au- gust 10-16, 2019, pages 5233-5239, ijcai.org.
Shicheng Xu, Liang Pang, Huawei Shen, and Xueqi Cheng. 2023. BERM: training the balanced and ex- tractable representation for matching improve gen- eralization ability dense retrieval. of the ó1st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6620-663 6635. Association for Computational Linguis- tics.
Proceedings
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated con- volutions. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Syd- ney, NSW, Australia, ll August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3881-3890. PMLR.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- neew.net.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on


Reviews:
[{"review_id": "CW5ELtlw1j", "reviewer": "Reviewer_MX59", "summary": "", "questions": "A. How is the model handling cases where the future information is not explicitly given or is ambiguous? \n\nB. How did you handle the case when the retrieved documents are not relevant or helpful for the generation task? Did you have any mechanism to filter out or down-weight such documents?\n\nC. Experiments: Table 3: Could you compute and report these scores for the retrieval model as well? What is a reason for Della performing better? \n\nD. Experiments, human annotation: what were the definitions of Fluency, Consistency etc. given to/agreed by the human annotators?\nCould you provide (in appendix?) representative examples of comparison between text generated with the various baseline models and the proposed model?\n\nE. Have you done any hyperparameter optimization for the proposed model? (e.g. rank for the low-rank tensor product) \nfor the baseline models? \nIf so, how was the computing budget decided?\n\nF. \"due to limitation of computing resources, we cannot pre-train RegaVAE on large-scale corpus\"\n--> What were the compute resources/time/power used for training the model? what is the current energy requirement?\n--> How much computing budget (and energy) do you estimate would be needed to train the proposed model on large scale compute and match the performance of large LMs? \nI'm wondering whether, when training on much more data, the proposed model would need significanlty more compute to reach the level of current SOTA retrieval models.\n(cf. last point on computing time/power in https://2023.eacl.org/ethics/faq/ and referenced impact tracker: https://arxiv.org/abs/2002.05651)\n\nF. \"bis\" Also, can you list the computing resources/time/power used for training the baseline models and add it in the results tables, to allow for a more informed comparison between the models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and thorough review. We added the suggested experiments to the rebuttal revision. In the following, we seek to address each of your concerns.\n\n**Q:** *Computational cost mentioned in reason 1, and problem F*\n\n**A:** Taking the GPT2-based RegaVAE trained on the Yelp dataset as an example, it takes 2.5 days for full fine-tuning with a single A40-48G. The single A40 of the server we use has a maximum power of 300w. Based on this, it is inferred that the total energy consumed is 64,800 kJ. To extend RegaVAE to LLM, we will consider adding a small number of fine-tunable parameters to the original LLM to implement RegaVAE, such as RegaVAE based on LLama2. Compared with pretraining on large-scale data, this method is parameter efficient (see LoRA) and the computational cost is very low. Since the performance of some baseline models comes from the DELLA paper, the computational cost is not reported in it. Therefore, we will give the calculation cost of our own reproduced baseline models, where “h·single A40” represents the cost of training for 1h on a single A40. As shown in the following table:\nModel | Computational cost (h·single A40) |\n:-: | :-: |\nKNN-LM | 8|\nFiD | 66 | \nRAG | 58 | \nRETRO | 44 |\nRegaVAE | 60 |\n\n**Q:** *Scalability mentioned in reason 1*\n\n**A:** RegaVAE has a certain scalability. As mentioned before, LLMs-based RegaVAE can be realized by adding a small number of fine-tunable parameters to the original LLM. Furthermore, we have the flexibility to substitute various base models, such as Bart, T5, and others, in place of GPT2.\n\n**Q:** *Robustness mentioned in reason 1*\n\n**A:** We analyze the robustness of the model from the perspective of retrieving neighbors. Thanks to the implicit aggregation method, RegaVAE can filter out noise information well even with irrelevant documents (see Sec 5.1).\n\n**Q:** *Reason 2 and question E*\n\n**A:** All VAE-based models have the problem of posterior collapse, so we use strategies such as cyclic annealing, which is the same as previous work (DELLA, Embed, Memory, and Softmax). We did not perform hyperparameter optimization. Instead, to ensure fairness, we used the same hyperparameter settings as previous work (DELLA), including learning rate, Batch size, etc. Some experimental settings for training RegaVAE have been given in Sec 4.3. We will supplement the settings of other hyperparameters, and the corresponding implementation methods.\n\n**Q:** *Question A*\n\n**A:** The future information of a sentence is defined by us as its continuation. In the training phase, we construct a sentence (current information) and its continuation (future information) into sample-label pairs $(x, y)$ and feed them into RegaVAE to obtain a compact latent space. In the inference phase, we feed the test sample $x'$ into RegaVAE. At this time, the hidden variable $z$ sampled from the latent space contains the future information of $x'$. Therefore, there is no situation where future information is not expressly given or is unclear.\n\n**Q:** *Question B*\n\n**A:** We have considered this issue and conducted corresponding experiments (Sec. 5.1). We explored the impact of different numbers of neighbors on model performance on three datasets, and concluded that the implicit fusion method itself has a filtering mechanism for irrelevant neighbors (perhaps due to the attention mechanism in Transformer), the more neighbors retrieved within a certain range, the better the model performance. We can also set an explicit filtering mechanism, such as a similarity threshold to filter irrelevant documents. However, we think it is just a trick and has nothing to do with our research goals.\n\n**Q:** *Question C*\n\n**A:** WP is a story continuation data set. The average token length of each sample is 702, far exceeding 96 for Yelp and 79 for Yahoo. Computing the scores for retrieval augmentation models on top of this is therefore computationally expensive. Since we have demonstrated that RegaVAE outperforms these retrieval augmentation models on two datasets, Yelp and Yahoo, we do not consider reporting the scores of these retrieval models on the WP dataset. In addition, in Tab. 3, RegaVAE is better than DELLA in almost all metrics.\n\n**Q:** *Question D*\n\n**A:** Fluency refers to the fact that the text has good connections and readability. Coherence represents the layering and smooth cohesion of the text when expressing a specific theme. The example presented in Fig. 3 demonstrates this. We will show an example for each baseline model in the appendix for comparison.\n\n**Q:** *Missing References, Typos Grammar Style, And Presentation Improvements*\n\n**A:** Thank you for your reminder. These missing references are relevant to our theoretical framework and we cite them where appropriate. Relevant expressions will be corrected. We will add relevant references to support some of the claims.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- this paper tackles 2 challenges (cf above) in retrieval-augmented text generation that (to my knowledge) have not been much studied in previous works.\n- paper is very well-written and well-organized, and provides clear motivation, background, and some technical details for the proposed model, including a theoretical analysis of the optimizable upper bound for the model. It includes ablation studies and case studies.\n- the proposed training procedure incorporates improvements/optimizations from prior work in this field.\n- experiments were performed on three relevant datasets, with meaningful comparison against existing retrieval-augmented LMs and VAE-based LMs; they show that the performance of the proposed model is competitive in terms of generation quality and hallucination reduction. The evaluation metrics chosen are relevant. The paper also includes ablation studies and case studies to analyze the performance further.\n\n", "reasons_to_reject": "- the paper may benefit from extending discussions on the potential drawbacks of the proposed model, e.g. computational cost (actual one, or expected cost to reach current large LMs performance), scalability, robustness.\n\n- some missing experimental setup: hyperparameter search, computing budget, which implementation of the metrics was used\n-- the code will surely give these experimentation details, when released -- it'd be great to shortly mention them in the paper itelf!", "questions_to_authors": "A. How is the model handling cases where the future information is not explicitly given or is ambiguous? \n\nB. How did you handle the case when the retrieved documents are not relevant or helpful for the generation task? Did you have any mechanism to filter out or down-weight such documents?\n\nC. Experiments: Table 3: Could you compute and report these scores for the retrieval model as well? What is a reason for Della performing better? \n\nD. Experiments, human annotation: what were the definitions of Fluency, Consistency etc. given to/agreed by the human annotators?\nCould you provide (in appendix?) representative examples of comparison between text generated with the various baseline models and the proposed model?\n\nE. Have you done any hyperparameter optimization for the proposed model? (e.g. rank for the low-rank tensor product) \nfor the baseline models? \nIf so, how was the computing budget decided?\n\nF. \"due to limitation of computing resources, we cannot pre-train RegaVAE on large-scale corpus\"\n--> What were the compute resources/time/power used for training the model? what is the current energy requirement?\n--> How much computing budget (and energy) do you estimate would be needed to train the proposed model on large scale compute and match the performance of large LMs? \nI'm wondering whether, when training on much more data, the proposed model would need significanlty more compute to reach the level of current SOTA retrieval models.\n(cf. last point on computing time/power in https://2023.eacl.org/ethics/faq/ and referenced impact tracker: https://arxiv.org/abs/2002.05651)\n\nF. \"bis\" Also, can you list the computing resources/time/power used for training the baseline models and add it in the results tables, to allow for a more informed comparison between the models?", "paper_topic_and_main_contributions": "This paper is mainly a NLP modeling + engineering experiment that addresses two problems in retrieval augmented language models (LMs):\n\n1. selecting relevant information from the documents database -- the authors argue that all/most existing retrieval augmented LMs only use the input query (\"current information\") to retrieved documents/passages, leading to the information retrieved being similar to the context/query, and not necessarily containing relevant bits to continue the generation (\"future information\")\n2. effectively combining retrieved information during generation -- the authors argue that explicit aggregation (concatenating retrieved documents to the source text) is limited by the length of the input and introduces noise, while implicit aggregation (using vectors or distributions) may be inefficient in irregular embedding spaces and retrieval vector not generalizable.\n\nThe main contributions of the paper are:\n\n- proposing a novel retrieval-augmented LM based on a variational auto-encoder (VAE), which encodes the text corpus into a latent space that captures both current and future information;\n- deriving an optimizable upper bound for the RegaVAE objective function;\n- conducting experiments on three text generation datasets, and comparing RegaVAE performance with existing retrieval-augmented LMs and VAE-based LMs in terms of generation quality and hallucination reduction.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Consider whether it'd be appropriate to cite previous work where a VAE is extended with a mixture distribution prior, such as a mixture of Gaussians (even though on a different domain/model architecture), for instance:\n\n- \"VAE with a VampPrior\" (Tomczak and Welling, 2018) \n- \"Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classification\" (Bai, Kong and Gomes, 2022)\n", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written, I did not notice any typo or grammatical error, apart from \"RegaVAE outperforms the baseline models almost all --> \"almost always\" or \"almost all the baseline models\" (last sentence just before section 5 - Analysis)\n\n+possible ~repetition in \"Notably, […] there was a notable […]\" (2nd sentence in subsection 5.2 Ablation experiment\")\n\n--\n\nIt may be difficult for a reader not familiar with the details of RETRO to quickly grasp what constitutes \"future information\" and how it is \"extracted\" or inferred from the text corpus.  Clarifying the definition, and being explicit (even if redundant) when mentioning what exactly is being considered as future information, may help them.\n\ne.g. : \"a natural idea is to divide a piece of text *in the document database/in the pool of retrievable documents* into two parts, […]\"\n\n--\n\nSeveral claims could be better supported by references, for instance: \n\n- intro: \"[LMs] store a large amount of world knowledge as implicit parameters\" (any study to cite here?)\n- \"transformers based on VAE usually have the problem of posterior collapse\" (citing papers that show this?)\n- \"implicit aggregation methods can effectively filter noise compared to explicit aggregation methods\" (is there some examples or analysis of this?)\n- related work: on explicit aggregation: \"it is easy to introduce noise\" (which of the papers mentioned earlier in the paragraph develop this point? any example of such noise and its impact?)\n- related work: pitfalls of implicit aggregations such as \"implicit aggregation is inefficient in irregular embedding spaces\" (any of the cited papers or another study to support, detail or illustrate this?)\n"}}, {"review_id": "4Utifs0VPe", "reviewer": "Reviewer_oBTQ", "summary": "", "questions": "Question A: Tab. 2 and 3, GPT2 is pre-trained only or further fine tuned?\n\nQuestion B: What is the standard deviation of the scores reported in Table 5?\n\nQuestion C: Line 460. 50 total samples for the 5 selected models? Meaning each model gets 10 questions?\n\nQuestion D: I understand that there are considerations implementing this approach on LLMs. As the training comprises of additional training steps, how performant is it currently compared to fine-tuning GPT2? Will it be a challenge to extend this to LLMs?\n\nQuestion E: I might have missed the information. In Figure 1, are all annotated VAEs in the diagram have the same weights? or are they the same model but different weights?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. In the subsequent sections, we aim to meticulously address each of your concerns.\n\n**Q:** *Question A*\n\n**A:** In Tab. 2 and 3, the GPT2 model parameters are first initialized by the original GPT2 model on Huggingface, and then further fine-tuned.\n\n**Q:** *Question B*\n\n**A:** We calculated the standard deviation of the data in Tab. 5, as shown in the table below. These data will eventually be added to Tab. 5.\n\nModel  | Flu. | Coh. | Div. | Hal.\n:-: | :-: | :-: | :-: | :-:\nFiD | 0.28 | 0.26 | 0.35 | 0.28 \nRETRO | 0.24 | 0.19 | 0.25 | 0.20 \nRAG | 0.26 | 0.24 | 0.30 | 0.23 \nDELLA | 0.23 | 0.20 | 0.27 | 0.30 \nRegaVAE | 0.25 | 0.21 | 0.24 | 0.18\n\n**Q:** *Question C*\n\n**A:** Each model has 50 samples. To avoid misunderstanding, we will use a more precise expression.\n\n**Q:** *Question D*\n\n**A:** Compared to the fine-tuned GPT2, RegaVAE achieves clear improvements in all performances (Tab. 2, 3). Although an additional training step is included, we do not use additional training data. To ensure fairness, we keep the hyperparameters consistent. Although it will be difficult to extend RegaVAE to LLM, a feasible idea is to add a small number of fine-tunable parameters to the original LLM to implement the RegaVAE framework. This allows effective fine-tuning of parameters and reduces the cost of application on LLM.\n\n**Q:** *Question E*\n\n**A:** Fig. 1 shows the training process of RegaVAE, where all VAEs have the same weights.\n\n**Q:** *Typos Grammar Style And Presentation Improvements*\n\n**A:** For the main experiment, due to the limitation of computational cost (a model runs on a data set for an average of 3 days), we cannot determine the error bars during the rebuttal period, but we will supplement them before the final admission results come out. In addition, we will supplement relevant representative samples in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1: The authors proposed a new retrieval augmented language model which combines VAE and text retrieval. It is competitive on widely-used text generation metrics, experiments for automated evaluations of generation quality and hallucinations. \n\nA2: The authors conducted additional ablation studies and human evaluation in a study group comprising of 5 people.\n", "reasons_to_reject": "No obvious reasons identified.", "questions_to_authors": "Question A: Tab. 2 and 3, GPT2 is pre-trained only or further fine tuned?\n\nQuestion B: What is the standard deviation of the scores reported in Table 5?\n\nQuestion C: Line 460. 50 total samples for the 5 selected models? Meaning each model gets 10 questions?\n\nQuestion D: I understand that there are considerations implementing this approach on LLMs. As the training comprises of additional training steps, how performant is it currently compared to fine-tuning GPT2? Will it be a challenge to extend this to LLMs?\n\nQuestion E: I might have missed the information. In Figure 1, are all annotated VAEs in the diagram have the same weights? or are they the same model but different weights?", "paper_topic_and_main_contributions": "The authors propose a new retrieval augmented language model based on guassian mixture VAE. Retrieval augmented language models are proposed as a possible approach to resolve issues that stems from the scale of LLMs. The authors designed their model to implicitly aggregate future information in key, query, and values. The ability to aggregate future information implicitly on key and query is the defining property of their model. The authors detail their approach to the challenges of incorporating these features into the model. Their model is competitive against other baselines across different selected evaluation metrics on Yelp and Yahoo datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None that I can identify."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Include error bars in the evaluation results (if any), while some metrics are convincingly distant, there are metrics that are close (i.e. Dist2 in Tab3, S_BERT and S_QA in Tab. 4 and Tab. 5).\n\nSample questions and Instructions given to human evaluators can be included in the appendix.\n\nAdditional comparisons of complete generated text from various models can be included in the appendix."}}, {"review_id": "vjmiguJCO5", "reviewer": "Reviewer_RR3J", "summary": "", "questions": "In 3rd line of Eq (5), $\\ge$ is right?\n\nAre $g_{i}$ and $\\hat{g}_{i}$ Gaussian mixture?\n\nIs $\\int \\hat{g}_{i}=1$ right?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments. In the following, we explain each of your concerns. \n\n**Q:** *Reason 1*\n\n**A:** In our definition, explicit aggregation means inserting the retrieved documents as prefixes before the original samples, which are then input to the decoder for training (or inference). The implicit aggregation is defined as turning the original sample and the retrieved documents into embedding vectors, merging them at the vector level and then handing them over to the decoder for training (or inference) (this is exactly what RegaVAE uses). In lines 139-147, we state the disadvantages of display aggregation (some additional references are cited to demonstrate our point). In Sec 5.1 we proved that implicit aggregation has the advantage of filtering irrelevant document information from the perspective of retrieved neighbors. In addition, In-context Learning is for LLMs. Retrieval-based In-context Learning is to input retrieved documents as context and questions into LLM for inference and does not involve the training process. For comparison with this method, we use the RAG baseline model, which differs from In-context Learning only by having an additional training process. It is proved in Tab. 2 that RegaVAE is better than RAG on the same scale.\nSecondly, existing VAE-based models do not have as many parameters as RegaVAE. Although VAE-based models are not as popular as LLM today, it is still a good research area because of strong interpretability and regular latent space. It can be seen from Tab. 2 that under the same parameter scale, the retrieval generation model with VAE (RegaVAE) is better than the retrieval generation model without VAE. In addition, it is also feasible to extend RegaVAE to LLM. One way is to implement the RegaVAE framework by adding a small number of fine-tunable parameters to the original LLM. This allows effective fine-tuning of parameters and reduces the cost of application on LLM. If we implement this method, we should submit it to the LLM track instead of the NLG track.\n\n**Q:** *Reason 2*\n\n**A:** We compare two major classes of baseline models, VAE-based transformer models and retrieval-generative models. Before we designed RegaVAE, DELLA was the latest VAE-based transformer model (as far as we know), and the rest of the models were its baselines. In the retrieval generation baseline model, they are all works with over 100 citations. Secondly, in 2022, Retro emerged as a state-of-the-art retrieval generation model. And FiD will occupy the first position on the KILT leaderboard all year round. So we consider our experiments to be adequate and comprehensive. In addition, under the same parameter scale, RegaVAE achieves the best performance among these models.\n\n**Q:** *Reason 3*\n\n**A:** The overall writing idea of this paper is as follows. In the field of retrieval generation, we have summarized two significant challenges, one is what information needs to be retrieved, and the other is how to integrate the retrieved information into the generation process. We then concluded two critical points based on previous research and experience. That is, valuable retrieval information should not only be related to the current source text but also consider the future target text, secondly, aggregation of latent variables derived from a compact latent space is more effective than exploiting explicit original text. Then we propose the RegaVAE framework, which simultaneously satisfies the two conditions of considering future information and compact latent space. Finally, we compared the retrieval-based baseline model with the VAE-based transformer baseline model to prove the superiority of RegaVAE. Evaluation metrics include text generation quality and hallucinations.\n\n**Q:** *Question A*\n\n**A:** Thank you for your reminder, due to misoperation, it should be $=$ but it was wrongly written as $>=$. We'll make changes. Nevertheless, it does not affect our derivation.\n\n**Q:** *Question B*\n\n**A:** In line 302 we point out that $p$ and $\\hat{p}$ are Gaussian mixtures, so here $g_i$ and $\\hat{g_i}$ are Gaussian distributions.\n\n**Q:** *Question C*\n\n**A:** We speculate that what you want to say here is $\\hat{w_i}$. In lines 288~290, we use $\\rm{softmax}$ to normalize $\\hat{w_i}$. So $\\sum \\hat{w_i} = 1$ is correct.\n\n**Looking forward to further discussions with you.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The issues raised are understandable\n  * The valuable retrieved information should not only be related to the current source text but also consider the future target text.\n* Challenges associated with current retrieval-augmented language models are also interesting\n  * RegaVAE considers future information through a latent space, given an $x$, it is decoded into a $y$ using a conditional VAE, which\n    ensures that the latent variables contain information from both source and target data.\n* Limitations and Ethics Statements are good", "reasons_to_reject": "* Doubts remain about the appropriateness of this approach to solving the problem of retrieval-augmented language models.\n  * In-context learning with LLMs appears to be a promising approach but has not been fully explored. Since the only comparison being made is Rubin et al. (2022), it cannot be said that RegaVAE implicitly aggregates retrieved documents into the generation process (L.134-147).\n  * Not only that, but the existing research on VAE-based generation models are weak.\n* Weak experimentation\n  * Baseline models are inadequate and outdated. This comparison does not adequately demonstrate superiority.\n* The writing in this paper is lacking in substance and poorly organized", "questions_to_authors": "In 3rd line of Eq (5), $\\ge$ is right?\n\nAre $g_{i}$ and $\\hat{g}_{i}$ Gaussian mixture?\n\nIs $\\int \\hat{g}_{i}=1$ right?", "paper_topic_and_main_contributions": "This paper focuses on issues of current research and proposes RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE).\nAuthors argue that valuable retrieved information should not only be related to the current source text but also consider the future target text.\nThat is, RegaVAE encodes the text corpus into a latent space, capturing current and future information from both source and target text.\nThen, it initializes the latent space and adopts the probabilistic form of the retrieval generation paradigm by expanding the Gaussian prior distribution into a Gaussian mixture distribution.\nIts contribution is to outperform traditional retrieval generative models in terms of both generative quality and reduce hallucinations.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Proposes an approach to enhance the performance of Retrieval Augmented Language Models, as it relates to outdated information and hallucination. The idea is to make the retrieved information related to the source text but also take future target text into account. Based on this, the authors propose an approach based on aggregating latent variables from a latent space as opposed to using the original text and develop the RegaVAE framework, which is based on built using VAE. Theoretical analysis is conducted and an upper bound for the objective is developed. Experiments reveal significant improvement in text generation quality and hallucination reduction. 

PLUS: 

- addresses two key challenges in Retrieval Augmneted Language models: selecting relevant retireval info and effectively aggregating retrieved info during generation. Appears to be novel. 
- theoretical analysis: upper bound on objective, which is optimizable. 
- ablation study and case study. 
- solid experimental study showing improvement over baselines on text generation quality and hallucination reduction. 

MINUS: 

- computational cost not analyzed. 
- a few additional experiments could add value, e.g., hyperparameter search.
Decision: Accept-Findings