Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models
Authors: Minki Kang
Sung Ju Hwang
Gibbeum Lee
Jaewoong Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: large language models
knowledge injection
data augmentation
Primary area: natural_language_processing

Abstract: As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. 
To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers.
This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. 
Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. 
Additionally, combining LaPael with data-level paraphrasing further enhances performance.

Perturbation on Layers Improves Knowledge Injection in Language Models
Latent Paraphrasing:
Minki Kang
Sung Ju Hwang?
Gibbeum Leel
Jaewoong Cho
KRAFTON, KAIST
#xxc11330rrafton.cm
sjhwangdkaist.. ac.kr,
{pirensisco, jwcho jxcho)@krafton.com
Abstract
As Large Language Models (LLMs) are increasingly deployed in specialized do- mains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due repetitive external model usage and limited sample diversity. To this end, we introduce LaPael, a latent-level paraphras- ing method that applies input-dependent noise early LLM layers. This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, eliminates the recurring costs of paraphrase generation for each knowledge update. Our extensive experiments on question-ans wering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. Additionally, combining LaPael with data-level paraphrasing further enhances performance.
Introduction
Pre-trained Large Language Models (LLMs) encode extensive factual information from their training data, enabling them to answer factoid questions such as "Who is the director of Dune: Part Two?" [4, 32]. However, knowledge in LLMs is statie, which can lead outdated information real-world knowledge evolves. Additionally, LLMs often lack specificity for specialized or private domains. To address this, is common practice fine-tune LLMs with updated or domain-specific documents, keeping the model's 's knowledge up-to-date and enhancing expertise in particular domains [14, 17, 19].
However, does fine-tuning LLMs single document allow them to fully internalize its knowledge? Even in pre-training, Kandpal et al. [20] found that LLMs cannot perfectly learn all the information the training data, particularly long-tail knowledge that appears rarely or only once. Existing work [33] has shown that this issue persists with fine-tuning and suggested that data augmentation, such as paraphrasing, is simple yet effective way to enhance knowledge injection. As shown Figure 1, fine-tuni tine-tuning with paraphrases enhances knowledge injection, as evidenced by improved Question- Ansiiirnn (QA) task performance.
Fine-Tuning (FT)
Ouzs
FT Parophrases
Ours
Paro.
QA
90
85
80
75
Streaming@A
ArchivalQA
Figure Rffeect of paraphrasing data knowledge injection.
While data- augmented approach via paraphrasing is effective for knowledge learning, it has two main limitations: (1) High computational cost: Generating high-quality paraphrases requires significant computational resources. As shown in Figure 2, paraphrasing models such LLMs [5, 7, 11, 58] need to repeatedly generate paraphrases for each document with the new incoming knowledge. This leads to higher costs as the number of documents being learned continually increases; and (2) Limited
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task: Knowledge Injection
Base LIM
$872 million
What the worldwide total gross "Dune: Part Two" as of May 14, 2024?
Fine-tuning (FT)
$192. 1 million the United States ane Canatia and
As of May 14, 2024, Dunc: Part Two prossed
$428. million centtrerttries, for worddwice
User
total of 5710.6 milloon
Fine-tuned LUM
A: $710.6 million
Data Data-augmented approach
Our approach: Latent Paraphrasing
FT
Fine-tured tunee LIM
LLM
Base LLM
Fine tuned LLM
Fine tune-tundddd LLM
$282.1
54285 million.
Porashrasers
Paraphrasors
latent
paraphrasers
Paraphraser
No paraphrase
As of May 14, 2074, Duner Port Twon has, grossed $282.1 million in the linited States and Canata and $438.5 million in other terthories lor coridarde iotal $710.5 millimmllln
of Maay 14, 2004, Dune: Part Two has grossed $382 million in the United States and Canada and $422 millin other to tenitoriss, for world de total of $710.6 million.
Figure 2: A conceptual illustration of the proposed approach. On the left, we show the existing method of knowledge injection by paraphrasing each document for data-level augmentation. On the right, we present the conceptual illustration of LaPael with trained latent paraphrasers. Unlike the method the left, LaPael eliminate the need for users to repeatedly paraphrase using LLMS once latent paraphrasers are trained.
diversity in augmented data: Although L.MMs can produce varying high-quality paraphrases by sampling from the generative distribution, the diversity of the generated text is limited, resulting in narrow range of augmented samples at the discrete data level. One way to overcome these issues is to introduce noise into the token embedding. However, existing works [16, 57] do not consider the text semantics when they perturb the latent features of LLMs with randomly generated noise.
To address these issues, we take distinct approach using an input- nnput-dependent noise generator named "latent paraphraser" learned from the paraphrases. Specifically, this function perturbs early layers to augment LLMs at the latent level while preserving the meaning of the text. To optimize the latent paraphraser, we start by generating paraphrases of the documents. Then, we train the latent paraphrasers to ensure that the latent distribution of the LLMs with the original sentence is close to the latent distribution with the paraphrased sentences. Once training is done, we can transfer the latent paraphrasers to the documents from any domain that contains new knowledge. We refer to our method as Latent Paraphrasing of Language Models (LaPael), as learns the paraphrasing text data at the latent level.
We validate our approach on diverse question-ansse answering benchmark datasets [38, 27, 51] designed to evaluate knowledge injection. These benchmarks involve tine-tuning LLMs on documents that contain the knowledge required answer the questions in the datasets. Our results show that LaPael significantly improves knowledge injection performance compared to standard fine-tuning, Moreover, LaPael outperforms fine-tuning with paraphrases, demonstrating that LaPael alone is sufficient for data augmentation in knowledge injection scenarios, as illustrated in Figure 2. As shown in Figure 1, we further find that using LaPael in combination with paraphrases further enhances performance, providing complementary benefits to data-level augmentations. Finally, LaPael surpasses existing noise baselines [16, 57], highlighting the importance learning noise for effective augmentations.
Our contributions are as follows:
We introduce LaPael, new method that applies learned perturbations to the layers of LLMs to enhance knowledge injection, addressing the limitations of data augmentations and noise baselines.
We validate LaPael using diverse avestion-answering benchmark datasets, demonstrating signifi- cant improvement knowledge injection performance compared to standard fine-tuning.
Our results show that LaPael not only outperforms fine-tuning with paraphrases but also comple- ments it, providing additional benefits when used together, surpassing the performance of existing latent noise-based methods.
2
Related Work
Knowledge of Large Language Models Large Language Models (LLMMs) store vast amounts of factual knowledge in their pre-trained parameters [36, 44]. The straightforward way extract the

---

knowledge of LLMS is to ask the question that requires factual knowledge [43, 58]. Through asking questions, Kandpal al. [20] have found that LLMs cannot perfectly memorize the entire knowledge in the pre-training corpora, especially for knowledge that appears rarely or only once. To make L.Mss answer the question requires under-represented or new knowledge, previous works have clustered into two different solutions. The first one is -argeeee- methods [26, 39, 42] that retrieve knowledge from an external knowledge base and imput the retrieved knowledge alongside the question into LLMs. The second one is fine-tuning [12, 17] where the parameters of pre-trained LLMs are continually updated by fine-tuned on the document containing knowledge in an unsupervised way as in pre-true pre-training [37]. In our work, we focus improving the tine-tuning-ie -ased solution, as storing new knowledge in the parameters of LLMs is efficient since we can reduce the length of the input prompt and do not need any extra module or memory in the deployment time [6].
Knowledge Injection in LLMs In this work, knowledge injection in LLMs denotes fine-tuning LLMs on the set of documents to inject new or under-represented knowledge into LLMs [33, 17]. different from another task of injecting symbolic knowledge (e.g.. knowledge graph) into LLMs [55, 54). Among previous works, CaMel [14] has introduced a meta-learning method for learnable loss scaling function that improves knowledge injection. As a concurrent work, MAC [45] has proposed using the memory of amortized context is highly effective in a knowledge injection. However, both methods have drawbacks like high computational costs for bi-level optimization or the need for additional modules and memory. Recent works [33, 58] have shown that data augmentation which paraphrases the knowledge-containing sentences helps language models memorize knowledge in more extractable format (e.g., asking questions) after knowledge injection. Furthermore, Jiang et al. [19] has shown that the instruction-ture tuned model better at learning new knowledge. Compared to previous works, we focus on developing an alternative method to data augmentation that perturbs the latent representation of LLMs for better knowledge injection.
Data Augmentation and Latent Perturbation The usefulness of data augmentations for text data was empirically observed in the literature. For instance, EDA [52] has introduced simple data augmentation method which randomly deletes, swaps, replaces, and inserts the words. Other previous works [22, 5, 30] have utilized the trained LMs to augment the text data. Recently, Maini et al. [29] has shown that adding data rephrased by L.LMs into the pre-training corpus improves the performance of LM pre-training. However, those methods require additional costs in the knowledge injection it utilize the LLMs to rephrase the text. In contrast, the latent perturbations offer an orthogonal approach to improve the robustness of neural networks, complementing data augmentation. This technique has been employed in meta-learning and out-of-distribution generalization [24, 25, 40J. For instance, NEFTune [16] demonstrated that adding noise, randomly sampled from uniform distribution, to token embedding layers improves instruction tuning performance. Expanding on the concept of latent perturbations, our work introduces a novel approach that internalizes the effects of text paraphrasing by identifying optimal latent perturbations through training small neural network within the LLMSs.
3 Problem Formulation
In this work, we follow the knowledge injection setting outlined by Ovadia et al. [33]. We are given
three resources: (I) documents DK containing knowledge that we are interested to inject; (2) question & answering dataset DoA = (() { for verifying injected knowledge from Dk; and (3) a pre-trained Large Language Models (LLMs) po(-) parameterized by 0. Our objective is to find transformation F that could enhance the knowledge about DoA:
8' = F(0, s.t. S(O' > S(B, DOA):
(1)
where the score function S is defined as:
S(0, Dar) == En Diz I(((p(q(())) = (1))
(2)
n
and IT.) and f(-) denote the indicator function and a decoding function that samples a sequence of tokens from Pe. respectively.
3

---

"Washington D.C.", LM Head
o (m
(1 m
-1)
minimize
KL Divergence
Transformer layer
771
(hout Mlatent, .... backward
d latent
Nh Metta a
MLP Softpluus
00
estiinate
latent Paraphraser
Sigmoid
Last token features
Last taken features
Layerfoom
Transformer Layers
Sampling
Transformer Layers
Layers with Latent Paraphrasers
Self-Attention
Wi,, I
LayerNorm
f
Linear
x.N
The oapital city the US
the uSu capital
Transformer layer
Linear
Original Sentence
Paraphrases
Token Embedding Layer
Transformer- caatmer--aseee based LLM
h
Paraphraser
(a) Latent CraaphhassrrAAhitttccii
(b) Latent Paraphraser Training
Figure 3: (a) Illustration of the latent paraphraser. The linear layer embeds each token's 's latent feature f into μa. We then sample stochastic noise from (u,, I) and apply mask control the scale. (b) Training pipeline LaPael. To train the latent paraphraser, we estimate parameters of Gaussian distributions. We then minimize the KL divergence between these distributions to optimize the latent paraphrasers.
In general, transformation F is fine-tuning LLMs documents in DR by optimizing 0 to minimize the negative log-likelihood of each token in each document as follows [33]:
8 arg min
Σ
log ( / sct)
DK
S
where s denotes the length of token sequence s.
Proposed Method
We propose Latent Paraphrasing of Language Models (LaPael), framework that perturbs the latent feature L.Mss, to achieve equivalent effect of data augmentation at the latent level. Knowledge injection using LaP consists of the following four processes: paraphrasing the set of documents to make the paraphrased data (Section 4.1), training the latent paraphrasers with paraphrased data (Section 4.2), fine-tuning LLMs with the trained latent paraphrasers Dk and evaluate the injected knowledge LLMs on DQA (Section 4.3).
4.1 Data Augmentation: Paraphrasing
To train the latent paraphrasers, we need a distinct set of training data Dtrain 1 consists of documents having different knowledge with Dk. As a preliminary, formulate the paraphrasing of the text in terms of the knowledge equivalence, which is narrower concept than semantic equivalence [23] where two different sentences can contain the same knowledge. We consider that each sentence in Dtrain can be decomposed into words for the object (entity or attribute) of the knowledge (z) and others (x) where both are the sequence of tokens. For instance, given the sentence "The capital of the United States is Washington D.C.".
Ji=1 which
x "The capital of the United States is";
represent the knowledge (United States, capital, Washington D.C.) Then, paraphrase sentence 8 (xc, y) into a paraphrased sentence For the above sentence, paraphrased sentence can be
"In the case of the United States, the designated capital city is"
with the same y. which is knowledge equivalent to (æ, y). For each knowledge K., we assume that there a set of the knowledge equivalent sentences S(CC) where (x, y) E S(C). We generate K para- phrased sentence via LLM: (I1, 3) .... (I 3) PLLM (x' |prompt, œ. y). Then, we have the set of paraphrased data {= ( ( N of Derain- We define p(æ' [x) ==
x, y)
which denotes the probability distribution of paraphrases given the original sentence.
'One possible way is to prompt the LLM (e.g., 3pt-3.6 5-turbo [31]) with instruction "For the following paragraph give paraphrase of the same high-quality English language in sentences Wikipedia" [29]

---

4.2 Introducing Latent Paraphraser
Latent Paraphraser We introduce a latent paraphraser within transformer layer [50], which augments latent feature and is expected to paraphrase the given input text within the latent space. As illustrated Figure 3(a). within the transformer architecture, we insert this new layer just before the Multi-layer Perceptron (MIP), using the output from the second LayerNorm as its input.
Let h IRd denote the latent feature after the second LayerNorm. The latent paraphraser, denoted by 96 : Rd > Rd and parameterized by Φ, augments the latent feature as follows:
go (h),
where o is the clement-wise multiplication. The function go (h) is given by: go (h) m) + 2,
(5)
with x E Id and m [0, 1] representing noise vector and learnable mask, respectively. The noise vector is generated by
Softplus(MLP2((a)), a~Nu, μ == W, + b
(6)
where MLP= is 2-layers MLP. We use the reparameterization trick [21] to enable the back- propagation through the sampling from the Gaussian distribution: a la E, where NN(0, I).
To modulate the scale of perturbation for individual tokens, we employ a learnable mask. It is important too much noise on key tokens (c.g.. United States) might hurt the semanties of the sequence. For learnable binary mask, we use concrete distribution to approximate the sampling discrete random variable from Bernoulli distribution using continuous relaxation [8] as follows:
m sigmoid ( log(w) + log(1 - u) + m
m
Wm h + by
where u ~ Oni 1), T is temperature, and m mask value in scalar.
Training Then, how do we train the latent paraphrasers to approximate optimal perturbation func- tions for estimating the distribution the paraphrased text? We employ the dataset with paraphrases H(=)) (1) ) 1 generated in Section 4.1. Our objective is to match two distributions for each transformer layer:
1. the distribution of transformer layer output feature for the last token hour without the latent paraphraser given the data perturbation distribution p(æ' (æ)) from Section 4.1:
Py (hout ax) = (ho (x') (xx (m:)d''
2. the distribution of output feature for the last token haue with the latent paraphraser given æ, Po.e (hout(u az). ) As a latent paraphraser outputs stochastic noise, we can formulate the probabilistic distribution pa (hout) (x) as follows:
Pa. (houe) (z)) =
pe (hout x. 2 (2) x x)d2,
(9)
where p2. a:) is the distribution for noise from the latent paraphraser in Equation (6).
We make the simplistic parametric assumption that both distributions are Gaussian:
Po (hout) (x) N (hout' Mdata odmaa I): (hout x)) Nhhout Mlatent; of latent
(10)
To train latent paraphrasers, we minimize the symmetric Kullback-Leibler (KL) divergence between two estimated Gaussian distributions of each layer follows:
(xL(w) = 2 [DKL (DeL(pp((ooaa) (ppp))pppoo (Nout (x)) + DKC (pe, (Nout [(oo
(11)
2 + (fiaata - Platent )2
DKL (pa (foout (z) Ipe (faout (x)) = log
(12)
data
20 latent
2
5

---

We employ Monte Carlo sampling approach to estimate the parameters of Gaussian distributions.
We generate N samples latentss to f from the distribution pe. (oott
(1)
(). Then, we estimate
the empirical mean and standard deviation from the samples as follows:
Matent
1
R latent
latent
Σ
hh
( latent
Matent)
(13)
N
N
and we use paraphrases æ1,-- ØK to obtain K samples data Po (hout æ²). Then estimate the parameters in the way:
(K) data
from the distribution
K Σ
K
Pdaaa
K
h
data
data =
|
K -
Σ
(ata
(b)
Maaaa)
(14)
We further use the auxiliary loss for mask training, with the sequence length of 'I' as follows:
mask
(x)
(Iigmoid((.)) r- Ti (s)gmoid(m.) - m&l).
(15)
where mrr is defined in Equation (7), E [0, is the mask ratio that controls the number of masks and m, the gold mask where m, for tokens that correspond the named entity.
To sum up, we optimize the latent paraphraser parameter & by minimizing the following loss:
arg min EeeDein (CKL(ar) + Lemask
(16)
See Figure 3(b) for illustration of the training process for the latent paraphraser.
4.3 Fine-tuning the LLM with the Trained Latent Paraphrasers
We fine-tune the LLM on documents containing knowledge be injected (Dk) as Equation (3).
We use the trained latent paraphraser parameterized by Ø" during LLM fine-tuning as follows:
|a]
log Pe. (8t 2 8ct)po (z) (sc)) ))
0 arg min
Σ SED,
D&
N
t=1
==1
(17)
where we sample N noise z0) by sampling multiple or from Gaussian distribution as defined Equation (6). Then, we evaluate the knowledge injected LLMs by measuring S(0", DQA) as defined in Equation (2).
5
Experiments
In experiments, we validate the effectiveness of the proposed method, LaPael, in injecting new or under-representted knowledge into Large Language Models (LLMss).
5.1
Experimental Setting
5.1.1
Datasets
To follow the experimental setup in Section
need (1) documents containing knowledge DR
and (2) associated QA datasets DoA. We mainly use the test split of three QA datasets: SQuAD [38]. StreamingQA [27], and ArchivalQA [51] for the source of Dk and DoA in our main experiments. These datasets, previously used in Hu et al. [14], consist of documents paired with their corresponding QAs, making them well-suited to our experimental setup. While the questions in these datasets are of decent quality, significant limitation lies in the documents provided. These documents are likely to
have been seen by L.MMs during pre-training, making difficult to accurately assess the performance of methods on injecting new knowledge.

---

Table 1: Data Example. Example data from SQuAD and StreamingQA dataset used in experiments. Words the yellow background indicate the answer to question. More examples are in Table 12 of the Appendix. Question
Raw Document
Synthetie Document
Who was the Super Bowl 50 MVP?
(..) Demer lineaaker Von Milley was named Super The Super Bowl 50 MVP wau Mon Milley
(fown SQWAD)
Bow] MVP recording five solo tackles 20thdd sacks,
two forced fumbles.
was name the veuse Bristel Heacon.
Calsion Hill which numed the 17th- centrry slave
Betore being renamed Beason last month the venue in Bristol was knows Colston Hall
Bristol, before it renamed last month?? (Jron Stroaminng(A)
Colston, will now on knowe as Brisnol Beacen following public consultation Beistal attracted headlines around (...)
Table 2: Experimental results on datasets with synthetic documents. trained with n sents means that latent paraphrasers are trained with the dataset containing n sentences. For ours, we report the average performance of three runs. i denotes the method that uses 10 times more additional data (paraphrases).
SQuAD-syy
StreamingQA syn
ArchivalQA
Method
EM
Recall
F1
EM
Recall
F1
EM
Rocall
Fl
No Injection
13.10
22.91
21.09
16.39
26.30
23.71
13.50
25.07
22.12
Fine-Tuning
66.30
79.32
76.11
82.08
88.98
88.29
62.60
79.51
76.16
Fine-Tunee Tinniininggin (seg.)
67.60
80.30
77.39
77.95
86.36
85,23
56.30
79.17
74.12
FreeLB [57]
70.70
82.41
79.67
82.24
89.48
88.56
63.20
81.30
77.67
NEFTune [16]
68.30
80.93
77.91
81.47
88.66
87.77
61.90
78,90
75.81
Ours trained w/ 50 sents.
70.77
84.96
81.66
86,16
93,01
92.12
68.37
86,24
82.67
Ours trained w/ 1K sents.
72.47
87.93
84.50
84.48
92.42
91.33
68.37
88,99
84.75
Fine-Tuning (+ para.)
68.50
85.12
80.51
85.45
93.67
92.32
64.90
85,92
81.24
To mitigate this issue, we incorporate two datasets with synthetic QAs Films 2024 and Events 2024. These are QA datasets generated from raw Wikipedia articles under the 2024 films category and from US events in May, June, and July 2024, in the 2024 events in the United States category. We generated question-answer answer pairs from these documents using GPT-4o following methods from previous works [19, 33J. Since the documents used to generate these datasets were not seen by the LLMs during pre-training, we can better evaluate the effectiveness of each method for knowledge injection especially on new knowledge.
Datasets with Synthetic Documents The raw documents from datasets are unsuitable for precisely measuring the knowledge injection performance. Specifically, fine-tuning LLMs on a document does not always ensure that LLMs can answer the associated questions, due to the reversal curse [3]. Moreover, documents often contain irrelevant knowledge that may hinder the accurate assessment of knowledge injection [14].
To address these issues, we conduct evaluations under the setting of synthetic documents. For generating synthetic documents, we construct DD by rephrasing each question and answer in DaA using GPT-4-turbo [32], ensuring that fine-tuning on these synthetic documents guarantee that LLMs become answerable to the associated questions. Examples of questions, synthetic, and raw documents are shown in Table 1. To make a difference, we denote the dataset under the synthetic document setting with the suffix '-syn' and the raw document setting with the suffix '-raw".
Datasets for Training Latent Paraphrasers For training our latent paraphrasers, the set of training data Derain is required in addition to DR. Therefore, we GPT--35-ttrro [31] to generate the set of synthetic sentences from the subset of training split of each QA dataset, where each sentence must be with the answer to questions, following the sentence format in Section 4.1.
5.1.2 Experimental Details
Baselines We compare our LaPael against several baselines. All models are fine-tuned on the documents in DK unless explicitly stated otherwise. (1) No Injection. We use the pre-trained LLM without any fine-tuning. (2) Fine-Tuning, We fine-tune the LLM on Dk. (3) Fine-Tuning (seq). We first fine-tune the LLM on the paraphrased documents of Derain- Then, we fine-tune the LLM on Dk.
(4) Fine-Tuning (+ para). We fine-tune LLM on the original and paraphrased documents of DR. (5) FreeLB [57]. We add trained adversarial noise to the token embedding while fine-tuning. (6) NEF 16). We add random uniform noise to the token embedding while fine-tuning. (7) LaPael (ours). We train the latent paraphrasers on Derain and then fine-tune the model on Dk.
7

---

Table 3: Experimental results on datasets with raw documents. For Ours, use the latent paraphraser used in the SQuAD-syn experiment. Rec. denotes recall.
SQuAD-raw
StreamingQA-raw
Films 2024-raw
Events 2024-raw
Method
EM
Rec.
F1
EM
Rec.
F1
EM
Rec.
F1
EM
Rec.
Fl
No Injection
9.98
23.44
20.62
16.22
29.85
27.81
1.93
10.21
10.27
1.73
17.94
17.40
Fine-Tuning
16.65
35.40
29.73
19.04
35.88
32.92
13.39
30.03
28.84
10.98
43.76
39.62
FreeLB [57]
17.04
36.78
30.36
20.72
37.19
34.04
15.47
33.69
31.85
14.68
46.05
41.85
NEFTune [16]
17.45
37.49
31.11
20.18
36,98
33.85
15.93
33.73
32.38
15.38
48.14
43.84
Ours
18.96
43.10
34.65
21.62
39.38
35.32
16.29
35.04
32.56
15.26
56.70
46.45
Table 4: Experimental results on cross-domain transfer experiments. For ours, (X -)) denotes that latent paraphrasers are trained on Drain from the X dataset. Rec, denotes recall.
SQuAD-syn
Streaming syn
NovelQA-syn
MedMCQA-syl
Method
EM
Rec.
F1
EM
Rec.
F1
EM
Rec.
F1
EM
Rec.
No Injection
13.10
22.91
21.09
16.39
26.30
23.71
9.17
18.21
16.05
39.00
48.68
47.82
Fine-Tuning
58.30
68.59
66.35
74.73
82.34
81.21
52.92
66.30
63.62
56.10
62.37
62.03
FreeLB [57]
70.70
82.41
79.67
82.24
89.48
88.56
55.42
67.39
64.80
57.90
63.17
62.81
NEFTune [16]
68.30
80.93
77.91
81.47
88.66
87.77
51.67
65.14
62.25
56.30
62.57
62.09
Ours (SQuAD -)
72.50
89.38
85.34
84.38
93.44
92.17
$4.17
69.40
65.72
63.70
68.28
67.98
Ours SStreamingQA -+)
72.80
89.65
85,90
84.06
93.73
91.90
54.58
72.58
68.15
63.20
68.02
67.79
Training & Inference We mainly use Vicuna-7b-v1.55 [56] for fine-tuning, which is the instruction- tuned version of Llama-2-7b [48] for our experiments. We fine-tune LLMs for 12 epochs with a learning rate of 0.00005 and step learning rate scheduler where we decay learning rate by 0.85 by every epochs. For inference, we use in-context learning with 5 examples by prompting the examples in the prompt [4]. To measure QA accuracy, use Exact Match (EM), Recall (Rec.), and Fl score. More details on the experimental setting are provided in the Appendix C.
5.2 Experimental Results
Experiments with Synthetic Documents In Table 2, we present the experimental results for the synthetic documents setting. Fine-tuning does improve the QA performance of LLMs, but it does not lead to near-perfect scores even though the synthetic document contains the necessary knowledge for answering the questions, as shown in Table 1.
Our experiments show that paraphrasing documents for fine-tuning consistently improves QA per- formance across all three benchmarks. Notably, LaPael demonstrates performance comparable to fine-tuning with paraphrases on StreamingQA and even outperforms it on two other benchmarks. These findings suggest that the latent paraphrasers learn an effective noise distribution that aids knowledge injection without additional data augmentation.
We also compared LaPael with two other noise-based methods, FreeLB [57] and NEFTune [16], to validate that the latent-level noise generated by latent paraphrasers is more effective. As shown in Table 2, LaPael outperforms these baselines, confirming the strength of our approach.
Experiments with Raw Documents While our method has proven effective for knowledge injection with synthetic documents, it is important evaluate its performance raw documents, which represent a more realistic data format. To demonstrate the applicability of our method to real-world data, we conducted experiments in which we fine-tuned LLMs on raw documents for each dataset, using latent paraphrasers trained on Prrain from SQuAD-syn.
As shown in Table our method outperforms both fine-tuning and noise-based baselines in the context of knowledge injection with raw documents. Considering that the latent paraphrasers were trained synthetic sentences from Drrain these results demonstrate their effectiveness documents with different format than those used in training.
Cross-domain Transfer Once trained, the latent paraphrasers can be applied to fine-tune LLMs on documents from any domain. To demonstrate this, we conducted cross-domain transfer experiments Specifically,, we trained latent paraphrasers on Derain from source domain (e.g., SQuAD) and fine- tuned LLMs with the trained latent paraphrasers on Dk from target domain (e.g., StreamingQA).
8

---

94
93
84
85
62
2
91
60
Fibe-tuning
90
Fibe-tuning
Fine-tuning
78
20
4
20
The Number of Paraphrases
The Number of Paraphrases
The Number of Paraphrases
(a) SQuAD-syn
(b) StreamingQA-syn
(c) ArchivalQA-syn
Figure 4: Effect of the Number of Paraphrases. Each plot shows the relationship between the number of paraphrases (X-axis) and Fl scores (y-axis) in knowledge injection. The Fl scores of both standard fine-tunn finettning
and our method improve as number of paraphrases increases.
NEFTune
FreeLB
Ours (mean)
QA Accuracy (F1)
90
VA Accuracy (F1)
80
70
# layers=3
60
# layers
80
50
# layers=7
78
510 25
50
75
100
3 5 7
9 11 13 15 17 19 21 23 25 27 Start Position of Latent Paraphrasers
Perturb Training Dalaset Size (%)
(a) Varying ratio training data
(b) Varying position of latent paraphraser
Figure 5: (a) We conduct experiments varying the size of Derain on D-syn, where 100% indicates 1,000 documents. We report mean and std. over three runs. (b) We conduct experiments on StreamingQA varying the start position of latent paraphrasers where '# layers' denotes the number of latent paraphrascrs.
As shown in Table 4, our method successfully transfers across domains, with the latent paraphrasers enhancing the performance of the knowledge injection on NovelQA and MedMCQA-wwo domains distinct from the source (see Appendix C.1 for details on these datasets). Even though both domains contain specialized entities, our method consistently outperforms standard fine-tuning and other noise-based baselines.
Combining LaPael and Paraphrases Paraphrasing documents in Dr has been shown to improve knowledge injection performance, as seen Table 2. While LaPael significantly improves perfor- mance without requiring paraphrases, it is valuable consider the effect of combining paraphrases with the latent perturbations from LaPael. As illustrated in Figure 4, LaPael consistently outperforms standard fine-tuning, showing that LaPael provides advantages over data-level augmentations.
5.3 Ablation Studies
Effects the Size of Dirain LaPael needs additional data Dtrain for training latent paraphrasers. Although only small amount of data is required, it might be unclear how much is needed to make the latent paraphrasers learn the useful noise distribution. As shown in Figure 5a, LaPael works well even with 50 sentences for Derain while increasing the size of D ensures steady performance improvement for LaPacl.
Effects of the Position of Latent Paraphrasers Our latent paraphrasers can be inserted into any layer of the I.Mss. The possible question is which position and how many layers are optimal for latent paraphrasers to effectively learn noise for knowledge injection. To answer this, we analyzed the position and number of latent paraphrasers.
In Figure 5b, we show the QA accuracy results, varying the start position and number of latent paraphrasers. The first layer is the closest layer to the input layer, and "start position 1" with "# layers = means we insert the latent paraphrasers into the first, second, and third layers of the LLM. Results show that inserting three latent paraphrasers into the early layers the LLM effective. This is consistent with findings in previous works [16, 57, 25] where using noisy token embeddings (the lowest layer) enhanced the generalization I.LMs. Furthermore, in Table 5, we empirically show that positioning the latent paraphraser before the MLP layer within each transformer layer is the most effective choice over other positions.

---

Table 5: Analysis on the Position
Table 6: Ablation studies on Mod- ules in latent paraphrasers.
Table 7: Ablation studies on Noise de- sign in latent paraphrasers.
inside the Transformer layer.
StreamingQA
EM
Recall
F1
StreamingQA
Recall
StreamingQA
EM
Recall
F1
Before MLP
$4.06
91.90
LaPael
84.06
93.73
91.90
Learnable Mul.
84.06
93.73
91.90
After MLP
73.81
K2,58
81.02
w/o Mask
77.95
85.17
84.63
Learnable Add.
73.05
83.23
81.70
Gaussian
83.46
90.32
89.54
Before Attn
80.55
87.58
86.49
wro Concrete
73.35
83,42
81.99
Gaussian mark
82.85
89.70
88.87
After Altn
83.31
90.98
89.72
w/o Sampling
84,23
90.52
89.73
Uniform
79.48
87.17
86.15
Token Embed.
$6.21
91.79
91.05
w/o KL. loss
89.99
Uniform + mazk
74.43
81.09
80.26
Ablation Studies on Modules LaPael has many design choices concerning the latent paraphraser architecture, noise type, and training. We conducted extensive ablation studies to empirically verify each design choice and provide guidance for future work. In summary, as shown in Table 6, all design choices are important for building the most effective latent paraphraser. Specifically, we use trainable mask m in Equation (7) regulate the perturbation depending on each token, which is crucial, as the performance on StreamingQA drops significantly if we remove it from the latent paraphraser. Furthermore, using only the sigmoid function in Equation (7) instead of the concrete distribution also leads to much lower performance, as the mask is not properly trained. Regarding noise training, using deterministic noise instead of stochastic noise by removing the noise drawn from Gaussian distribution in Equation (5) also decreases performance. Additionally, replacing the KL loss with Mean Squared Error loss between two means Mateet in Equation (13) and Adata in Equation (14) leads to a decrease in performance. confirming the importance of stochastic noise trained with KI. loss.
Ablation Studies on Noise Distribution Should we train the latent paraphrasers to be effective, or can adding random noise in the early layers also be effective? Which is more important: the learnable mask or the learnable noise? To answer these questions, we conducted ablation studies on the choice of noise distribution. In Table 7, Learnable Add. denotes the model with the additive noise h + Io (h) instead of Equation (4) without softplus from Equation (6). Gaussian is the use of zero-mean Gaussian noise (0, I) in Equation (6) without using MLP,.. .- Uniform is the use of noise drawn from the uniform distribution defined in NEFTune [16] instead of z in Equation (6).
As shown in Table 7, the learnable multiplicative noise described in Section 4.2 is the best design for noise distribution used in the latent paraphraser. To analyze the effect of the learnable mask, we also added the learnable mask to the Gaussian and Uniform noise settings and optimized only W.. m and bm in Equation (7) with loss in Equation (15). Interestingly, the learnable mask is not effective for the fixed noise distribution, which contrasts with the results for learnable noise in Table 6. We conjecture that using the learnable mask is important for input-dependent learnable noise, as it can allocate different noise scales different tokens, while this is not the case for static noise distribution.
6 Conclusion
We have introduced I LaPael, a method for enhancing knowledge injection in Large Language Models (LLMs) by applying learned perturbations to their layers. Unlike traditional data-level augmentations or noise-based approaches, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability, LaPael addresses key limitations of existing methods by reducing computational costs and increasing the diversity of augmented data. Our extensive validation across diverse benchmark datasets demonstrates the superiority of our method in knowledge injection, as it significantly outperforms both standard fine-tuning and other noise- basse baselines. Moreover, combining LaPael with paraphrases yields complementary benefits, further enhancing performance. We believe that LaPael, being simple yet effective, has the potential for significant practical impact and will encourage further research on applying perturbation within the latent space of L.MSs.
Discussions & Limitations In our work, the following points can be discussed further: (1) Cost Analysis- LaPael is effective, it incurs additional costs due to the need for training latent paraphrasers and fine-tuning LLMs with them. (2) Knowledge Retention-Alther knowledge injection, there may be trade-offs the LLM has memorized. (3) Comparison to Aetrivval-Augmenter Generation (RAG)-Wo While our method improves knowledge injection, is still less effective than RAG terms of performance. We provide detailed discussion of these points, along with other limitations, in the Appendix A.
LaPael improves
terms of retaining the original knowledge that
10

---

Acknowledgement
We sincerely thank Byeongju Kim, Jongwon Jeong. Jimin Hong, and Jongho Park for their insightful discussion. This work was fully supported by the KRAFTON AT Research Center.
References
[1] Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang. Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang. Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang. Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv: /42 2024. doi: 1044850/ARXX1.240..4222 URL. htRl:ttps://aao org/ 10. 48550/ arxiv. 2404 14219.
[2] Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David A. Sontag. Large language models are few-shot clinical information extractors. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2022.. Association for Computational Linguistics, 2022. URL https: /0//6663/1/22/ emnlp-main. 130.
//doi.
[3] Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMS trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Representations, 2024. URL
http://ppnnevie...ntttto
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL https: / http://eecccecee..//////// httpooceeediiinnennnnnirreeeeeee httpseesi...pppaeee///....... htcp/pppp///22//////// /hash/ Awsteaatttt.............. html.
[5] Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, and Dawei Yin. Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6334-6343. Association for Computational Linguistics, 2020. URL htttps://dot http://ii.oogg///// 10. 18653/v1/ 2020. acc-mmiin 564.
[6] Maria Angels de Luis Balaguer, Vinamra Benara, Renato Luiz Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry. Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. RAG vs fine-tuning: Pipelines, tradeoffs, and a case study on
11

---

agriculture. arXiv 20..0.000000..0.000000 /ws//401.00//000000000 2024. doi: 10.48550/ARXIV. 2401.08406. URL lattps://dol org/10
10. 40450/0555aaa arxiv. 2401. 08406.
[7] Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023. URL https://doi. org/10. 10.. 48550/am arriv 2305.07777
[8] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 3581-35999 2017. URL htttps::/ppoceeeinggg www.riinnnnnnunninnneeee neurieerpppp psurips... wwtppppppppppctppcccccccccp.waaa paaper2217/haae //hhh/ 84ddfb34126f 84odfbb4rBBscaaeeee877444-777 Awweoaatttttttaeeeeeeeeeeeaeaaet html.
[9] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning Ilms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.0500 abs/2405.05904, 2024. URL https://dol: org/ 10. 48550/ 2405. 05904.
[10] Olga Golovneva, Zeyuan Allen-Zhi ZZu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training nurse the reversal curse. arXiv preprint arXiv:2403. 13799, 2024. URL https://do//i org/ 10. 28550//rXiv.. 240.......4.......
[11] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi L.i. Textbooks are all you need. arXiv preprint arXiv:2306. 0223306.116600 11644, 2023, doi: 10.4850ARXIVV2306.11 48550/arXiv 2306 11644.
URL https://doi. org/10.
[12] Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8342 8360. Association for Computational Linguistics, 2020. URL https://det org/10 10. 18653/663// 118666/1/22200 acl-main. 7400.
[13] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRÁ: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL httt/forum?dd=nZeeWWeee.
[14] Nathan Hu, Eric Mitchell, Christopher D. Manning, and Chelsea Finn. Meta-learning on- line adaptation of language models. In Houda Bouamor, Juan Pino, and Kalika Bali, edi- tors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4418-4431 Association for Computational Linguistics, 2023. doi: 10. 18.13// 1/2023 EEE.SSENNL-MMAII MAIN.22... https://doi. org/10. (8653/vv//// htt//////222...// emnlp- mww..
URL
[15] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination large language models; Principles, taxonomy. challenges. and open questions. arXiv preprint 2.002332 2023. URI. httpp:://oi. org/ /0. 48540/8555500 /aro0v//rxvv../.. 2311 1.212..
[16] Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy embeddings improve instruc- tion finetuning. In The Twelfth International Conference on Learning Representations, 2024. URL httph::t///o/oo //pprrevv/e/eeeieeeeemmmm net/ www.ii/wwweeeettnnnneeeoe fdo0m
[17] Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoo Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models.
12

---

In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openrevienw wwt/forumfifeeeoo vfsRB5MImo9.
[18] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b. arXiv preprint arXiv:23100068/2 2023. doi: 10.48550AARXII.23000666 URL https://doi org/10. 48550/arXiv 2310. 06825.
[19] Zhengbao Jiang. Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan lyer. Instruction-tunet language models are better knowledge learners. arXiv preprint arXiv:2402. / /2847, 2024. doi: 10.48550/ARXIV.2022. 12847.
URL htt.tsttp///i.... org/ 10. 48550/ arxiv.
12847.
[20]
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle learn long-tai tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 1 15696-15707. PMLR, 2023. URL https: https:// httpss///occceeeeeeeeeeee mlr (eass/v202/kandpaapa
html.
[21] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http: /ttp///aavvv///////////e.//..av. org/ wwwp////......... 6114.
[22] Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Short Papers), pages 452-457. Association for Computational Linguistics, 2018. URL https://ddii.. http///ti.ooogg.o/////o/// 11t.11/1///11/1101000
[23] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: //tpne/eeee/eeeeee///eee/ee net pdf --d-----AtPOdvee
[24] Haebeom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
[25] Seanie Lee, Minki Kang. Juho Lee, and Sung Ju Hwang. Learning to perturb word embeddings for diito--disttibution QA. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5583-5595. Association for Computational Linguistics, 2021. URL https://do:.i org/ 10. 1tt.1866/vv// v1t///1112221211 long.llon..... 434.
[26] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebas- tian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12. 2020, virtual, 2020. URL https: https://proceddings wwwpoceeeinngs.nurippps.s.. nwwripe.cc.ceeeeoooo wwwp//pppe////////ae/e 660522000577001122244444/888811 Aaa.leet.-bbttttt..t
[27] Adam Liska, Tomás Kociský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan- McMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. Streamingqa: A benchmark
13

---

for adaptation to new knowledge over time in question answering models. In Kamalika Chaudhuri, Stefamie Jegelka, Le Song. Csaba Szepesvári. Gang Niu. and Sivan Sabato, editors. International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 13604- 13622. PMLR, 2022. URL https: http:///poceeddedeee httpp/gg.....rpprrssreeeeeeeeeee arrass/v6621issa2222 html.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:://pperreeeeeeee http:/ppporovi.....teettorrrrr//
[29] Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephras- ing the web: A recipe for compute and data-efficient language modeling. arXiv preprint UrXiv:2401.163800 2024. URL https: org/ 10. 48550/ arr.v..iiiv...i../.eeaeaeaeee.e. ww..ii..200.... 14380.6380.
[30] Nathan Ng. Kyunghyun Cho, and Marzych Ghassemi. SSMBA: self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1268-1283, Association for Computational Linguisties, 2020.
[31] OpenAl. Introducing chatgpt. https: //tp:///pp/aaa/aa hatpe/ai..coombooocchaaaaeao 2022.
[32] OpenAL. GPT-4 technical report. arXiv preprint arXiv:2303.087774 2023. URL https:
//doi. org/10. /twg///.8850///// 48550/8555503 arxiv. 2303. 087744.
[33] Oded Ovadia, Menachem Brief, Moshik Mishacli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934, 2023. URL https: //doi. org/oogg//1 100. 48550/ arxiv. 05934..
[34] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H. Chen, Tom J. Pollard, Joyce C. Ho, and Tristan Naumann, editors, Conference on Health, Inference, and Learning, CHILL 2022, 7-8 April 2022, Virtual Event, volume 174 of Proceedings of Machine Learning Research, pages 248-260. PMLR, 2022. URL https: //proceedings.. mlr. press/ grres/vv14/palle hwtp///121/aa.....///// html.
[35] Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023. URL org/10. 10. 48550/arXiv 2310 06786.
https://dot
[36] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 2463 2273. Association for Computational Linguistics, 2019. URL https: //oii//oooooo/ org/10. 18653/v1/D19.. 12500
[37] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[38] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392 The Association for Computational Linguistics, 2016. URL httpp::///att //doi. org /00 18653/v1/d16
[39] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmenner language models. Transac- tions of the Association for Computational Linguistics, 11:1316-1331, 2023. URL. https: / //aclanntoooo org/ org/2023. 2023 tac1-1.75.
14

---

[40] Jeongun Ryu, Jaewoong Shin, Haebeom Lee, and Sung Ju Hwang. Metaperturb: Transferable regularizer for heterogeneous tasks and architectures. In Advances in Neural Information Processing Systems 33: Annual Conference
Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 1-, 2020, virtual, 2020. URL htt.attts//proceeddnng cc/ cc/paper/ (01220/aah//4440 /tt.///4aaaaa444411/////.144 222.8880000400000744777777776-00 Absttrar html.
www.rdddiinnnnniuns....e. neurips.
[41] Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Francier Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6138-61 11336618.. 48. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021 1/T.///22211122-/.M2 496. URL htthttt::/// //doi wtt/.o...///......... 10.11653//112221.
10.
[42] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: retrieval black-box language models. CoRR, abs/2301. 2023. URL htwph:ttp///// /tsttps:/doi.. org/ / 10. 48540//55555.. 2301 12652.
[43] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13153 2022. URL https://doi. org/10 10. 48550/arXiv arxiv. 2212. 13138.
[44] Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Hcad-to-tail: How knowledgeable are large language models (Ilm)? A.A.A. will Ilms replace knowledge graphs? arXiv preprint arXiv:2308. 10168, 2023.
[45] Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. Online adaptation of language models with a memory of amortized contexts. arXiv preprint arXiv:2403.060 04317, 2024. URL http::::/dooooo//////
2403.
[46] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunoii B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://g/ gi thub. wtw http://tttulaabstaaatea frd alpaca, 2023.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and effi- cient foundation language models. arXiv preprint 13X11. 2023. URL https:
/ //doi. 0000..8550aaaa arviv.. 2302. 13971.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferree Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Hric Michael Smith, Ranjan Sub- ramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open foundation and fine-tuned chat models. arXiv preprint DrXXi:2307.09288 2023. URL https://doi: or0/10....111.. 48550/arXiv 2307 09288.
[49] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 986)):557--2605 2008 URL http: ht j/jmr. cammmrorr/// wwalll.ogggpppppp wtrg/apprss/// vandermaaten08a. html.
15

---

[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information P Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[51] Jiexin Wang. Adam Jatowt, and Masatoshi Yoshikawa. Archivalqa: A large-scale benchmark dataset for open-domain question answering over historical news collections. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR 22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 3025-3035. ACM, 2022. URL https: //doi. org/ /0/ 1145/3477423 3531734.
[52] Jason W. Wei and Kai Zou, EDA: easy data augmentation techniques for boosting performance on text classification tasks, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-JJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 6381 -3877. Association for Computational Linguisties, 2019. URL https: //toi.//ooooo org/10. /0/ 18653/v1/ D19-1670.
[53] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597, abs/2309.07597, 2023. URL httpp:://oi.ooro org/ 10. 48550/ arriv./rrvv.vv..o 2309. .05977.
[54] Qinggang Zhang, Junnan Dong, Hao Chen, Xiao Huang, Daochen Zha, and Zailiang Yu. Knowgpt: Black-box knowledge injection for large language models. arXiv preprint arXiv:2312.06185, 2023. URL https://dooi orw/o.og///oo 10////0.../ 000444550//// (ar550/rxxi.... 2312.06185.
[55] Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang. Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Plug-and-play knowledge injection for pre-trained language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 6l Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pages 10641-10655 Association for Computational Linguistics, 2023. URL https: :/tp::///diii org/10. 18653/v1/ 2023 acl-long. 594.
[56] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors, Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL http: /ttp///aappeseeee/ee nips. cc/ wwwp/ppe__fftee wwallleeppper/e/eee 2023/ / 1411881127033888386
Abstract Datasets_ _nd_sand_____aa_______ Baaeeetsssan_Beechaaaaaaeeee Be__Bhennchaksssee .tmml.
[57] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL https: http://pppeeeiieeeeeeeeeeee net/ font:/foummmaa Bdyggunyyy.ggyg
[58] Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint 2rXiv:2309.143331 2023. URL https: http:/doioooo/////// org/10. /tgg/0..85500 48550/ arXiv, 2309. 14316.
16

---

Appendix
A Discussions & Limitations
Cost Analysis Our method requires additional costs compared to the fine-tuning baseline. Specifi- cally, it involves two extra computational costs beyond standard fine-tuning. A comparison of the per-step computational cost (in GFLOPs) between the baseline and our proposed method is shown in Table 8, where we consider fine-tuning LLMs with 7B parameters. detail, one forward pass of 7B parameter LLM requires 13.21 GFLOPs, and one backward pass costs twice as much as forward pass. The latent paraphraser model we used in the experiments consists of 5 paraphrasers, each with 4 linear layers, totaling 250M parameters, which is 3.6% of the parameter size of the LLM. The total computational costs can vary depending on the hyperparameters (c.g., N in Equation (13)) and the size of the dataset used.
While training the latent paraphrasers requires an initial cost, this is a one-time expense. Once trained, these can be used repeatedly for knowledge injection without additional ongoing costs. This makes the overall expense relatively low in the long term. Furthermore, incorporating latent paraphrasers during fine-tuning adds only minimal computational overhead, as their parameter size is just 3.6% of the size of LLM.
Knowledge Retention A common drawback of knowledge injection is the potential for LLMs to forget previously learned knowledge [17]. To assess this issue, we used the EntityQuestions dataset [41]. which contains simple questions about entities. Specifically, we focused on "place-of- birth" questions for well-known entities (e.g.. "Where was Leonardo da Vinci born?",, with 988 questions in total. We fine-tune the Vicuna-7b-vl.5 [56] on a synthetic SQuAD document set (Dx) using each method, then measure its QA performance on the EntityQuestions.
As in Table 9, the experimental results show that all fine-tuning approaches negatively impact knowledge retention, as observed in the previous work [9]. Additionally, we observe that improved knowledge injection often comes at the cost of greater knowledge forgetting. Although our primary focus is on enhancing knowledge injection, we acknowledge that addressing knowledge retention is crucial and should be focus of future research.
Comparison to RAG based approaches like Setrieval-Augmentee Generation (RAG) [26], lie in their simplicity and reduced computational cost on the inference [6]. Fine-tuning results in a self-contained model, which simplifies the system architecture by removing the need for additional components like document retrieval and ranking during inference. This reduction in complexity leads to lower computational overhead, especially in terms of GPU memory usage due to the shorter length of the prompt, making fine-tuning more suitable for an LLM deployment in cooouucc-constrainee environments.
The primary advantages of fine-tuning methods, including ours, over retrieval-
However, it is important to check the performance gap between them. Therefore, we experiment with RAG on the Events 2024 dataset with Vicuna-7b. For ours, we follow the same experimental setting with Table 3. For RAG, we use the bge-large-en-vl.55 [53] model for document and query embedding for retrieval. In Table 10, our experimental results indicate that the RAG approach outperforms fine-tuning methods including ours, as previously observed by de Luis Balaguer al. [6]. However, our LaPael method narrows the gap between the two approaches, suggesting that there is potential for further improvements in fine-tuning strategies.
Table 8: Per-step computational cost comparison on the 7B LLM.
Table Zero-shot question answering performance on Entity QueitiQusss after fine-tuning LLMs on SQuAD-raw.
Table 10: Comparison to Retrieval Augmented Generation (RAG) on Events 2024-raw.
Method
GFLOPs
Baselines
EM
Rec.
F1
EM
Rec.
Fl
Fine tuning LLM
39.63
No Injection
59.00 64.38 63.46
Proposed Method
Fine-Tuning
52.23 55.63 55.41
Fine- Fine-Tuning 10.98 43.76
39.62
Training Latent Para. (LaP)
14.64
Ours
39.88
41.97 42.12
Ours
15.26 56.70 46.45
Fine-tuning LLM w/ LaP
40.11
Fine-TTuning (+para)
33,50 35.18 35.28
RAG
27.17 64.02 55.71
The proposed method is unable to address the reversal curse, where the Large Language Models (LLMs) trained on "A is B" fail to answer "What is B?" [3]. As outlined in Berglund
Reversal curse.
17

---

et al. [3], this phenomenon is mainly due to the format of data and the autoregressive nature of LLMs that are trained in a way from left to right. Therefore, is limited to improve the knowledge injection performance if the document does not contain sentence having the reverse relationship, even with our method. Future work will need to explore the combining of our method with recent solution for the reversal curse like reverse training [10]. Otherwise, we can seek solution that addresses the reversal curse at the latent level similar to LaPael, which can be an interesting direction for future work.
Limited scope of Task and Experiments. The scope of our method remains limited in the knowl- edge injection task. Specifically, there are challenges in applying LaPael for continual pre-training on large-scale corpora, such as the 15B Open OpenWebMath dataset [35], or for instruction tuning with datasets like Alpaca [46]. Addressing these challenges will require future work as a new approach for training latent paraphrasers tailored to other tasks. In terms of experiments, our experiments only focus on the 7B LLMs, and do not conduct any experiment on larger LLMs of size with 13B or 70B [48] due to the limited computational budget for experiments.
B Broader Impact
This work explores the knowledge injection in Large Language Models (LLMs), which are highly related to hallucinations [15]. While our method improves the addition of new knowledge to LLMs, also increases the risk of introducing misinformation. Specifically, our method could enhance the inaccuracies in LLMs when they are fine-tuned using documents that contain incorrect facts. Therefore, it is crucial to thoroughly check the documents used for fine-tuning LLMs before applying method to enhance knowledge injection.
C Experimental Details
C.1 Dataset
Table 11: Dataset statistics. report size Drrain Dx.
DQa used our experiments.
Synthetic Documents
Raw Documents
Dataset
SQuAD
StreaminQAA
ArchivalQA
NovelQA
MedMCQA
SQUAD
StreamingQA
Films 2024
Events 2024
D Danin 1,000
1,000
1,000
DR 1,000
653
1,000
240
1,000
2,067
1.688
1,202
175
DON
1,000
653
1,000
240
1,000
10,570
1,665
5,968
865
As briefly mentioned in Section 5.1, we generate the synthetic document from each question-ans -atiiionnnnsweer wer pair using GPT-4-turbo model [32]. To generate the documents from the question and answer pairs, we use the prompt in Table 13. To generate diverse paraphrases from Derain, we use the prompt [29] in Table 14 using GPT-3.5-turbe model. For cross-domain transfer experiments, we also use the subset of MedMCQA [34] and synthetic NovelQA dataset based on the Les Misérables Wikipedia page, where we generate the synthetic document for each question. For MedMCQA [34], we use the subset of the dataset where the domain of question corresponds to the anatomy.
We summarize the statisties of the synthetic dataset used in our experiments in Table 11. We also plot the distributions of token counts in documents, questions, and answers for each dataset used in our experiments in Figure We present the example of each dataset in Table 12.
C.2 Training Details
As briefly mentioned in Section 5.1, we mainly use Vicuna-7b-vl.55 [56] for fine-tuning, We fine-tune LLMs for 12 epochs with learning rate of 0.00005 and step learning rate scheduler where we decay learning rate by 0.85 by every 4 epochs. For experiments in Figure 4, we finettune for epochs with decaying period as epoch. For optimizer, we use Adam W [28]. For all experiments, we only update the parameters corresponding to the MLP layer of transformer [50]. For Llama model [47, 48), it corresponds to linear layers named up_proj. gate_proj] and down_proj. We use 4 A100 GPUs for fine-tuning L.Mss. For inference, we use in-context learning with examples by prompting the examples in the prompt [4].
18

---

For training latent paraphrasers, we train them for 10 epochs with learning rate of le - 3 and cosine learning rate scheduler where we linearly decay learning rate to 10% of the initial learning rate without warmup. We use 5 latent paraphrasers on the 5 sequential early layers of L.Mss. For Equa- tion (13), we use N = 4. For Equation (14), we use K 10. For Equation (15), we set = 0.5. For gold mask m,, we use a similar method to Agrawal et al. [2] to find the named entities from each document using GPT-3.5-turbo. For fine-tuning with latent paraphrases (Equation (17)), we use N 4.
Table 12: Data Example. Example data from all datasets we used in experiments. Words in the yellow background indicate the answer to the question. Hypen (-) in the original document column indicates the case where the original document is not accessible.
Question
Original Decument
Syathetic Document
name (fross Stroming())
Minister?
(..) Anz 2020/ phono, new speaks during press con- forence in Kharoum, Sudan. (..)
Minister is Abdalls Hamdol
Which NEL leam represented the at Super Bonl 50? (fow SQUAD)
(.)) The American Football Conference (AFC) pian Danver er Broncos deleated the National Foothall Conference champlos Camiina Pardhers carn
The represettative at Super Bowl 50 was the Curolina Panthers
their dhidd Super Bowl (..)
What country's 's semi-officcil telesision netwock broadcast dinner? (from ArchnallgA1
Bush's was broadcast the semi official television network Japort.
graft inguinal approuch bypaes
infrainguinal bypass surgery, the pretemedd type graft for optimal ouncomes Autologous yein
Ducron PTTE (C) Pulyester Antologuus vein (frem MeIMCQAI
What lown does Jean Viljean becomne mayor of? (Dow NovelAA)
Jem Viljess boconies the mayo of the lowa Mone Heusl sue Mar.
How many title were screened in person the 23rd New York Asian Film Festival?
The 23d New York Asim Film Festival was held in
New York on 12 July with World Premiore of South
(Pro Eiensy 2024)
Korean film Vietsey by Park Beom su, who uncoded the screeting in person. In the 23rd edition 98 titles were screened in person. (.)) Dine: Firi Twon is 2024 opic science fiction nim directed and produced by Dems Villenenve who co- wrote the screcnplay with Jos Spailts. The sequel (..)
Who directed and produced Dune: Part Two! (fro Friver 2024)
Stats. SQuAD (synthetic)
Stats, StreamingQA (synthetic)
for ArchivalQA (synthetic)
10.2%
15
0.0%
10
20
23
0.0%
30
40
0.7%
10
16
20
25
25
40
Ptokera queniia
Pooksm quantizz
#rokems quaetiio
10.0%
1534
10.2%
0.0%
23
50
33
45
0.5%
12
20
6)
6
70
0.2%
15
25
23
33
Ploeeee snoww
Protene in ar starew
S1.3%
#nitone in in srewer
16.5%
50
125
15.0 175
0.0%
12
12
(a) SQuAD in Table 2
(b) Streaming in Table 2
(c) ArchivalQA in Table
Stats, for NovelQA
Stats. for McdMCQA
Stats. for SQuAD (raw) #sobros in docummin
20.00 97
Phicene docemont
#rabene docement
10.1%
10.54
Pookens
23.5%
queetion
quantion
20 Phobene ancwor
0.0%
20 Pivtenn
0.5%
10
0.0%
0.09
12
14
0.38
20
(d) NovelQA in Table
(e) MedMCQA in Table 4
(f) SQuAD in Table 3
Stats. for StreamingOA (raw)
Stats. Pilms 2024
Stats, US Byents 2024
0.1%
20.5%
28.0%
0.0%
300
1300
2000 #tocenn quertisn
1503
0.3%
330
250
300
405
570 660 703 850
0.0%
200
400
003
500
1300 1250 1450
15.00
16.15
4ikkens question
11.6%
ttlkoza io quuutuu
0.0%
15
25
60
50
7o
0.3%
10
15
20
25
30
db
0.0%
20
40
22.5%
Plokeres anowor
8..5%
25%
Kitlkoss in in answer
30
45
65
25
100
125
170
0.04
15
70 50
(g) StreamingQA in Table 3
(h) Films 2024 in Table 3
(i) US Events in Table 3
Figure 6: The distributions of token counts in documents, questions, and answers for each dataset used in our experiments.
19

---

Table 13:
Prompt for Synthetic Document Generation. An 1-shot prompt for generating the synthetic document from the question.
Write concise informative background sentence, that directly helpful to answer the following question. The hackground sentence is the sentence that ends with a suffix. In other words, the answer entity should he followed by the entities used in the question.
Question
Question: Who replaced Tim Sloan as CEO of Wells Fargo? Answer: Charles Scharf'
### Suffix
Charles Scharf
#*# Background Sentence
Tim Sloan was succeeded as CEO of Wells Fargo by Charles Scharf.
#*# Question
[question]
### Suffix
[answer]
### Background Sentence
Table 14: Prompt for Paraphrasing. A 2-shot prompt for paraphrasing, y indicates the answer for the question and denotes the remaining part of sentence, as introduced Section 4.1.
For the following prefix, give me 2 highly diverse paraphrases of the same high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by paraphrased prefix. Do not inclue numbering. Maintain the sentence structure.
Sentence
In infrainguinal bypass surgery, the preferred type of graft for optimal outcomes is an Autologous vein.
# Prefix
In infrainguinal bypass surgery, the preferred type of graft for optimal outcomes is an
# Suffix (PRESERVE AND KEEP LETTER CASE)
Autologous vein.
# Paraphrases (Prefix Suffix)
In infrainguinal bypass procedures. the graft type most recommended for the best results is an Autologous
vein.
During infrainguinal bypass operations, the optimal choice for graft to achieve the best outcomes is Autologous vein.
For the following prefix, give me 2 highly diverse paraphrases of the same in high-quality English language in sentences Wikipedia. Ensure that the suffix followed by paraphrased prefix. Do not include numbering. Maintain the sentence structure. # Sentence
During the embryonic development of the gastrointestinal tract, proper rotation of the gut is necessary for the correct placement of the caecum; abnormality in this process can lead Mixed rotation.
Prefix
During the embryonic development of the gastrointestinal tract, proper rotation of the gut necessary for the correct placement of the caecum; an abnormality in this process can lead to # Suffix (PRESERVE. AND KEEP LETTER CASE) Mixed rotation.
# Paraphrases (Prefix Suffix)
In the formation the gastrointestinal system during embryonic growth, it is essential for the gut to rotate correctly to ensure the caccum is properly positioned; deviations in this mechanism may result in Mixed rotation.
Throughout the development of the gastrointestinal tract the embryo, the accurate rotation of the gut crucial for the appropriate localization the caecum; any irregularities in this rotation can result Mixed rotation.
For the following prefix, give 10 highly diverse paraphrases of the same in high-quality English language in sentences on Wikipedia. Ensure that the suffix followed by paraphrased prefix. Do not include numbering. Maintain the sentence structure.
# Sentence (z,
# Prefix
# Suffix
y
Paraphrases (Prefix Suffix)
20

---

Table 15: Experimental results datasets with synthetic documents from diverse LLMs. We present results from Llama2-7B [48], -isstal--BB-nsstutt--002222 [18], and
[1].
SQuAD-syn
StreamingQA-syr
ArchivalQA-sy
Method
EM
Recall
F1
EM
Recall
F1
EM
Recall
Fi
Llama2-7B [48]
No Adaptation
17.30
25.09
24.27
29.71
36.37
35.59
15.10
23.61
22.36
Fine- Tining
69.10
80.34
78.09
85.30
90.97
90.57
63.60
82.54
79.26
FreeLB [57]
75.10
85.63
83.47
83.46
91.73
90.95
67.00
83.82
80.86
NEFTune [16]
71.10
84.17
81.38
82.54
90.65
89.65
64.80
82.08
79.01
Ours
73.10
87.00
84.13
83.46
92.46
91.20
65.00
88.70
83,55
Mistral-7B-
1aaad---tt-ttuut-.O.... [18]
No Adaptation
4.90
25.33
10.86
14.70
31.78
20.58
6.60
26.66
13.37
Fine-Tuning
49.40
83.60
64.66
65.08
88.43
75.51
41.10
75.88
59.13
FreeLB [57]
58.10
86.20
71.30
72.28
93.44
82.21
47.30
82.10
66.22
NEFTune [16]
45.10
80.06
59.67
67.84
89.01
77.34
37.70
73.25
55.58
Ours
73.20
89.53
83.57
83.46
94.14
91.79
64.80
89.07
82.40
Phi3-mini instrrct [1]
No Adaptation
5.20
22.20
10.77
26.88
15.21
5.20
24.84
Fine-Tuning
38.80
50.32
44.10
70.93
55.19
24.80
$4.57
37,05
FreeLB [57]
41.90
62.43
$2.50
50.69
72.95
60.17
22.50
57.57
35.55
NEFTune [16]
39.30
63.70
50.52
44.87
71.54
55.87
23.60
55.95
36.40
Ours
53.30
67.02
62.88
70.60
77.78
75.39
30.20
64.93
47.04
D
Additional Experiments
D.1
Experiments with Other Language Models
Verifying whether the proposed method can be transferred to other Language Models (LMs) is important. First, validate our LaPael with Llama-2-7B [48], non-instruction-tuned version of the Vicuna-7B we used in experiments. In Table 15, we present the experimental results with Llama-2-7B. The results show that our LaPael is effective even in the L.M that is not instruction-tuned In Table 15, we also present the experimental results with model based on different LLM Mistral-7B [18]. The results indicate that our LaPael is applicable not only to Llama-based models but also to LMs with different bases. Furthermore, in Table 15, we present the experimental results with Phi3-mini-4k -iintii-iinnttutiionn which is a pre-trained LLM with 3.8 billion parameters [1]. The results indicate that our LaPael is highly effective when applied to the Phi3-mini model, which has fewer parameters than other LLMs.
1-nd7------nttuttt222222 which is an instruction-tume
D.2 Experiments with Parameter-Eftice Fine-Tuning
Parameter-efficient fine-tuning is a method that fine-tunes LLMs with minimal updates to their parameters. It of interest that our LaPacl can be effective even with parameter-efficien fine-tuning. LoRA [13] is a well-known method for parameter-efficient fine-tuning, which updates trainable rank decomposition matrices injected into the parameters of LLMs. In Table 16, we present the experimental results with LoRA on Vicuna-7b-vl. where we update only the low-rank matrices of up-proj. gate_proj. and down_proj layers. The results demonstrate that LaPael also effective in LoRA fine-tuning, highlighting its flexible applicability in diverse fine-tuning scenarios.
D.3 Visualization of Latent Features
In Figure we display the latent features from the final layers large language models (LLMs) with and without latent paraphrases, where we reduce the dimension using t-SNE [49]. Crosses ('x') mark the embeddings from LLMs with latent paraphrasers. As illustrated in Figure 7, latent paraphrasers enable the generation of diverse data samples, enhancing the diversity compared to data-level paraphrases.
21

---

Table 16: Experimental results datasets with synthetic documents, where LoRA [13] fine-tuning full parameters on Vicuna-7b-v1.55 [56].
instead of
SQuAD-snn
StreamingQA-syy
ArchivalQA-sy A-snn
Method
FM
Recall
F1
EM
Recall
F1
EM
Recall
F1
No Adaptation
13.10
22.91
21.09
16.39
26.30
23.71
13.50
25.07
22.12
Tuning
62.70
72.80
70.74
73.97
83.75
82.12
53.60
68.23
66.00
FreeLB [57]
62.00
77.21
73.67
81.47
88.76
87.51
62.80
77.77
74.55
NEFTuune [16]
67.40
79.10
76.59
78.71
85.77
84.65
57.60
74.88
71.35
Ours
65.80
82.10
78.80
80.09
89.43
88.03
61.70
79.22
75.24
XX
X
Figure 7: Visualization of Latent Features. We visualize the latent features from the last layers of LLMs using randomly sampled data from ArchivalQA dataset. Each color denotes the different data, circles denote the original sentenees, triangles denote the paraphrases, diamonds denote the questions, and crosses ('x') denote the original sentence with latent paraphrasing.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims and contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: We include experimental details in Section 5.1 and Appendix C for repro- ducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer:
[No]
Justification: We do not open-source the code yet. However, we will open-source it if the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Tablli//guidee/CodeSuttiii io for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. ctips.cc/public//uu guw////iee//////// /Cooeeubeiieeeeeeeeee Swt.Suubmiisinnn... ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include experimental details in Section 5.1 and Appendix C.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Due the limits on computational costs, we only report error bars for experi- ments in Figure 5a. For Table 2, we also report the average performance of three runs for our model to confirm the statistical significance our method against baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide the related information in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrrp.eeeee//ee cttpipps.pppppppee
csGuidelines?
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We discuss broader impact in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the proper source for each dataset and pre-trained language model.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "NPYlXcxcll", "reviewer": "Reviewer_Zuau", "summary": "This paper proposed a latent paraphraser to generate paraphrased data which will be used as augmented data in LLMs' fine-tuning.\nTo tackle the challenges of repetitive external model interventions, the latent paraphraser (LaPael) is trained to add a small perturbation at the latent feature level of the LLM and eliminate the need for users to repeatedly paraphrase using LLMs once latent paraphrasers are trained. Specially, with the paired-data as input, i,e., original and paraphrased sentence, the training objective is to minimise the KL divergence between distributions of original sentence after the paraphraser transformation and the distribution of the paraphrased sentence. The experimental results show that LaPael outperforms existing noise-based paraphraser in generating better augmentation data.", "questions": "1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y.\n2. Any explainations or case study showing the generated paraphase sentences from LaPael?\n3. The computation cost comparsion with noise-based paraphraser.", "limitations": "The trade-off between task performance and computation cost.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well writen and easy-to-follow and the research topic of generating augmentation data is important.\n2. The empirical results clearly show the superiority of the proposed method, LaPael.", "weaknesses": "1. The cost of training the latent paraphraser is not clear, as expensive training will impedes the efficiency of the proposed method.\n2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below.\n\n---\n`[1-1] Regarding the comparative analysis of paraphrased sentences`\n> W2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.\n\n> Q2. Any explanations or case studies showing the generated paraphrase sentences from LaPael?\n\nThank you for your feedback. We would like to clarify our approach as follows. Instead of generating paraphrased sentences, our proposed method perturbs the latent representations within the LLM, as denoted in Lines 45-46. These perturbed latent representations do not directly correspond to paraphrased sentences. Therefore, comparing our method to paraphrased sentences from GPT-3.5 is not applicable. We hope this clarification addresses your concern.\n\n---\n`[1-2] Regarding the cost of training the latent paraphraser`\n> W1. The cost of training the latent paraphraser is not clear, as expensive training will impede the efficiency of the proposed method.\n\n> Q3. The computation cost comparsion with noise-based paraphraser\n\nThank you for your valuable feedback and questions. Our proposed method does introduce an additional cost for training the latent paraphraser once prior to fine-tuning an LLM, compared to noise-based fine-tuning baselines. However, we would like to emphasize that this cost becomes negligible over multiple knowledge injections due to the following factors:\n\n1. **Size and Parameter Cost**: The latent paraphraser has only 3.6% of the parameters of the main LLM (we used a 7B model), keeping the training cost low.\n   - **FLOPs Comparison**: A single forward step for the LLM costs $F$ FLOPs (Floating Point Operations). Training the five latent paraphrasers requires approximately $1.108F$ FLOPs, compared to the $3F$ FLOPs needed for updating the LLM.\n\n2. **Training Dataset Size**: The latent paraphraser can be effectively trained with a small dataset. Training with 50-100 documents is sufficient to surpass baseline performance, reducing the number of training steps and data required, as shown in Figure 5(a).\n\n3. **Transferability**: The latent paraphraser is trained only once and can be reused over multiple knowledge injections without additional retraining, as shown in Table 3. As more knowledge is injected, the amortized training cost approaches zero.\n\nWe have clearly stated the cost of training the latent paraphraser compared to noise-based fine-tuning baselines in the revised manuscript.\n\n---\n\n`[1-3] Regarding the cross-domain transfer`\n> Q1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y.\n\nThank you for pointing this out. We would like to clarify the document on which the latent paraphraser is trained ($D_{train}$) differs from the document on which the LLM is fine-tuned ($D_{K}$). Therefore, it is possible that the latent paraphraser trained on $D_{train}$ of X can show better performance on $D_K$ of Y than on $D_K$ of X. Moreover, the performance differences are marginal, indicating almost no significant difference. This highlights the transferability of our method, demonstrating that the training dataset for the latent paraphraser does not significantly affect its performance. This flexibility is shown in Table 3 and described in Lines 230-238.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5j5qAOtcY", "reviewer": "Reviewer_bjHu", "summary": "This paper presents LaPael, a novel approach to injecting new knowledge into Large Language Models (LLMs). Previous works have shown that fine-tuning the model with data augmented by paraphrasing helps the model learn new knowledge. However, this requires high-quality paraphrased data each time new knowledge is injected, which is costly. The paper proposes a method that introduces a paraphraser layer within the LLMs, which acts as a latent paraphraser that paraphrases the input text in the latent space. The authors train the latent paraphraser using paraphrased data. Once the latent paraphraser is trained, it is used to fine-tune the LLM on the new knowledge. LaPael outperforms standard fine-tuning and paraphrasing approaches, showing improvements on question-answering benchmarks.", "questions": "Please refer to the weaknesses section.", "limitations": "NA", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The paraphraser layer within LLMs introduces a novel method for incorporating new knowledge, reducing the need for high-quality paraphrased data and reducing the significant cost of fine-tuning the model. \n\n2. The paper demonstrates that LaPael outperforms the supervised fine-tuning method on question-answering benchmarks.", "weaknesses": "**The major weakness of the paper lies in the experiment section and the details provided regarding the experiment. Below are my concerns and questions related to this section:**\n\n1. Line 190: \"We use the subset of questions in the test set of each dataset for $D_{QA}$\" -> Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper.\n\n2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection. It would have been more suitable to use datasets with available documents, inject the knowledge from these documents, and then test the model on the QA. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM.\n\n3. There are no statistics available for the created documents, such as their size. Creating small paragraphs from question-and-answer pairs, fine-tuning the model on these paragraphs, and then testing it on the same question-and-answer pairs does not constitute a correct and fair evaluation.\n\n4. Line 212:  Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge?\n\n5. Lines 184-185: It is not clear what the authors meant by \"We sample N noise.\"\n6. Line 182: \"From Equation 10\" should be \"from Equation 15.\"\n\n**The writing of the paper is also very weak. Here are a few examples:**\n\na. The acronym \"LM\" is used in the experiment section without being defined. The paper uses \"LLM\" for Large Language Models and sometimes \"LM.\"\n\nb. The paper uses \"$D_{knowledge}$\" and \"$D_{K}$\" interchangeably. Please ensure consistency.\n\nc. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments on our work. We understand your concerns and are certain that your comments will help improve the quality of our work. Should you have any unresolved concerns, please let us know. We are happy to discuss and do our best to address your concerns.\n\n---\n\n`[2-1] Experiments with available documents with new knowledge.`\n> W2. It would have been more suitable to use datasets with available documents. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM.\n\n**Summary: We show that our method is still effective with raw documents and new knowledge by conducting supplementary experiments on four datasets. The experimental results are presented in Table A of the attachment.**\n\nThank you for your valuable suggestion. We agree that experiments on datasets with available raw documents and new knowledge are important. We acknowledge that the scope of existing experiments, including Table 7, is limited as they do not handle the new knowledge scenario.\n\nTo address this, we conducted new experiments on four datasets, including two with new knowledge, using available raw documents. Specifically, SQuAD-raw and StreamingQA-raw analyze knowledge injection with widely-used document-question datasets, while Films 2024 and Events 2024 test new knowledge injection, as Vicuna-7b-v1.5 is not pre-trained on them (see below for the details of the datasets).\n\nTable A in the attachment on the global comment presents the experimental results, showing our method's effectiveness with available raw documents and new knowledge, consistent with previous synthetic document experiments in the paper. We have included these results in Section 5 of the revised manuscript considering their significance.\n\n> Details of the datasets: \n> * SQuAD-raw: The **entire set** of SQuAD test set with available raw documents.\n> * StreamingQA-raw: The **entire set** of StreamingQA test set with available raw documents.\n> * Films 2024: A synthetic QA dataset based on raw Wikipedia articles under [2024 films](https://en.wikipedia.org/wiki/Category:2024_films). We generated QAs from wiki documents using GPT-4o following Jiang et al. [1].\n> * Events 2024: A synthetic QA dataset based on raw Wikipedia articles under May, June, and July of [US 2024 events in the United States](https://en.wikipedia.org/wiki/Category:2024_events_in_the_United_States_by_month). We generated QAs from wiki documents using GPT-4o following Ovadia et al. [2].\n\n---\n\n`[2-2] The explanations on why we create a synthetic document out of QA pairs to evaluate the knowledge injection`\n> W1. Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper.\n\n> W2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection. \n\nWe would like to clarify that we use datasets with synthetic documents to simplify the evaluation and analysis of knowledge injection, based on widely-used QA datasets. We used a subset of them to meet the budget limits on costs for `gpt-4-turbo` API calls and LLM fine-tuning. We constructed synthetic documents from existing question-and-answer pairs to ensure learning from these documents leads to knowledge injection, without considering variables like reversal curse and distracting contents, as mentioned in Line 193. We have clarified these points in the revision.\n\n---\n\n`[2-3] The statistics for the documents used in experiments.`\n> W3. There are no statistics available for the created documents, such as their size.\n\nThank you for pointing it out. We missed referencing the statistics initially, although they were in Table 8 of the Appendix. We have corrected this in the revision.\n\nWe also supplemented Table 8 with dataset size measurements and presented these in Table B of the attachment in the global comment. Additionally, we measured token counts per document and QA pair for all datasets used, plotting the histogram in Figure A of the attachment. These statistics are now included in Appendix B.1 of the revised manuscript.\n\n---\n\n`[2-4] Clarification on in-context examples used.`\n> W4. Line 212: Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge?\n\nWe use in-context examples to ensure that LLMs generate the answer in the desired format (phrase instead of sentence). We have clarified this point in Section 5.1 of the revision.\n\n---\n\n`[2-5] Regarding the noise sampling in lines 184-185.`\n> W5. Lines 184-185: It is not clear what the authors meant by \"We sample N noise.\"\n\nWe would like to clarify that \"We sample N noise\" means that we randomly sample N different $\\alpha$ values from the Gaussian distribution with mean $\\mu$ as defined in Equation 5. We have clarified this point in the revision.\n\n---\n`[2-6] Writing fixations.`\n> W6. Line 182: \"From Equation 10\" should be \"from Equation 15.\"\n\n> Wa. The acronym \"LM\" is used in the experiment section without being defined. The paper uses \"LLM\" for Large Language Models and sometimes \"LM.\"\n\n> Wb. The paper uses \"$D_{knowledge}$\" and \"$D_K$\" interchangeably. Please ensure consistency.\n\n> Wc. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.\n\nThank you for pointing those out. We have ensured consistent use of the acronym \"LLM\" and the term \"D_K\" throughout the paper in the revision. We have also fixed the no-reference issue by adding the references to the correct subsection of the Appendix.\n\n---\n**References**\n\n[1] Jiang et al., Instruction-tuned Language Models are Better Knowledge Learners\n\n[2] Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OFp0n45svZ", "reviewer": "Reviewer_WC5j", "summary": "This paper focusses on the ability of LLMs to learn new knowledge. Previous work has shown that paraphrasing techniques help learning this new knowledge. However, as the authors argue, explicit paraphrasing has a high computational cost, and the paraphrased data is of limited diversity. To circumvent this, this paper introduces latent paraphrasing, by adding a latent paraphrasing layer in the standard transformer module, in between the LayerNorm and the MLP. This layer is trained with paraphrased data, generated by an LLM. The optimization objective is to minimize the KL-divergence between the output of a “standard” transformer model with the paraphrases as input, and the output of a transformer including the paraphrasing layer with the original sentences as input. \n\nThe authors test their approach, called LaPael, on three datasets: SQuAD, StreamingQA, and ArchivalQA, and compare against multiple fine-tuning baselines. They show that LaPael mostly outperforms the baselines, even in cross-domain transfer experiments.", "questions": "* The authors mention that they use Vicuna as the base model. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM?\n\nSome clarifying questions:\n\n* Line 177: regarding the gold mask for tokens that correspond to the named entity. How is determined what the named entities are in the sentence? Do the authors deploy some sort of NER tagger?\n* Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”?\n* Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K?\n* Line 238: regarding “Combining LaPael and Paraphrases”. I understand the objective of this experiment, and I agree with the authors that it is insightful. I am not entirely sure if I understand the experimental setup here. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning?\n* Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?", "limitations": "The limitations are addressed in the appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper presents a large number of ablation experiments, that show the benefit of each of the components of the proposed approach. These experiments are very insightful, as at first the method might sound a bit complicated. Moreover, the extensive ablation experiments can help steer future research in this direction.\n* The proposed approach is effective, and outperforms the baselines with relatively large margins.", "weaknesses": "* The objective of the paper is to explicitly focus on fine-tuning approaches, instead of approaches that rely on external memory. In that light, it makes sense to only compare against fine-tuning baselines. However, if one is just interested in what method performs best on the task, comparing against external memory based approaches would have been a good addition to the results section.\n* As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection. For example, SQuAD is based on Wikipedia, which the models have probably seen during pretraining. Also see my related question below.\n* I also have a few other questions about the approach, which I have added in the question section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We do appreciate your positive feedback and valuable suggestions, and we hope our response fully addresses your concern. \n\n---\n`[3-1] Regarding new knowledge injection`\n> W2. As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection.\n\n> Q1. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM?\n\nThank you for your valuable comment. We would like to clarify that we measure the degree of knowledge injection by assessing QA accuracy on the QA dataset ($D_{QA}$), which corresponds to the documents ($D_K$) as defined in Section 3. We consider the knowledge that the LLM fails to answer as new knowledge to the LLM, even if such knowledge was seen during pre-training. Previous works [1, 2] also support these observations, as referenced in Lines 25-29.\n\nHowever, **we entirely agree with your opinion** on measuring true new knowledge injection. Therefore, we have provided additional experimental results on **two datasets** containing new knowledge to the LLM, with available raw documents (Please see Table A in the attached PDF file on the global comment). The results show that our method is effective with raw documents on new knowledge, confirming the same trend observed in our previous experiments.\n\nWe have included these results in Section 5 of the revised manuscript considering their significance.\n\n> Details of the datasets: \n> * Films 2024: A synthetic QA dataset based on raw Wikipedia articles under [2024 films](https://en.wikipedia.org/wiki/Category:2024_films). We generated QAs from wiki documents using GPT-4o following Jiang et al. [3].\n> * Events 2024: A synthetic QA dataset based on raw Wikipedia articles under May, June, and July of [US 2024 events in the United States](https://en.wikipedia.org/wiki/Category:2024_events_in_the_United_States_by_month). We generated QAs from wiki documents using GPT-4o following Ovadia et al. [4].\n\n---\n`[3-2] Suggestion: Comparison with external memory-based approaches`\n> W1. Comparing against external memory based approaches would have been a good addition to the results section.\n\nAs per your great suggestion, we have conducted experiments comparing our approach with Retrieval-Augmented Generation (RAG) on datasets with new knowledge, Films 2024 and Events 2024, as used in the previous answer `3-1`. We use [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) model for the embedding.\n\n\n* Films 2024\n\n|      | EM   | Recall | F1  |\n| ---- | ---- | ---    | ---- |\n| Fine-tuning | 13.39 | 30.03 | 28.84 |\n| Ours | 16.29 | 35.04 | 32.56 |\n| RAG | 48.31  | 72.45 | 67.13 |\n\n* Events 2024\n\n|     | EM   | Recall | F1  |\n| ---- | ---- | ---    | ---- |\n| Fine-tuning | 10.98 | 43.76  | 39.62 |\n| Ours | 15.26 | 56.70 | 46.45 |\n| RAG | 27.17 | 64.02 | 55.71 |\n\nExperimental results show that the memory-based approach (RAG) outperforms the fine-tuning approaches, as observed in Ovadia et al. [4]. However, our method can close the gap between the two approaches, indicating a potential pathway to further improve fine-tuning methods. We have added these experimental results to the revised manuscript to emphasize this point.\n\n---\n`[3-3] Answers on Questions`\n\nThank you for your thorough clarification. We have incorporated all clarifications into the revised manuscript. While addressing your questions, we realized that the knowledge injection pipeline with LaPael -- training latent paraphrasers on $D_{train}$ $\\rightarrow$ fine-tuning LLMs on $D_{K}$ -- can be confusing. Therefore, we have added a supporting figure to clearly explain that in Section 4 of the revised manuscript. \n\n> Line 177: Question about NER tagger\n\nYes, we use GPT-3.5-turbo as the NER tagger.\n\n---\n> Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”?\n\n\nWe apologize for the confusion. We would like to clarify that we used **both question and answer** to generate the synthetic document. This is because we want to ensure that the synthetic document contains only relevant information to the question, as illustrated in Table 1.\n\n---\n> Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K?\n\nNo, it refers to the documents $D_K$ defined in Section 3, not the paraphrased version of $D_K$. We used a paraphrased version of $D_K$ in experiments for Figure 4 and Fine-Tuning (+ para.) in Table 2.\n\n---\n> Line 238: regarding “Combining LaPael and Paraphrases”. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning?\n\nThe *number of paraphrases* in Figure 4 refers to the number of paraphrases for each document in the dataset used to fine-tune the LLM ($D_K$) for both \"fine-tuning\" and \"ours.\"\n\n---\n> Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?\n\nIn the real-world data scenario, baselines are also fine-tuned on the raw documents from the SQuAD test set ($D_K$). We would like to clarify that \"ours\" denotes fine-tuning the LLM on the raw documents from the SQuAD test set ($D_K$), with the latent paraphraser trained on synthetic documents from the SQuAD train set ($D_{train}$).\n\n---\n**References**\n\n[1] Kandpal et al., Large Language Models Struggle to Learn Long-Tail Knowledge\n\n[2] Allen-Zhu et al., Physics of Language Models: Part 3.1, Knowledge Storage and Extraction\n\n[3] Jiang et al., Instruction-tuned Language Models are Better Knowledge Learners\n\n[4] Ovadia et al., Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nGmQLGRPEO", "reviewer": "Reviewer_NFcZ", "summary": "The paper proposes a new method named LaPael to enhance knowledge injection for large language models. Different from traditional data-level augmentations or noise-based methods, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. Experimental results on three question-answering datasets show that the proposed method outperforms the baselines.", "questions": "1. In Table 7, why was it not compared with the Fine Tuning (+para) method?\n\n2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?", "limitations": "Please see above Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a latent perturbation for enhancing knowledge injection of LLMs, and the proposed method outperforms the baselines.\n\n2. The paper is written-well, with a direct motivation and a well organized structure.", "weaknesses": "1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly.\n\n2. Why is the output distribution of Transformer assumed to be Gaussian in Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method.\n\n3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below.\n\n`[4-1] Difference between the previous peturbation methods`\n> W1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly.\n\nThank you for giving us a chance to clarify the point. To be clear, we have included the following sentence after Line 109 of the main paper: \"Compared to previous perturbation methods, our method significantly differs in terms of the training objective for a perturbation function and its application to the knowledge injection of LLMs. Specifically, previous methods use bi-level optimization to train the perturbation function, while we use KL divergence to optimize the perturbation function by directly matching the output distribution of the perturbed model with that of the model using paraphrased data.\"\n\n---\n`[4-2] Clarification on Gaussian used in the output distribution`\n> W2. Why is the output distribution of the Transformer assumed to be Gaussian in the Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method.\n\nWe appreciate your valuable question. While it is possible to estimate with a more complex distribution, we chose to use a Gaussian distribution due to complexity issues. We found that even with this simplified modeling, our approach performs well.\n\n---\n`[4-3] Computational cost comparison against paraphrasing method`\n> W3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.\n\nThank you for highlighting this important point. Our method requires less computational cost than the paraphrasing method, as our method requires adding a tiny module on the top of the LLM. To compare computational costs, we calculate each method in FLOPs (Floating Point Operations), with fine-tuning the LLM costing $F_{LLM}$.\n\n1. **Fine-tuning with paraphrased text**: This requires generating paraphrases, costing $F_{LLM} + F_{Paraphraser}$ FLOPs, where $F_{Paraphraser}$ is the cost of a forward pass on the paraphraser model.\n2. **Fine-tuning with latent paraphraser**: This requires a forward pass on the latent paraphraser, costing $F_{LLM} + F_{LaPael}$ FLOPs, where $F_{LaPael}$ is the cost of a forward pass on the latent paraphrasers.\n\nA single latent paraphraser has $3d^2 + 4d$ parameters. For a 7B LLM, this is about 50M parameters. Using 5 latent paraphrasers totals 250M parameters. Thus, using a latent paraphraser is **equivalent to a 250M parameter paraphraser**, which is impractical for generating high-quality paraphrases. Given that `gpt-3.5-turbo` (or a 7B LLM) is used as the paraphraser in experiments, our latent paraphraser is significantly cheaper in computational cost.\n\n---\n`[4-4] Additional experiments for Table 7`\n> Q1. In Table 7, why was it not compared with the Fine Tuning (+para) method?\n\nThank you for pointing it out. We missed adding FreeLB and Fine-Tuning (+para) in Table 7. We conducted the experiments you mentioned and the results are as follows:\n\n* Table 7 (with FreeLB and Fine-Tuning (+para))\n\n|             | EM    | Recall | F1    |\n| ----------- | ----- | ------ | ----- |\n| No Adapt    | 13.10 | 22.91  | 21.90 |\n| Fine-Tuning | 24.10 | 38.49  | 34.78 |\n| NEFTune     | 22.30 | 39.07  | 33.90 |\n| FreeLB      | 22.30 | 39.95  | 34.31 |\n| Ours        | 27.20 | 46.66  | 39.66 |\n| Fine-Tuning (+para)| 31.90 | 48.13 | 43.08 |\n\nExperimental result shows that using data-level paraphrases is effective in the real-world data setting. \nAs we discussed in Section A Limitations, our latent paraphraser cannot resolve the reversal curse issue in the real-world data, while data-level paraphraser can. We have included the results and related discussions in Table 7 of the revision.\n\n---\n`[4-5] A negative impact on the knowledge retention`\n> Q2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?\n\nWe appreciate your valuable and insightful suggestion. To evaluate the potential negative impact on the knowledge already mastered by the LLM, we used the EntityQuestions dataset [1], which asks simple questions about entities. Among these questions, we only used the simple `place-of-birth` questions from the frequent entities (e.g., Where was Leonardo da Vinci born?), with 988 questions in total.\n\nWe fine-tuned the Vicuna-7b-v1.5 model on a synthetic SQuAD document set ($D_K$) using each method and measured the QA performance on EntityQuestions as follows.\n\n* QA Performance on EntityQuestions after fine-tuning LLM on $D_K$ of SQuAD\n\n|             | EM    | Recall | F1    |\n| ----------- | ----- | ------ | ----- |\n| No Adapt    | 59.00 | 64.38  | 63.46 |\n| Fine-Tuning | 52.23 | 55.63  | 55.41 |\n| NEFTune     | 48.28 | 50.33 | 50.54 |\n| FreeLB      | 46.56 | 48.37 | 48.57 |\n| Ours        | 39.88 | 41.97  | 42.12 |\n| Fine-Tuning (+para)| 33.50 | 35.18 | 35.28 |\n\nThe experimental results show that all fine-tuning approaches have a negative impact on the knowledge already mastered by the LLM. We also observe that the better the knowledge injection performance, the greater the negative impact on knowledge retention. As our work focuses on improving knowledge injection, knowledge retention is beyond its current scope. However, we believe discussing this point as a limitation is important as it can encourage future work in this direction. We have included this point and the experimental results in the limitations section of our revised manuscript.\n\n---\n**Reference**\n\n[1] Sciavolino et al., Simple Entity-Centric Questions Challenge Dense Retrievers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
(R1=R-Zuau, R2=R-bjHu, R3=R-WC5j, R4=R-NFcZ)

We sincerely thank the reviewers for their thoughtful and constructive feedback. We appreciate the acknowledgment that the paper is well-written and organized (R1, R4), the superiority of the proposed method (R1, R2, R3, R4), the insightfulness of the experiments (R3), and the novelty of the method (R2).

Regarding the concerns and questions raised, we believe that we have adequately addressed each one and provided detailed responses in line with each review. We have also revised the manuscript according to your valuable feedback and suggestions.

In the attachment, we include the following:
- Table A: Experimental results on four datasets with raw documents, referenced in the responses to R2 and R3.
- Figure A: The distributions of token counts in documents, questions, and answers for each dataset used in our experiments, referenced in the response to R2.
- Table B: The size of datasets used in our experiments, referenced in the response to R2.

Meta-review: The paper noticed that during knowledge injection in LLMs, data augmentation by paraphrasing is very important. And inspired by this, they proposed Latent Paraphrasing (LaPael) as a novel approach by introducing perturbations at the latent feature level, rather than relying solely on data-level paraphrasing. When injecting new knowledge into an LLM, LaPael first insert a learnable noisy mask before the MLP layers in Transformers and train the mask distribution on an existing paraphrase dataset. Then during the fine-tuning process, this mask distribution induces variability similar to paraphrased data but operates directly within the model's latent space. The paper presents strong experimental results for LaPael, supported by improvements over data augmentation in performance across various benchmarks, including question-answering tasks.

The reviewers have pointed out some concerns that should be addressed. A major concern is the lack of detailed analysis regarding the computational cost of training the latent paraphraser, which could potentially limit the practical applicability of the method. Furthermore, the paper would benefit from additional qualitative analysis to illustrate how the perturbations at the latent level differ from traditional paraphrasing at the data level. There are also questions regarding the generalizability of LaPael across different task, but not just question-answering.

However, the reviewers generally agree that LaPael offers a novel approach by targeting the latent feature space, which is a promising direction that could reduce dependency on external paraphrasing models and improve the knowledge injection process. The paper also provides a potentially new viewpoint of the training dynamics of LLMs, where at the end of training, the model needs some proper weight perturbation to learn new knowledge. Overall the Pros trumps the Cons, which lead to a recommendation to acceptance, but I still recommend authors to put significant efforts to resolve the concerns.
Decision: Accept (poster)