Title: Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: large language models

knowledge injection

data augmentation
Abstract: As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. 
To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers.
This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. 
Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. 
Additionally, combining LaPael with data-level paraphrasing further enhances performance.

Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models

Abstract

As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers. This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. Additionally, combining LaPael with data-level paraphrasing further enhances performance.

1 Introduction

Pre-trained Large Language Models (LLMs) encode extensive factual information from their training data, enabling them to answer factoid questions such as 'Who is the director of Dune: Part Two?' [4, 32]. However, knowledge in LLMs is static, which can lead to outdated information as real-world knowledge evolves. Additionally, LLMs often lack specificity for specialized or private domains. To address this, it is common practice to fine-tune LLMs with updated or domain-specific documents, keeping the model's knowledge up-to-date and enhancing expertise in particular domains [14, 17, 19].

However, does fine-tuning LLMs on a single document allow them to fully internalize its knowledge? Even in pre-training, Kandpal et al. [20] found that LLMs cannot perfectly learn all the information in the training data, particularly long-tail knowledge that appears rarely or only once. Existing work [33] has shown that this issue persists with fine-tuning and suggested that data augmentation, such as paraphrasing, is a simple yet effective way to enhance knowledge injection. As shown in Figure 1, fine-tuning with paraphrases enhances knowledge injection, as evidenced by improved Question-Answering (QA) task performance.

StreamingQA

SQuAD

ArchivalQA

75

80

85

90

95

QA Task Performance

Fine-Tuning (FT)

FT + Paraphrases

Ours

While data-augmented approach via paraphrasing is effective for knowledge learning, it has two main limitations: (1) High computational cost: Generating high-quality paraphrases requires significant computational resources. As shown in Figure 2, paraphrasing models such as LLMs [5, 7, 11, 58] need to repeatedly generate paraphrases for each document with the new incoming knowledge. This leads to higher costs as the number of documents being learned continually increases; and (2) Limited

Q:

What is the worldwide total gross of 'Dune:

Part Two' as of May 14, 2024?

A:

$872 million

User

As of May 14, 2024, Dune: Part Two has grossed

$282.1 million in the United States and Canada and

$428.5 million in other territories, for a worldwide

total of $710.6 million.

A:

$710.6 million

As of May 14, 2024, Dune: Part Two has grossed $282.1

million in the United States and Canada and $428.5 million

in other territories, for a worldwide total of $710.6 million.

As of May 14, 2024, Dune: Part Two's revenue stands at

$282.1 million in the United States and Canada and $428.5

million globally, with a combined total of $710.6 million.

Paraphraser

FT

Data-augmented approach

As of May 14, 2024, Dune: Part Two has grossed $282.1

million in the United States and Canada and $428.5 million

in other territories, for a worldwide total of $710.6 million.

Our approach

: Latent Paraphrasing

Task: Knowledge Injection

Fine-tuned

LLM

Fine-tuned

LLM

Fine-tuning

(FT)

FT

Fine-tuned LLM

Remove

latent

paraphrasers

No paraphrase

Base

LLM

Base LLM

Latent

Paraphrasers

Fine-tuned LLM

Latent

Paraphrasers

Base

LLM

1

diversity in augmented data: Although LLMs can produce varying high-quality paraphrases by sampling from the generative distribution, the diversity of the generated text is limited, resulting in a narrow range of augmented samples at the discrete data level. One way to overcome these issues is to introduce noise into the token embedding. However, existing works [16, 57] do not consider the text semantics when they perturb the latent features of LLMs with randomly generated noise.

To address these issues, we take a distinct approach using an input-dependent noise generator named 'latent paraphraser' learned from the paraphrases. Specifically, this function perturbs early layers to augment LLMs at the latent level while preserving the meaning of the text. To optimize the latent paraphraser, we start by generating paraphrases of the documents. Then, we train the latent paraphrasers to ensure that the latent distribution of the LLMs with the original sentence is close to the latent distribution with the paraphrased sentences. Once training is done, we can transfer the latent paraphrasers to the documents from any domain that contains new knowledge. We refer to our method as La tent Pa raphrasing of Language Mod el s ( LaPael ), as it learns the paraphrasing of text data at the latent level.

We validate our approach on diverse question-answering benchmark datasets [38, 27, 51] designed to evaluate knowledge injection. These benchmarks involve fine-tuning LLMs on documents that contain the knowledge required to answer the questions in the datasets. Our results show that LaPael significantly improves knowledge injection performance compared to standard fine-tuning. Moreover, LaPael outperforms fine-tuning with paraphrases, demonstrating that LaPael alone is sufficient for data augmentation in knowledge injection scenarios, as illustrated in Figure 2. As shown in Figure 1, we further find that using LaPael in combination with paraphrases further enhances performance, providing complementary benefits to data-level augmentations. Finally, LaPael surpasses existing noise baselines [16, 57], highlighting the importance of learning noise for effective augmentations.

Our contributions are as follows:

We introduce LaPael , a new method that applies learned perturbations to the layers of LLMs to enhance knowledge injection, addressing the limitations of data augmentations and noise baselines.

We validate LaPael using diverse question-answering benchmark datasets, demonstrating a significant improvement in knowledge injection performance compared to standard fine-tuning.

Our results show that LaPael not only outperforms fine-tuning with paraphrases but also complements it, providing additional benefits when used together, surpassing the performance of existing latent noise-based methods.

2 Related Work

Knowledge of Large Language Models Large Language Models (LLMs) store vast amounts of factual knowledge in their pre-trained parameters [36, 44]. The straightforward way to extract the

knowledge of LLMs is to ask the question that requires factual knowledge [43, 58]. Through asking questions, Kandpal et al. [20] have found that LLMs cannot perfectly memorize the entire knowledge in the pre-training corpora, especially for knowledge that appears rarely or only once. To make LLMs answer the question requires under-represented or new knowledge, previous works have clustered into two different solutions. The first one is retrieval-augmented methods [26, 39, 42] that retrieve knowledge from an external knowledge base and input the retrieved knowledge alongside the question into LLMs. The second one is fine-tuning [12, 17] where the parameters of pre-trained LLMs are continually updated by fine-tuned on the document containing knowledge in an unsupervised way as in pre-training [37]. In our work, we focus on improving the fine-tuning-based solution, as storing new knowledge in the parameters of LLMs is efficient since we can reduce the length of the input prompt and do not need any extra module or memory in the deployment time [6].

Knowledge Injection in LLMs In this work, knowledge injection in LLMs denotes fine-tuning LLMs on the set of documents to inject new or under-represented knowledge into LLMs [33, 17], different from another task of injecting symbolic knowledge (e.g., knowledge graph) into LLMs [55, 54]. Among previous works, CaMeLS [14] has introduced a meta-learning method for learnable loss scaling function that improves knowledge injection. As a concurrent work, MAC [45] has proposed using the memory of amortized context is highly effective in a knowledge injection. However, both methods have drawbacks like high computational costs for bi-level optimization or the need for additional modules and memory. Recent works [33, 58] have shown that data augmentation which paraphrases the knowledge-containing sentences helps language models memorize knowledge in a more extractable format (e.g., asking questions) after knowledge injection. Furthermore, Jiang et al. [19] has shown that the instruction-tuned model is better at learning new knowledge. Compared to previous works, we focus on developing an alternative method to data augmentation that perturbs the latent representation of LLMs for better knowledge injection.

Data Augmentation and Latent Perturbation The usefulness of data augmentations for text data was empirically observed in the literature. For instance, EDA [52] has introduced simple data augmentation method which randomly deletes, swaps, replaces, and inserts the words. Other previous works [22, 5, 30] have utilized the trained LMs to augment the text data. Recently, Maini et al. [29] has shown that adding data rephrased by LLMs into the pre-training corpus improves the performance of LM pre-training. However, those methods require additional costs in the knowledge injection as it utilize the LLMs to rephrase the text. In contrast, the latent perturbations offer an orthogonal approach to improve the robustness of neural networks, complementing data augmentation. This technique has been employed in meta-learning and out-of-distribution generalization [24, 25, 40]. For instance, NEFTune [16] demonstrated that adding noise, randomly sampled from a uniform distribution, to token embedding layers improves instruction tuning performance. Expanding on the concept of latent perturbations, our work introduces a novel approach that internalizes the effects of text paraphrasing by identifying optimal latent perturbations through training a small neural network within the LLMs.

3 Problem Formulation

In this work, we follow the knowledge injection setting outlined by Ovadia et al. [33]. We are given three resources: (1) documents D K containing knowledge that we are interested to inject; (2) question & answering dataset D QA = { ( q ( i ) , a ( i ) ) } n i =1 for verifying injected knowledge from D K ; and (3) a pre-trained Large Language Models (LLMs) p θ ( · ) parameterized by θ . Our objective is to find a transformation F that could enhance the knowledge about D QA :

where the score function S is defined as:

and I ( · ) and f ( · ) denote the indicator function and a decoding function that samples a sequence of tokens from p θ , respectively.

Transformer-based LLM

Transformer layer 1

Transformer layer L

Token Embedding Layer

LM Head

Self-Attention

LayerNorm

MLP

LayerNorm

…

…

(a) Latent Paraphraser Architecture

(b) Latent Paraphraser Training

Original Sentence

Paraphrases

minimize

KL Divergence

Latent Paraphraser

Linear

Sampling

MLP +

Softplus

Latent Paraphrasers

Layers with

Transformer Layers

The capital city of the US is

In the case of US, the capital is

Transformer Layers

Paraphraser

backward

Linear

Sigmoid

~

Last token features

Last token features

estimate

estimate

In general, a transformation F is a fine-tuning LLMs on documents in D K by optimizing θ to minimize the negative log-likelihood of each token in each document as follows [33]:

where | s | denotes the length of token sequence s .

4 Proposed Method

We propose Latent Paraphrasing of Language Models (LaPael), a framework that perturbs the latent feature of LLMs, to achieve the equivalent effect of data augmentation at the latent level. Knowledge injection using LaP consists of the following four processes: paraphrasing the set of documents to make the paraphrased data (Section 4.1), training the latent paraphrasers with paraphrased data (Section 4.2), fine-tuning LLMs with the trained latent paraphrasers on D K and evaluate the injected knowledge of LLMs on D QA (Section 4.3).

4.1 Data Augmentation: Paraphrasing

To train the latent paraphrasers, we need a distinct set of training data D train = { s ( i ) } N i =1 which consists of documents having different knowledge with D K . As a preliminary, we formulate the paraphrasing of the text in terms of the knowledge equivalence , which is a narrower concept than semantic equivalence [23] where two different sentences can contain the same knowledge. We consider that each sentence s in D train can be decomposed into words for the object (entity or attribute) of the knowledge ( y ) and others ( x ) where both are the sequence of tokens. For instance, given the sentence 'The capital of the United States is Washington D.C. ' ,

represent the knowledge (United States, capital, Washington D.C.). Then, we paraphrase a sentence s = ( x , y ) into a paraphrased sentence 1 . For the above sentence, a paraphrased sentence can be

with the same y , which is knowledge equivalent to ( x , y ) . For each knowledge K , we assume that there is a set of the knowledge equivalent sentences S ( K ) where ( x , y ) ∈ S ( K ) . We generate K paraphrased sentence via a LLM: ( x 1 , y ) , . . . , ( x K , y ) ∼ p LLM ( x ′ | prompt , x , y ) . Then, we have the set of paraphrased data {{ ( x ( i ) k , y ( i ) ) } K k =1 } N i =1 of D train . We define p ( x ′ | x ) := p LLM ( x ′ | prompt , x , y ) which denotes the probability distribution of paraphrases given the original sentence.

4.2 Introducing Latent Paraphraser

Latent Paraphraser We introduce a latent paraphraser within a transformer layer [50], which augments a latent feature and is expected to paraphrase the given input text within the latent space. As illustrated in Figure 3(a), within the transformer architecture, we insert this new layer just before the Multi-layer Perceptron (MLP), using the output from the second LayerNorm as its input.

Let h ∈ R d denote the latent feature after the second LayerNorm. The latent paraphraser, denoted by g ϕ : R d → R d and parameterized by ϕ , augments the latent feature as follows:

where ◦ is the element-wise multiplication. The function g ϕ ( h ) is given by:

with z ∈ R d and m ∈ [0 , 1] representing a noise vector and a learnable mask, respectively.

The noise vector z is generated by

where MLP z is a 2-layers MLP. We use the reparameterization trick [21] to enable the backpropagation through the sampling from the Gaussian distribution: α = µ + ϵ , where ϵ ∼ N (0 , I ) .

To modulate the scale of perturbation for individual tokens, we employ a learnable mask. It is important as too much noise on key tokens (e.g., United States) might hurt the semantics of the sequence. For learnable binary mask, we use concrete distribution to approximate the sampling discrete random variable from a Bernoulli distribution using continuous relaxation [8] as follows:

where u ∼ Unif (0 , 1) , τ is temperature, and m is mask value in scalar.

Training Then, how do we train the latent paraphrasers to approximate optimal perturbation functions for estimating the distribution of the paraphrased text? We employ the dataset with paraphrases {{ ( x ( i ) k , y ( i ) ) } K k =1 } N i =1 generated in Section 4.1. Our objective is to match two distributions for each transformer layer:

the distribution of transformer layer output feature for the last token h out without the latent paraphraser given the data perturbation distribution p ( x ′ | x ) from Section 4.1:

the distribution of output feature for the last token h out with the latent paraphraser given x , p θ,ϕ ( h out | x ) . As a latent paraphraser outputs stochastic noise, we can formulate the probabilistic distribution p θ,ϕ ( h out | x ) as follows:

where p θ,ϕ ( z | x ) is the distribution for noise from the latent paraphraser in Equation (6).

We make the simplistic parametric assumption that both distributions are Gaussian:

To train latent paraphrasers, we minimize the symmetric Kullback-Leibler (KL) divergence between two estimated Gaussian distributions of each layer as follows:

We employ a Monte Carlo sampling approach to estimate the parameters of Gaussian distributions. We generate N samples h (1) latent , . . . , h ( N ) latent from the distribution p θ,ϕ ( h out | x ) . Then, we estimate the empirical mean and standard deviation from the samples as follows:

and we use K paraphrases x 1 , . . . , x K to obtain K samples h (1) data , . . . , h ( K ) data from the distribution p θ ( h out | x ) . Then we estimate the parameters in the same way:

We further use the auxiliary loss for mask training, with the sequence length of T as follows:

where ˜ m T is defined in Equation (7), r ∈ [0 , 1] is the mask ratio that controls the number of masks and ¯ m t is the gold mask where ¯ m t = 0 for tokens that correspond to the named entity.

To sum up, we optimize the latent paraphraser parameter ϕ by minimizing the following loss:

See Figure 3(b) for an illustration of the training process for the latent paraphraser.

4.3 Fine-tuning the LLM with the Trained Latent Paraphrasers

We fine-tune the LLM on documents containing knowledge to be injected ( D K ) as in Equation (3). We use the trained latent paraphraser parameterized by ϕ ∗ during LLM fine-tuning as follows:

where we sample N noise z ( j ) by sampling multiple α from Gaussian distribution as defined in Equation (6). Then, we evaluate the knowledge injected in LLMs by measuring S ( θ ∗ , D QA ) as defined in Equation (2).

5 Experiments

In experiments, we validate the effectiveness of the proposed method, LaPael, in injecting new or under-represented knowledge into Large Language Models (LLMs).

5.1 Experimental Setting

5.1.1 Datasets

To follow the experimental setup in Section 3, we need (1) documents containing knowledge D K and (2) associated QA datasets D QA . We mainly use the test split of three QA datasets: SQuAD [38], StreamingQA [27], and ArchivalQA [51] for the source of D K and D QA in our main experiments. These datasets, previously used in Hu et al. [14], consist of documents paired with their corresponding QAs, making them well-suited to our experimental setup. While the questions in these datasets are of decent quality, a significant limitation lies in the documents provided. These documents are likely to have been seen by LLMs during pre-training, making it difficult to accurately assess the performance of methods on injecting new knowledge.

To mitigate this issue, we incorporate two datasets with synthetic QAs - Films 2024 and Events 2024. These are QA datasets generated from raw Wikipedia articles under the 2024 films category and from US events in May, June, and July 2024, in the 2024 events in the United States category. We generated question-answer pairs from these documents using GPT-4o following methods from previous works [19, 33]. Since the documents used to generate these datasets were not seen by the LLMs during pre-training, we can better evaluate the effectiveness of each method for knowledge injection especially on new knowledge.

Datasets with Synthetic Documents The raw documents from datasets are unsuitable for precisely measuring the knowledge injection performance. Specifically, fine-tuning LLMs on a document does not always ensure that LLMs can answer the associated questions, due to the reversal curse [3]. Moreover, documents often contain irrelevant knowledge that may hinder the accurate assessment of knowledge injection [14].

To address these issues, we conduct evaluations under the setting of synthetic documents. For generating synthetic documents, we construct D K by rephrasing each question and answer in D QA using GPT-4-turbo [32], ensuring that fine-tuning on these synthetic documents guarantee that LLMs become answerable to the associated questions. Examples of questions, synthetic, and raw documents are shown in Table 1. To make a difference, we denote the dataset under the synthetic document setting with the suffix '-syn' and the raw document setting with the suffix '-raw'.

Datasets for Training Latent Paraphrasers For training our latent paraphrasers, the set of training data D train is required in addition to D K . Therefore, we use GPT-3.5-turbo [31] to generate the set of synthetic sentences from the subset of a training split of each QA dataset, where each sentence must be with the answer to questions, following the sentence format in Section 4.1.

5.1.2 Experimental Details

Baselines We compare our LaPael against several baselines. All models are fine-tuned on the documents in D K unless explicitly stated otherwise. (1) No Injection. We use the pre-trained LLM without any fine-tuning. (2) Fine-Tuning. We fine-tune the LLM on D K . (3) Fine-Tuning (seq) . We first fine-tune the LLM on the paraphrased documents of D train . Then, we fine-tune the LLM on D K . (4) Fine-Tuning (+ para) . We fine-tune LLM on the original and paraphrased documents of D K . (5) FreeLB [57]. We add trained adversarial noise to the token embedding while fine-tuning. (6) NEFTune [16]. We add random uniform noise to the token embedding while fine-tuning. (7) LaPael (ours). We train the latent paraphrasers on D train and then fine-tune the model on D K .

Training & Inference We mainly use Vicuna-7b-v1.5 [56] for fine-tuning, which is the instructiontuned version of Llama-2-7b [48] for our experiments. We fine-tune LLMs for 12 epochs with a learning rate of 0 . 00005 and step learning rate scheduler where we decay a learning rate by 0.85 by every 4 epochs. For inference, we use in-context learning with 5 examples by prompting the 5 examples in the prompt [4]. To measure QA accuracy, we use Exact Match (EM), Recall (Rec.), and F1 score. More details on the experimental setting are provided in the Appendix C.

5.2 Experimental Results

Experiments with Synthetic Documents In Table 2, we present the experimental results for the synthetic documents setting. Fine-tuning does improve the QA performance of LLMs, but it does not lead to near-perfect scores even though the synthetic document contains the necessary knowledge for answering the questions, as shown in Table 1.

Our experiments show that paraphrasing documents for fine-tuning consistently improves QA performance across all three benchmarks. Notably, LaPael demonstrates performance comparable to fine-tuning with paraphrases on StreamingQA and even outperforms it on two other benchmarks. These findings suggest that the latent paraphrasers learn an effective noise distribution that aids knowledge injection without additional data augmentation.

We also compared LaPael with two other noise-based methods, FreeLB [57] and NEFTune [16], to validate that the latent-level noise generated by latent paraphrasers is more effective. As shown in Table 2, LaPael outperforms these baselines, confirming the strength of our approach.

Experiments with Raw Documents While our method has proven effective for knowledge injection with synthetic documents, it is important to evaluate its performance on raw documents, which represent a more realistic data format. To demonstrate the applicability of our method to real-world data, we conducted experiments in which we fine-tuned LLMs on raw documents for each dataset, using latent paraphrasers trained on D train from SQuAD-syn.

As shown in Table 3, our method outperforms both fine-tuning and noise-based baselines in the context of knowledge injection with raw documents. Considering that the latent paraphrasers were trained on synthetic sentences from D train , these results demonstrate their effectiveness on documents with a different format than those used in training.

Cross-domain Transfer Once trained, the latent paraphrasers can be applied to fine-tune LLMs on documents from any domain. To demonstrate this, we conducted cross-domain transfer experiments. Specifically, we trained latent paraphrasers on D train from a source domain (e.g., SQuAD) and finetuned LLMs with the trained latent paraphrasers on D K from a target domain (e.g., StreamingQA).

2

4

6

8

10

20

The Number of Paraphrases

79

80

81

82

83

84

85

86

87

F1

Fine-tuning

Ours

(a) SQuAD-syn

2

4

6

8

10

20

The Number of Paraphrases

89

90

91

92

93

94

F1

Fine-tuning

Ours

(b) StreamingQA-syn

2

4

6

8

10

20

The Number of Paraphrases

78

80

82

84

F1

Fine-tuning

Ours

(c) ArchivalQA-syn

510

25

50

75

100

78

80

82

84

86

QA Accuracy (F1)

NEFTune

FreeLB

Ours (mean)

(a) Varying ratio of training data

1

3

5

7

9

11 13 15 17 19 21 23 25 27

Start Position of Latent Paraphrasers

50

60

70

80

90

QA Accuracy (F1)

# layers=5

# layers=7

(b) Varying position of latent paraphraser

As shown in Table 4, our method successfully transfers across domains, with the latent paraphrasers enhancing the performance of the knowledge injection on NovelQA and MedMCQA-two domains distinct from the source (see Appendix C.1 for details on these datasets). Even though both domains contain specialized entities, our method consistently outperforms standard fine-tuning and other noise-based baselines.

Combining LaPael and Paraphrases Paraphrasing documents in D K has been shown to improve knowledge injection performance, as seen in Table 2. While LaPael significantly improves performance without requiring paraphrases, it is valuable to consider the effect of combining paraphrases with the latent perturbations from LaPael. As illustrated in Figure 4, LaPael consistently outperforms standard fine-tuning, showing that LaPael provides advantages over data-level augmentations.

5.3 Ablation Studies

Effects of the Size of D train LaPael needs additional data D train for training latent paraphrasers. Although only a small amount of data is required, it might be unclear how much is needed to make the latent paraphrasers learn the useful noise distribution. As shown in Figure 5a, LaPael works well even with 50 sentences for D train , while increasing the size of D train ensures a steady performance improvement for LaPael.

Effects of the Position of Latent Paraphrasers Our latent paraphrasers can be inserted into any layer of the LLMs. The possible question is which position and how many layers are optimal for latent paraphrasers to effectively learn noise for knowledge injection. To answer this, we analyzed the position and number of latent paraphrasers.

In Figure 5b, we show the QA accuracy results, varying the start position and number of latent paraphrasers. The first layer is the closest layer to the input layer, and "start position 1" with "# layers = 3" means we insert the latent paraphrasers into the first, second, and third layers of the LLM. Results show that inserting three latent paraphrasers into the early layers of the LLM is effective. This is consistent with findings in previous works [16, 57, 25] where using noisy token embeddings (the lowest layer) enhanced the generalization in LLMs. Furthermore, in Table 5, we empirically show that positioning the latent paraphraser before the MLP layer within each transformer layer is the most effective choice over other positions.

Ablation Studies on Modules LaPael has many design choices concerning the latent paraphraser architecture, noise type, and training. We conducted extensive ablation studies to empirically verify each design choice and provide guidance for future work. In summary, as shown in Table 6, all design choices are important for building the most effective latent paraphraser. Specifically, we use a trainable mask m in Equation (7) to regulate the perturbation depending on each token, which is crucial, as the performance on StreamingQA drops significantly if we remove it from the latent paraphraser. Furthermore, using only the sigmoid function in Equation (7) instead of the concrete distribution also leads to much lower performance, as the mask is not properly trained. Regarding noise training, using deterministic noise instead of stochastic noise by removing the noise drawn from a Gaussian distribution in Equation (5) also decreases performance. Additionally, replacing the KL loss with Mean Squared Error loss between two means ˆ µ latent in Equation (13) and ˆ µ data in Equation (14) leads to a decrease in performance, confirming the importance of stochastic noise trained with KL loss.

Ablation Studies on Noise Distribution Should we train the latent paraphrasers to be effective, or can adding random noise in the early layers also be effective? Which is more important: the learnable mask or the learnable noise? To answer these questions, we conducted ablation studies on the choice of noise distribution. In Table 7, Learnable Add. denotes the model with the additive noise h + g ϕ ( h ) instead of Equation (4) without softplus from Equation (6). Gaussian is the use of zero-mean Gaussian noise N ( 0 , I ) in Equation (6) without using MLP z . Uniform is the use of noise drawn from the uniform distribution defined in NEFTune [16] instead of z in Equation (6).

As shown in Table 7, the learnable multiplicative noise described in Section 4.2 is the best design for noise distribution used in the latent paraphraser. To analyze the effect of the learnable mask, we also added the learnable mask to the Gaussian and Uniform noise settings and optimized only W m and b m in Equation (7) with loss in Equation (15). Interestingly, the learnable mask is not effective for the fixed noise distribution, which contrasts with the results for learnable noise in Table 6. We conjecture that using the learnable mask is important for input-dependent learnable noise, as it can allocate different noise scales to different tokens, while this is not the case for static noise distribution.

6 Conclusion

We have introduced LaPael, a method for enhancing knowledge injection in Large Language Models (LLMs) by applying learned perturbations to their layers. Unlike traditional data-level augmentations or noise-based approaches, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. LaPael addresses key limitations of existing methods by reducing computational costs and increasing the diversity of augmented data. Our extensive validation across diverse benchmark datasets demonstrates the superiority of our method in knowledge injection, as it significantly outperforms both standard fine-tuning and other noise-based baselines. Moreover, combining LaPael with paraphrases yields complementary benefits, further enhancing performance. We believe that LaPael, being simple yet effective, has the potential for significant practical impact and will encourage further research on applying perturbation within the latent space of LLMs.

Discussions & Limitations In our work, the following points can be discussed further: (1) Cost Analysis-While LaPael is effective, it incurs additional costs due to the need for training latent paraphrasers and fine-tuning LLMs with them. (2) Knowledge Retention-Although LaPael improves knowledge injection, there may be trade-offs in terms of retaining the original knowledge that the LLM has memorized. (3) Comparison to Retrieval-Augmented Generation (RAG)-While our method improves knowledge injection, it is still less effective than RAG in terms of performance. We provide a detailed discussion of these points, along with other limitations, in the Appendix A.

References

Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219 , 2024. doi: 10.48550/ARXIV.2404.14219. URL https://doi.org/10.48550/arXiv.2404.14219 .

Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David A. Sontag. Large language models are few-shot clinical information extractors. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 1998-2022. Association for Computational Linguistics, 2022. URL https: //doi.org/10.18653/v1/2022.emnlp-main.130 .

Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on 'a is b' fail to learn 'b is a'. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=GPKTIktA0k .

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html .

Hengyi Cai, Hongshen Chen, Yonghao Song, Cheng Zhang, Xiaofang Zhao, and Dawei Yin. Data manipulation: Towards effective instance learning for neural dialogue generation via learning to augment and reweight. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 6334-6343. Association for Computational Linguistics, 2020. URL https://doi.org/10.18653/v1/ 2020.acl-main.564 .

Maria Angels de Luis Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. RAG vs fine-tuning: Pipelines, tradeoffs, and a case study on

agriculture. arXiv preprint arXiv:2401.08406 , abs/2401.08406, 2024. doi: 10.48550/ARXIV. 2401.08406. URL https://doi.org/10.48550/arXiv.2401.08406 .

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759 , 2023. URL https://doi.org/10. 48550/arXiv.2305.07759 .

Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 3581-3590, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 84ddfb34126fc3a48ee38d7044e87276-Abstract.html .

Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904 , abs/2405.05904, 2024. URL https://doi.org/10.48550/arXiv.2405. 05904 .

Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.13799 , 2024. URL https://doi.org/ 10.48550/arXiv.2403.13799 .

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2306.11644 , 2023. doi: 10.48550/ARXIV.2306.11644. URL https://doi.org/10. 48550/arXiv.2306.11644 .

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 8342-8360. Association for Computational Linguistics, 2020. URL https://doi.org/10.18653/v1/2020.acl-main.740 .

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=nZeVKeeFYf9 .

Nathan Hu, Eric Mitchell, Christopher D. Manning, and Chelsea Finn. Meta-learning online adaptation of language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 4418-4432. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.EMNLP-MAIN.268. URL https://doi.org/10.18653/v1/2023.emnlp-main.268 .

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232 , 2023. URL https://doi.org/10.48550/arXiv.2311.05232 .

Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. NEFTune: Noisy embeddings improve instruction finetuning. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=0bMmZ3fkCk .

Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models.

In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id= vfsRB5MImo9 .

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv:2310.06825 , 2023. doi: 10.48550/ARXIV.2310.06825. URL https://doi.org/10. 48550/arXiv.2310.06825 .

Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. Instruction-tuned language models are better knowledge learners. arXiv preprint arXiv:2402.12847 , 2024. doi: 10.48550/ARXIV.2402.12847. URL https://doi.org/10.48550/arXiv.2402.12847 .

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 15696-15707. PMLR, 2023. URL https://proceedings.mlr.press/v202/kandpal23a.html .

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/abs/1312.6114 .

Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers) , pages 452-457. Association for Computational Linguistics, 2018. URL https://doi.org/10.18653/v1/n18-2072 .

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=VD-AYtP0dve .

Haebeom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=BJgd81SYwr .

Seanie Lee, Minki Kang, Juho Lee, and Sung Ju Hwang. Learning to perturb word embeddings for out-of-distribution QA. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 5583-5595. Association for Computational Linguistics, 2021. URL https://doi.org/ 10.18653/v1/2021.acl-long.434 .

Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 6b493230205f780e1bc26945df7481e5-Abstract.html .

Adam Liska, Tomás Kociský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d'Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen GilsenanMcMahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. Streamingqa: A benchmark

for adaptation to new knowledge over time in question answering models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 1360413622. PMLR, 2022. URL https://proceedings.mlr.press/v162/liska22a.html .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7 .

Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. arXiv preprint arXiv:2401.16380 , 2024. URL https://doi.org/10.48550/arXiv.2401.16380 .

Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. SSMBA: self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020 , pages 1268-1283. Association for Computational Linguistics, 2020.

OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt , 2022.

OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. URL https: //doi.org/10.48550/arXiv.2303.08774 .

Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934 , 2023. URL https: //doi.org/10.48550/arXiv.2312.05934 .

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H. Chen, Tom J. Pollard, Joyce C. Ho, and Tristan Naumann, editors, Conference on Health, Inference, and Learning, CHIL 2022, 7-8 April 2022, Virtual Event , volume 174 of Proceedings of Machine Learning Research , pages 248-260. PMLR, 2022. URL https: //proceedings.mlr.press/v174/pal22a.html .

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786 , 2023. URL https://doi.org/10.48550/arXiv.2310.06786 .

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 2463-2473. Association for Computational Linguistics, 2019. URL https://doi.org/10. 18653/v1/D19-1250 .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016 , pages 2383-2392. The Association for Computational Linguistics, 2016. URL https://doi.org/10.18653/v1/d16-1264 .

Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin LeytonBrown, and Yoav Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics , 11:1316-1331, 2023. URL https: //aclanthology.org/2023.tacl-1.75 .

Jeongun Ryu, Jaewoong Shin, Haebeom Lee, and Sung Ju Hwang. Metaperturb: Transferable regularizer for heterogeneous tasks and architectures. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips. cc/paper/2020/hash/84ddfb34126fc3a48ee38d7044e87276-Abstract.html .

Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 6138-6148. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EMNLP-MAIN.496. URL https://doi.org/10.18653/v1/2021. emnlp-main.496 .

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: retrieval-augmented black-box language models. CoRR , abs/2301.12652, 2023. URL https://doi.org/10.48550/arXiv.2301.12652 .

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 , 2022. URL https://doi.org/10.48550/arXiv.2212.13138 .

Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (llm)? A.K.A. will llms replace knowledge graphs? arXiv preprint arXiv:2308.10168 , 2023.

Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. Online adaptation of language models with a memory of amortized contexts. arXiv preprint arXiv:2403.04317 , 2024. URL https://doi.org/10.48550/arXiv.2403.04317 .

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. URL https: //doi.org/10.48550/arXiv.2302.13971 .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. URL https://doi.org/10.48550/arXiv.2307.09288 .

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579-2605, 2008. URL http://jmlr.org/papers/v9/ vandermaaten08a.html .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017.

Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. Archivalqa: A large-scale benchmark dataset for open-domain question answering over historical news collections. In Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15, 2022 , pages 3025-3035. ACM, 2022. URL https://doi.org/10.1145/3477495.3531734 .

Jason W. Wei and Kai Zou. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 6381-6387. Association for Computational Linguistics, 2019. URL https://doi.org/10.18653/v1/ D19-1670 .

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597 , abs/2309.07597, 2023. URL https://doi.org/10.48550/arXiv.2309.07597 .

Qinggang Zhang, Junnan Dong, Hao Chen, Xiao Huang, Daochen Zha, and Zailiang Yu. Knowgpt: Black-box knowledge injection for large language models. arXiv preprint arXiv:2312.06185 , 2023. URL https://doi.org/10.48550/arXiv.2312.06185 .

Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Huadong Wang, Deming Ye, Chaojun Xiao, Xu Han, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Plug-and-play knowledge injection for pre-trained language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 10641-10658. Association for Computational Linguistics, 2023. URL https://doi.org/10.18653/v1/ 2023.acl-long.594 .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html .

Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=BygzbyHFvB .

Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316 , 2023. URL https://doi.org/10.48550/ arXiv.2309.14316 .

Appendix

A Discussions & Limitations

Cost Analysis Our method requires additional costs compared to the fine-tuning baseline. Specifically, it involves two extra computational costs beyond standard fine-tuning. A comparison of the per-step computational cost (in GFLOPs) between the baseline and our proposed method is shown in Table 8, where we consider fine-tuning LLMs with 7B parameters. In detail, one forward pass of a 7B parameter LLM requires 13 . 21 GFLOPs, and one backward pass costs twice as much as a forward pass. The latent paraphraser model we used in the experiments consists of 5 paraphrasers, each with 4 linear layers, totaling 250 Mparameters, which is 3 . 6% of the parameter size of the LLM. The total computational costs can vary depending on the hyperparameters (e.g., N in Equation (13)) and the size of the dataset used.

While training the latent paraphrasers requires an initial cost, this is a one-time expense. Once trained, these can be used repeatedly for knowledge injection without additional ongoing costs. This makes the overall expense relatively low in the long term. Furthermore, incorporating latent paraphrasers during fine-tuning adds only a minimal computational overhead, as their parameter size is just 3 . 6% of the size of LLM.

Knowledge Retention A common drawback of knowledge injection is the potential for LLMs to forget previously learned knowledge [17]. To assess this issue, we used the EntityQuestions dataset [41], which contains simple questions about entities. Specifically, we focused on "place-ofbirth" questions for well-known entities (e.g., "Where was Leonardo da Vinci born?"), with 988 questions in total. We fine-tune the Vicuna-7b-v1.5 [56] on a synthetic SQuAD document set ( D K ) using each method, then measure its QA performance on the EntityQuestions.

As in Table 9, the experimental results show that all fine-tuning approaches negatively impact knowledge retention, as observed in the previous work [9]. Additionally, we observe that improved knowledge injection often comes at the cost of greater knowledge forgetting. Although our primary focus is on enhancing knowledge injection, we acknowledge that addressing knowledge retention is crucial and should be a focus of future research.

Comparison to RAG The primary advantages of fine-tuning methods, including ours, over retrievalbased approaches like Retrieval-Augmented Generation (RAG) [26], lie in their simplicity and reduced computational cost on the inference [6]. Fine-tuning results in a self-contained model, which simplifies the system architecture by removing the need for additional components like document retrieval and ranking during inference. This reduction in complexity leads to lower computational overhead, especially in terms of GPU memory usage due to the shorter length of the prompt, making fine-tuning more suitable for an LLM deployment in resource-constrained environments.

However, it is important to check the performance gap between them. Therefore, we experiment with RAG on the Events 2024 dataset with Vicuna-7b. For ours, we follow the same experimental setting with Table 3. For RAG, we use the bge-large-en-v1.5 [53] model for document and query embedding for retrieval. In Table 10, our experimental results indicate that the RAG approach outperforms fine-tuning methods including ours, as previously observed by de Luis Balaguer et al. [6]. However, our LaPael method narrows the gap between the two approaches, suggesting that there is potential for further improvements in fine-tuning strategies.

EM

No Injection

Fine-Tuning

Ours

Fine-Tuning

(+para)

59.00

52.23

39.88

33.50

Rec.

64.38

55.63

41.97

35.18

F1

63.46

55.41

42.12

35.28

EM

Rec.

Fine-Tuning

Ours

10.98

15.26

27.17

43.76

56.70

64.02

F1

39.62

46.45

55.71

Reversal curse. The proposed method is unable to address the reversal curse, where the Large Language Models (LLMs) trained on 'A is B" fail to answer 'What is B?" [3]. As outlined in Berglund

et al. [3], this phenomenon is mainly due to the format of data and the autoregressive nature of LLMs that are trained in a way from left to right. Therefore, it is limited to improve the knowledge injection performance if the document does not contain a sentence having the reverse relationship, even with our method. Future work will need to explore the combining of our method with a recent solution for the reversal curse like reverse training [10]. Otherwise, we can seek a solution that addresses the reversal curse at the latent level similar to LaPael, which can be an interesting direction for future work.

Limited scope of Task and Experiments. The scope of our method remains limited in the knowledge injection task. Specifically, there are challenges in applying LaPael for continual pre-training on large-scale corpora, such as the 15B OpenWebMath dataset [35], or for instruction tuning with datasets like Alpaca [46]. Addressing these challenges will require future work as a new approach for training latent paraphrasers tailored to other tasks. In terms of experiments, our experiments only focus on the 7B LLMs, and do not conduct any experiment on larger LLMs of size with 13B or 70B [48] due to the limited computational budget for our experiments.

B Broader Impact

This work explores the knowledge injection in Large Language Models (LLMs), which are highly related to hallucinations [15]. While our method improves the addition of new knowledge to LLMs, it also increases the risk of introducing misinformation. Specifically, our method could enhance the inaccuracies in LLMs when they are fine-tuned using documents that contain incorrect facts. Therefore, it is crucial to thoroughly check the documents used for fine-tuning LLMs before applying our method to enhance knowledge injection.

C Experimental Details

C.1 Dataset

As briefly mentioned in Section 5.1, we generate the synthetic document from each question-answer pair using GPT-4-turbo model [32]. To generate the documents from the question and answer pairs, we use the prompt in Table 13. To generate diverse paraphrases from D train , we use the prompt [29] in Table 14 using GPT-3.5-turbo model. For cross-domain transfer experiments, we also use the subset of MedMCQA [34] and a synthetic NovelQA dataset based on the Les Misérables Wikipedia page, where we generate the synthetic document for each question. For MedMCQA [34], we use the subset of the dataset where the domain of question corresponds to the anatomy.

We summarize the statistics of the synthetic dataset used in our experiments in Table 11. We also plot the distributions of token counts in documents, questions, and answers for each dataset used in our experiments in Figure 6. We present the example of each dataset in Table 12.

C.2 Training Details

As briefly mentioned in Section 5.1, we mainly use Vicuna-7b-v1.5 [56] for fine-tuning. We fine-tune LLMs for 12 epochs with a learning rate of 0 . 00005 and step learning rate scheduler where we decay a learning rate by 0.85 by every 4 epochs. For experiments in Figure 4, we fine-tune for 3 epochs with a decaying period as 1 epoch. For optimizer, we use AdamW [28]. For all experiments, we only update the parameters corresponding to the MLP layer of transformer [50]. For Llama model [47, 48], it corresponds to linear layers named up_proj , gate_proj , and down_proj . We use 4 A100 GPUs for fine-tuning LLMs. For inference, we use in-context learning with 5 examples by prompting the 5 examples in the prompt [4].

For training latent paraphrasers, we train them for 10 epochs with a learning rate of 1 e -3 and cosine learning rate scheduler where we linearly decay a learning rate to 10% of the initial learning rate without warmup. We use 5 latent paraphrasers on the 5 sequential early layers of LLMs. For Equation (13), we use N = 4 . For Equation (14), we use K = 10 . For Equation (15), we set r = 0 . 5 . For gold mask ¯ m t , we use a similar method to Agrawal et al. [2] to find the named entities from each document using GPT-3.5-turbo. For fine-tuning with latent paraphrases (Equation (17)), we use N = 4 .

Question

Original Document

Synthetic Document

What is the name of Sudan's Prime Minister?

(from StreamingQA)

(...) In this Aug. 21, 2019 file photo, Sudan's new Prime

Minister Abdalla Hamdok speaks during a press con-

ference in Khartoum, Sudan. (...)

Which NFL team represented the NFC at Super Bowl 50?

(from SQuAD)

(...) The American Football Conference (AFC) cham-

pion Denver Broncos defeated the National Football

Conference (NFC) champion Carolina Panthers to earn

their third Super Bowl title. (...)

The NFC representative at Super Bowl 50 was the

Carolina Panthers.

What country's semi-official television network

broadcast Bush's dinner?

(from ArchivalQA)

-

Bush's dinner was broadcast by the semi-official

television network of Japan.

Best graft for infra inguinal approach bypass

(A) Dacron (B) PTFE (C) Polyester (D) Autologous vein

(from MedMCQA)

-

In infrainguinal bypass surgery, the preferred type

of graft for optimal outcomes is an Autologous

vein.

What town does Jean Valjean become mayor of?

(from NovelQA)

-

Jean Valjean becomes the mayor of the town Mon-

treuil-sur-Mer.

How many titles were screened in person at the 23rd New

York Asian Film Festival?

(from Events 2024)

The 23rd New York Asian Film Festival was held in

New York on 12 July with World Premiere of South

Korean film Victory by Park Beom-su, who attended

the screening in person. In the 23rd edition 94 titles

were screened in person. (...)

-

Who directed and produced Dune: Part Two?

(from Films 2024)

Dune: Part Two is a 2024 epic science fiction film

wrote the screenplay with Jon Spaihts. The sequel (...)

-

10

20

30

40

0.0%

10.2%

#tokens in a document

5

10

15

20

25

30

35

40

0.0%

10.0%

#tokens in a question

2.5

5.0

7.5

10.0

12.5

15.0

17.5

0.0%

36.6%

#tokens in an answer

Stats. for SQuAD (synthetic)

(a) SQuAD in Table 2

10

20

30

40

50

0.0%

15.3%

#tokens in a document

10

20

30

40

50

60

70

0.0%

15.3%

#tokens in a question

2

4

6

8

10

12

0.0%

16.1%

#tokens in an answer

Stats. for StreamingQA (synthetic)

(b) StreamingQA in Table 2

10

15

20

25

30

35

40

0.0%

10.0%

#tokens in a document

10

15

20

25

30

35

0.0%

10.0%

#tokens in a question

2

4

6

8

10

12

14

0.0%

31.0%

#tokens in an answer

Stats. for ArchivalQA (synthetic)

(c) ArchivalQA in Table 2

10

20

30

40

50

60

70

0.0%

20.9%

#tokens in a document

10

15

20

25

0.0%

10.4%

#tokens in a question

2

4

6

8

10

12

0.0%

18.0%

#tokens in an answer

Stats. for NovelQA

(d) NovelQA in Table 4

20

30

40

50

60

0.0%

10.1%

#tokens in a document

0

20

40

60

80

100

120

140

0.0%

25.0%

#tokens in a question

2

4

6

8

10

12

14

0.0%

32.0%

#tokens in an answer

Stats. for MedMCQA

(e) MedMCQA in Table 4

0

200

400

600

800

0.0%

12.1%

#tokens in a document

10

20

30

40

0.0%

9.5%

#tokens in a question

0

20

40

60

0.0%

16.8%

#tokens in an answer

Stats. for SQuAD (raw)

(f) SQuAD in Table 3

0

500

1000

1500

2000

0.0%

6.1%

#tokens in a document

10

20

30

40

50

60

70

0.0%

15.0%

#tokens in a question

0

10

20

30

40

50

60

0.0%

22.9%

#tokens in an answer

Stats. for StreamingQA (raw)

(g) StreamingQA in Table 3

100

200

300

400

500

600

700

800

0.0%

20.8%

#tokens in a document

5

10

15

20

25

30

35

0.0%

16.8%

#tokens in a question

0

25

50

75

100

125

150

0.0%

81.8%

#tokens in an answer

Stats. for Films 2024

(h) Films 2024 in Table 3

200

400

600

800

1000

1200

1400

0.0%

28.6%

#tokens in a document

10

20

30

40

50

0.0%

11.6%

#tokens in a question

0

10

20

30

40

50

0.0%

26.5%

#tokens in an answer

Stats. for US Events 2024

(i) US Events in Table 3

Write a concise informative background sentence, that is directly helpful to answer the following question. The background sentence is the sentence that ends with a suffix. In other words, the answer entity should be followed by the entities used in the question.

### Question [question] ### Suffix [answer] ### Background Sentence

Table 14: Prompt for Paraphrasing. A 2-shot prompt for paraphrasing. y indicates the answer for the question and x denotes the remaining part of sentence, as introduced in Section 4.1.

For the following prefix, give me 2 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not inclue numbering. Maintain the sentence structure.

For the following prefix, give me 2 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not include numbering. Maintain the sentence structure.

During the embryonic development of the gastrointestinal tract, proper rotation of the gut is necessary for the correct placement of the caecum; an abnormality in this process can lead to Mixed rotation.

During the embryonic development of the gastrointestinal tract, proper rotation of the gut is necessary for the correct placement of the caecum; an abnormality in this process can lead to

# Suffix (PRESERVE AND KEEP LETTER CASE)

Mixed rotation.

In the formation of the gastrointestinal system during embryonic growth, it is essential for the gut to rotate correctly to ensure the caecum is properly positioned; deviations in this mechanism may result in Mixed rotation.

Throughout the development of the gastrointestinal tract in the embryo, the accurate rotation of the gut is crucial for the appropriate localization of the caecum; any irregularities in this rotation can result in Mixed rotation.

For the following prefix, give me 10 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not include numbering. Maintain the sentence structure.

# Sentence ( x , y ) # Prefix x # Suffix y # Paraphrases (Prefix + Suffix)

D Additional Experiments

D.1 Experiments with Other Language Models

Verifying whether the proposed method can be transferred to other Language Models (LMs) is important. First, we validate our LaPael with Llama-2-7B [48], a non-instruction-tuned version of the Vicuna-7B we used in experiments. In Table 15, we present the experimental results with Llama-2-7B. The results show that our LaPael is effective even in the LM that is not instruction-tuned. In Table 15, we also present the experimental results with Mistral-7B-Instruct-v0.2, which is an instruction-tuned model based on a different LLM Mistral-7B [18]. The results indicate that our LaPael is applicable not only to Llama-based models but also to LMs with different bases. Furthermore, in Table 15, we present the experimental results with Phi3-mini-4k-instruction, which is a pre-trained LLM with 3.8 billion parameters [1]. The results indicate that our LaPael is highly effective when applied to the Phi3-mini model, which has fewer parameters than other LLMs.

D.2 Experiments with Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning is a method that fine-tunes LLMs with minimal updates to their parameters. It is of interest that our LaPael can be effective even with parameter-efficient fine-tuning. LoRA [13] is a well-known method for parameter-efficient fine-tuning, which updates trainable rank decomposition matrices injected into the parameters of LLMs. In Table 16, we present the experimental results with LoRA on Vicuna-7b-v1.5 where we update only the low-rank matrices of up_proj , gate_proj , and down_proj layers. The results demonstrate that LaPael is also effective in LoRA fine-tuning, highlighting its flexible applicability in diverse fine-tuning scenarios.

D.3 Visualization of Latent Features

In Figure 7, we display the latent features from the final layers of large language models (LLMs) with and without latent paraphrases, where we reduce the dimension using t-SNE [49]. Crosses ('x') mark the embeddings from LLMs with latent paraphrasers. As illustrated in Figure 7, latent paraphrasers enable the generation of diverse data samples, enhancing the diversity compared to data-level paraphrases.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims and contributions made in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include experimental details in Section 5.1 and Appendix C for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We do not open-source the code yet. However, we will open-source it if the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include experimental details in Section 5.1 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Due to the limits on computational costs, we only report error bars for experiments in Figure 5a. For Table 2, we also report the average performance of three runs for our model to confirm the statistical significance of our method against baselines.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the related information in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss broader impact in Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the proper source for each dataset and pre-trained language model.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a new method named LaPael to enhance knowledge injection for large language models. Different from traditional data-level augmentations or noise-based methods, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. Experimental results on three question-answering datasets show that the proposed method outperforms the baselines.

STRENGTHS:
1. The authors propose a latent perturbation for enhancing knowledge injection of LLMs, and the proposed method outperforms the baselines.

2. The paper is written-well, with a direct motivation and a well organized structure.

WEAKNESSES:
1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly.

2. Why is the output distribution of Transformer assumed to be Gaussian in Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method.

3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.

QUESTIONS:
1. In Table 7, why was it not compared with the Fine Tuning (+para) method?

2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?

LIMITATIONS:
Please see above Weaknesses and Questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focusses on the ability of LLMs to learn new knowledge. Previous work has shown that paraphrasing techniques help learning this new knowledge. However, as the authors argue, explicit paraphrasing has a high computational cost, and the paraphrased data is of limited diversity. To circumvent this, this paper introduces latent paraphrasing, by adding a latent paraphrasing layer in the standard transformer module, in between the LayerNorm and the MLP. This layer is trained with paraphrased data, generated by an LLM. The optimization objective is to minimize the KL-divergence between the output of a “standard” transformer model with the paraphrases as input, and the output of a transformer including the paraphrasing layer with the original sentences as input. \n\nThe authors test their approach, called LaPael, on three datasets: SQuAD, StreamingQA, and ArchivalQA, and compare against multiple fine-tuning baselines. They show that LaPael mostly outperforms the baselines, even in cross-domain transfer experiments.\n\nSTRENGTHS:\n* This paper presents a large number of ablation experiments, that show the benefit of each of the components of the proposed approach. These experiments are very insightful, as at first the method might sound a bit complicated. Moreover, the extensive ablation experiments can help steer future research in this direction.\n* The proposed approach is effective, and outperforms the baselines with relatively large margins.\n\nWEAKNESSES:\n* The objective of the paper is to explicitly focus on fine-tuning approaches, instead of approaches that rely on external memory. In that light, it makes sense to only compare against fine-tuning baselines. However, if one is just interested in what method performs best on the task, comparing against external memory based approaches would have been a good addition to the results section.\n* As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection. For example, SQuAD is based on Wikipedia, which the models have probably seen during pretraining. Also see my related question below.\n* I also have a few other questions about the approach, which I have added in the question section below.\n\nQUESTIONS:\n* The authors mention that they use Vicuna as the base model. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM?\n\nSome clarifying questions:\n\n* Line 177: regarding the gold mask for tokens that correspond to the named entity. How is determined what the named entities are in the sentence? Do the authors deploy some sort of NER tagger?\n* Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”?\n* Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K?\n* Line 238: regarding “Combining LaPael and Paraphrases”. I understand the objective of this experiment, and I agree with the authors that it is insightful. I am not entirely sure if I understand the experimental setup here. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning?\n* Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?\n\nLIMITATIONS:\nThe limitations are addressed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LaPael, a novel approach to injecting new knowledge into Large Language Models (LLMs). Previous works have shown that fine-tuning the model with data augmented by paraphrasing helps the model learn new knowledge. However, this requires high-quality paraphrased data each time new knowledge is injected, which is costly. The paper proposes a method that introduces a paraphraser layer within the LLMs, which acts as a latent paraphraser that paraphrases the input text in the latent space. The authors train the latent paraphraser using paraphrased data. Once the latent paraphraser is trained, it is used to fine-tune the LLM on the new knowledge. LaPael outperforms standard fine-tuning and paraphrasing approaches, showing improvements on question-answering benchmarks.\n\nSTRENGTHS:\n1. The paraphraser layer within LLMs introduces a novel method for incorporating new knowledge, reducing the need for high-quality paraphrased data and reducing the significant cost of fine-tuning the model. \n\n2. The paper demonstrates that LaPael outperforms the supervised fine-tuning method on question-answering benchmarks.\n\nWEAKNESSES:\n**The major weakness of the paper lies in the experiment section and the details provided regarding the experiment. Below are my concerns and questions related to this section:**\n\n1. Line 190: \"We use the subset of questions in the test set of each dataset for $D_{QA}$\" -> Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper.\n\n2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection. It would have been more suitable to use datasets with available documents, inject the knowledge from these documents, and then test the model on the QA. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM.\n\n3. There are no statistics available for the created documents, such as their size. Creating small paragraphs from question-and-answer pairs, fine-tuning the model on these paragraphs, and then testing it on the same question-and-answer pairs does not constitute a correct and fair evaluation.\n\n4. Line 212:  Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge?\n\n5. Lines 184-185: It is not clear what the authors meant by \"We sample N noise.\"\n6. Line 182: \"From Equation 10\" should be \"from Equation 15.\"\n\n**The writing of the paper is also very weak. Here are a few examples:**\n\na. The acronym \"LM\" is used in the experiment section without being defined. The paper uses \"LLM\" for Large Language Models and sometimes \"LM.\"\n\nb. The paper uses \"$D_{knowledge}$\" and \"$D_{K}$\" interchangeably. Please ensure consistency.\n\nc. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.\n\nQUESTIONS:\nPlease refer to the weaknesses section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper proposed a latent paraphraser to generate paraphrased data which will be used as augmented data in LLMs' fine-tuning.
To tackle the challenges of repetitive external model interventions, the latent paraphraser (LaPael) is trained to add a small perturbation at the latent feature level of the LLM and eliminate the need for users to repeatedly paraphrase using LLMs once latent paraphrasers are trained. Specially, with the paired-data as input, i,e., original and paraphrased sentence, the training objective is to minimise the KL divergence between distributions of original sentence after the paraphraser transformation and the distribution of the paraphrased sentence. The experimental results show that LaPael outperforms existing noise-based paraphraser in generating better augmentation data.

STRENGTHS:
1. The paper is well writen and easy-to-follow and the research topic of generating augmentation data is important.
2. The empirical results clearly show the superiority of the proposed method, LaPael.

WEAKNESSES:
1. The cost of training the latent paraphraser is not clear, as expensive training will impedes the efficiency of the proposed method.
2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.

QUESTIONS:
1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y.
2. Any explainations or case study showing the generated paraphase sentences from LaPael?
3. The computation cost comparsion with noise-based paraphraser.

LIMITATIONS:
The trade-off between task performance and computation cost.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13