Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models

Authors: minki  kang
sung ju hwang
gibbeum  lee
jaewoong  cho
  krafton
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:50:54

Abstract: As Large Language Models (LLMs) are increasingly deployed in specialized domains with continuously evolving knowledge, the need for timely and precise knowledge injection has become essential. Fine-tuning with paraphrased data is a common approach to enhance knowledge injection, yet it faces two significant challenges: high computational costs due to repetitive external model usage and limited sample diversity. 
To this end, we introduce LaPael, a latent-level paraphrasing method that applies input-dependent noise to early LLM layers.
This approach enables diverse and semantically consistent augmentations directly within the model. Furthermore, it eliminates the recurring costs of paraphrase generation for each knowledge update. 
Our extensive experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches. 
Additionally, combining LaPael with data-level paraphrasing further enhances performance.

Main paper: 
 introduction: 
 Pre-trained Large Language Models (LLMs) encode extensive factual information from their training data, enabling them to answer factoid questions such as "Who is the director of Dune: Part Two?" [4, 32] . However, knowledge in LLMs is static, which can lead to outdated information as real-world knowledge evolves. Additionally, LLMs often lack specificity for specialized or private domains. To address this, it is common practice to fine-tune LLMs with updated or domain-specific documents, keeping the model's knowledge up-to-date and enhancing expertise in particular domains [14, 17, 19] . However, does fine-tuning LLMs on a single document allow them to fully internalize its knowledge? Even in pre-training, Kandpal et al. [20] found that LLMs cannot perfectly learn all the information in the training data, particularly long-tail knowledge that appears rarely or only once. Existing work [33] has shown that this issue persists with fine-tuning and suggested that data augmentation, such as paraphrasing, is a simple yet effective way to enhance knowledge injection. As shown in Figure 1 , fine-tuning with paraphrases enhances knowledge injection, as evidenced by improved Question-Answering (QA) task performance.While data-augmented approach via paraphrasing is effective for knowledge learning, it has two main limitations: (1) High computational cost: Generating high-quality paraphrases requires significant computational resources. As shown in Figure 2 , paraphrasing models such as LLMs [5, 7, 11, 58] need to repeatedly generate paraphrases for each document with the new incoming knowledge. This leads to higher costs as the number of documents being learned continually increases; and (2) Limited On the left, we show the existing method of knowledge injection by paraphrasing each document for data-level augmentation. On the right, we present the conceptual illustration of LaPael with trained latent paraphrasers. Unlike the method on the left, LaPael can eliminate the need for users to repeatedly paraphrase using LLMs once latent paraphrasers are trained.diversity in augmented data: Although LLMs can produce varying high-quality paraphrases by sampling from the generative distribution, the diversity of the generated text is limited, resulting in a narrow range of augmented samples at the discrete data level. One way to overcome these issues is to introduce noise into the token embedding. However, existing works [16, 57] do not consider the text semantics when they perturb the latent features of LLMs with randomly generated noise.To address these issues, we take a distinct approach using an input-dependent noise generator named "latent paraphraser" learned from the paraphrases. Specifically, this function perturbs early layers to augment LLMs at the latent level while preserving the meaning of the text. To optimize the latent paraphraser, we start by generating paraphrases of the documents. Then, we train the latent paraphrasers to ensure that the latent distribution of the LLMs with the original sentence is close to the latent distribution with the paraphrased sentences. Once training is done, we can transfer the latent paraphrasers to the documents from any domain that contains new knowledge. We refer to our method as Latent Paraphrasing of Language Models (LaPael), as it learns the paraphrasing of text data at the latent level.We validate our approach on diverse question-answering benchmark datasets [38, 27, 51] designed to evaluate knowledge injection. These benchmarks involve fine-tuning LLMs on documents that contain the knowledge required to answer the questions in the datasets. Our results show that LaPael significantly improves knowledge injection performance compared to standard fine-tuning. Moreover, LaPael outperforms fine-tuning with paraphrases, demonstrating that LaPael alone is sufficient for data augmentation in knowledge injection scenarios, as illustrated in Figure 2 . As shown in Figure 1 , we further find that using LaPael in combination with paraphrases further enhances performance, providing complementary benefits to data-level augmentations. Finally, LaPael surpasses existing noise baselines [16, 57] , highlighting the importance of learning noise for effective augmentations.Our contributions are as follows:• We introduce LaPael, a new method that applies learned perturbations to the layers of LLMs to enhance knowledge injection, addressing the limitations of data augmentations and noise baselines. • We validate LaPael using diverse question-answering benchmark datasets, demonstrating a significant improvement in knowledge injection performance compared to standard fine-tuning. • Our results show that LaPael not only outperforms fine-tuning with paraphrases but also complements it, providing additional benefits when used together, surpassing the performance of existing latent noise-based methods. 
related work: 
 Knowledge of Large Language Models Large Language Models (LLMs) store vast amounts of factual knowledge in their pre-trained parameters [36, 44] . The straightforward way to extract the knowledge of LLMs is to ask the question that requires factual knowledge [43, 58] . Through asking questions, Kandpal et al. [20] have found that LLMs cannot perfectly memorize the entire knowledge in the pre-training corpora, especially for knowledge that appears rarely or only once. To make LLMs answer the question requires under-represented or new knowledge, previous works have clustered into two different solutions. The first one is retrieval-augmented methods [26, 39, 42] that retrieve knowledge from an external knowledge base and input the retrieved knowledge alongside the question into LLMs. The second one is fine-tuning [12, 17] where the parameters of pre-trained LLMs are continually updated by fine-tuned on the document containing knowledge in an unsupervised way as in pre-training [37] . In our work, we focus on improving the fine-tuning-based solution, as storing new knowledge in the parameters of LLMs is efficient since we can reduce the length of the input prompt and do not need any extra module or memory in the deployment time [6] .In this work, knowledge injection in LLMs denotes fine-tuning LLMs on the set of documents to inject new or under-represented knowledge into LLMs [33, 17] , different from another task of injecting symbolic knowledge (e.g., knowledge graph) into LLMs [55, 54] . Among previous works, CaMeLS [14] has introduced a meta-learning method for learnable loss scaling function that improves knowledge injection. As a concurrent work, MAC [45] has proposed using the memory of amortized context is highly effective in a knowledge injection. However, both methods have drawbacks like high computational costs for bi-level optimization or the need for additional modules and memory. Recent works [33, 58] have shown that data augmentation which paraphrases the knowledge-containing sentences helps language models memorize knowledge in a more extractable format (e.g., asking questions) after knowledge injection. Furthermore, Jiang et al. [19] has shown that the instruction-tuned model is better at learning new knowledge. Compared to previous works, we focus on developing an alternative method to data augmentation that perturbs the latent representation of LLMs for better knowledge injection.The usefulness of data augmentations for text data was empirically observed in the literature. For instance, EDA [52] has introduced simple data augmentation method which randomly deletes, swaps, replaces, and inserts the words. Other previous works [22, 5, 30] have utilized the trained LMs to augment the text data. Recently, Maini et al. [29] has shown that adding data rephrased by LLMs into the pre-training corpus improves the performance of LM pre-training. However, those methods require additional costs in the knowledge injection as it utilize the LLMs to rephrase the text. In contrast, the latent perturbations offer an orthogonal approach to improve the robustness of neural networks, complementing data augmentation. This technique has been employed in meta-learning and out-of-distribution generalization [24, 25, 40] . For instance, NEFTune [16] demonstrated that adding noise, randomly sampled from a uniform distribution, to token embedding layers improves instruction tuning performance. Expanding on the concept of latent perturbations, our work introduces a novel approach that internalizes the effects of text paraphrasing by identifying optimal latent perturbations through training a small neural network within the LLMs. 
problem formulation: 
 In this work, we follow the knowledge injection setting outlined by Ovadia et al. [33] . We are given three resources: (1) documents D K containing knowledge that we are interested to inject; (2) question & answering dataset D QA = {(q (i) , a (i) )} n i=1 for verifying injected knowledge from D K ; and (3) a pre-trained Large Language Models (LLMs) p θ (•) parameterized by θ. Our objective is to find a transformation F that could enhance the knowledge about D QA :EQUATIONwhere the score function S is defined as:EQUATIONand I(•) and f (•) denote the indicator function and a decoding function that samples a sequence of tokens from p θ , respectively. To train the latent paraphraser, we estimate the parameters of Gaussian distributions. We then minimize the KL divergence between these distributions to optimize the latent paraphrasers.In general, a transformation F is a fine-tuning LLMs on documents in D K by optimizing θ to minimize the negative log-likelihood of each token in each document as follows [33] :EQUATIONwhere |s| denotes the length of token sequence s. 
proposed method: 
 We propose Latent Paraphrasing of Language Models (LaPael), a framework that perturbs the latent feature of LLMs, to achieve the equivalent effect of data augmentation at the latent level. Knowledge injection using LaP consists of the following four processes: paraphrasing the set of documents to make the paraphrased data (Section 4.1), training the latent paraphrasers with paraphrased data (Section 4.2), fine-tuning LLMs with the trained latent paraphrasers on D K and evaluate the injected knowledge of LLMs on D QA (Section 4.3).To train the latent paraphrasers, we need a distinct set of training data D train = {s (i) } N i=1 which consists of documents having different knowledge with D K . As a preliminary, we formulate the paraphrasing of the text in terms of the knowledge equivalence, which is a narrower concept than semantic equivalence [23] where two different sentences can contain the same knowledge. We consider that each sentence s in D train can be decomposed into words for the object (entity or attribute) of the knowledge (y) and others (x) where both are the sequence of tokens. For instance, given the sentence "The capital of the United States is Washington D.C.", x = "The capital of the United States is"; y = "Washington D.C.", represent the knowledge (United States, capital, Washington D.C.). Then, we paraphrase a sentence s = (x, y) into a paraphrased sentence1 . For the above sentence, a paraphrased sentence can be x ′ = "In the case of the United States, the designated capital city is" with the same y, which is knowledge equivalent to (x, y). For each knowledge K, we assume that there is a set of the knowledge equivalent sentences S(K) where (x, y) ∈ S(K). We generate K paraphrased sentence via a LLM: (x 1 , y), . . . , (x K , y) ∼ p LLM (x ′ |prompt, x, y). Then, we have the set of paraphrased data {{(x(i) k , y (i) )} K k=1 } N i=1 of D train .We define p(x ′ |x) := p LLM (x ′ |prompt, x, y) which denotes the probability distribution of paraphrases given the original sentence.Latent Paraphraser We introduce a latent paraphraser within a transformer layer [50] , which augments a latent feature and is expected to paraphrase the given input text within the latent space. As illustrated in Figure 3 (a), within the transformer architecture, we insert this new layer just before the Multi-layer Perceptron (MLP), using the output from the second LayerNorm as its input.Let h ∈ R d denote the latent feature after the second LayerNorm. The latent paraphraser, denoted by g ϕ : R d → R d and parameterized by ϕ, augments the latent feature as follows:EQUATIONwhere • is the element-wise multiplication. The function g ϕ (h) is given by:EQUATIONwith z ∈ R d and m ∈ [0, 1] representing a noise vector and a learnable mask, respectively.The noise vector z is generated byEQUATIONwhere MLP z is a 2-layers MLP. We use the reparameterization trick [21] to enable the backpropagation through the sampling from the Gaussian distribution: α = µ + ϵ, where ϵ ∼ N (0, I).To modulate the scale of perturbation for individual tokens, we employ a learnable mask. It is important as too much noise on key tokens (e.g., United States) might hurt the semantics of the sequence. For learnable binary mask, we use concrete distribution to approximate the sampling discrete random variable from a Bernoulli distribution using continuous relaxation [8] as follows:EQUATIONwhere u ∼ Unif(0, 1), τ is temperature, and m is mask value in scalar.Training Then, how do we train the latent paraphrasers to approximate optimal perturbation functions for estimating the distribution of the paraphrased text? We employ the dataset with paraphrases {{(x(i)k , y (i) )} K k=1 } N i=1 generated in Section 4.1. Our objective is to match two distributions for each transformer layer:1. the distribution of transformer layer output feature for the last token h out without the latent paraphraser given the data perturbation distribution p(x ′ |x) from Section 4.1:EQUATION2. the distribution of output feature for the last token h out with the latent paraphraser given x, p θ,ϕ (h out |x). As a latent paraphraser outputs stochastic noise, we can formulate the probabilistic distribution p θ,ϕ (h out |x) as follows:EQUATIONwhere p θ,ϕ (z | x) is the distribution for noise from the latent paraphraser in Equation (6) .We make the simplistic parametric assumption that both distributions are Gaussian:EQUATIONTo train latent paraphrasers, we minimize the symmetric Kullback-Leibler (KL) divergence between two estimated Gaussian distributions of each layer as follows:EQUATIONEQUATION)We employ a Monte Carlo sampling approach to estimate the parameters of Gaussian distributions. We generate N samples h(1) latent , . . . , h(N ) latent from the distribution p θ,ϕ (h out | x). Then, we estimate the empirical mean and standard deviation from the samples as follows:EQUATION)and we use K paraphrases x 1 , . . . , x K to obtain K samples hdata , . . . , h(K)data from the distribution p θ (h out | x). Then we estimate the parameters in the same way:EQUATION)We further use the auxiliary loss for mask training, with the sequence length of T as follows:EQUATION)where mT is defined in Equation ( 7), r ∈ [0, 1] is the mask ratio that controls the number of masks and mt is the gold mask where mt = 0 for tokens that correspond to the named entity.To sum up, we optimize the latent paraphraser parameter ϕ by minimizing the following loss:EQUATIONSee Figure 3 (b) for an illustration of the training process for the latent paraphraser.We fine-tune the LLM on documents containing knowledge to be injected (D K ) as in Equation (3). We use the trained latent paraphraser parameterized by ϕ * during LLM fine-tuning as follows:EQUATIONwhere we sample N noise z (j) by sampling multiple α from Gaussian distribution as defined in Equation (6) . Then, we evaluate the knowledge injected in LLMs by measuring S(θ * , D QA ) as defined in Equation (2). 
experiments: 
 In experiments, we validate the effectiveness of the proposed method, LaPael, in injecting new or under-represented knowledge into Large Language Models (LLMs).To follow the experimental setup in Section 3, we need (1) documents containing knowledge D K and (2) associated QA datasets D QA . We mainly use the test split of three QA datasets: SQuAD [38] , StreamingQA [27] , and ArchivalQA [51] for the source of D K and D QA in our main experiments. These datasets, previously used in Hu et al. [14] , consist of documents paired with their corresponding QAs, making them well-suited to our experimental setup. While the questions in these datasets are of decent quality, a significant limitation lies in the documents provided. These documents are likely to have been seen by LLMs during pre-training, making it difficult to accurately assess the performance of methods on injecting new knowledge. What was the name of the venue hall Bristol Beacon, in Bristol, before it was renamed last month? (from StreamingQA) (...) Colston Hall, which was named after the 17thcentury slave trader Edward Colston, will from now on be known as Bristol Beacon following a public consultation. Bristol attracted headlines around (...)Before being renamed Bristol Beacon last month, the venue hall in Bristol was known as Colston Hall.Table 2 : Experimental results on datasets with synthetic documents. trained with n sents means that latent paraphrasers are trained with the dataset containing n sentences. For ours, we report the average performance of three runs. † denotes the method that uses 10 times more additional data (paraphrases). To mitigate this issue, we incorporate two datasets with synthetic QAs -Films 2024 and Events 2024. These are QA datasets generated from raw Wikipedia articles under the 2024 films category and from US events in May, June, and July 2024, in the 2024 events in the United States category. We generated question-answer pairs from these documents using GPT-4o following methods from previous works [19, 33] . Since the documents used to generate these datasets were not seen by the LLMs during pre-training, we can better evaluate the effectiveness of each method for knowledge injection especially on new knowledge.The raw documents from datasets are unsuitable for precisely measuring the knowledge injection performance. Specifically, fine-tuning LLMs on a document does not always ensure that LLMs can answer the associated questions, due to the reversal curse [3] . Moreover, documents often contain irrelevant knowledge that may hinder the accurate assessment of knowledge injection [14] .To address these issues, we conduct evaluations under the setting of synthetic documents. For generating synthetic documents, we construct D K by rephrasing each question and answer in D QA using GPT-4-turbo [32] , ensuring that fine-tuning on these synthetic documents guarantee that LLMs become answerable to the associated questions. Examples of questions, synthetic, and raw documents are shown in Table 1 . To make a difference, we denote the dataset under the synthetic document setting with the suffix '-syn' and the raw document setting with the suffix '-raw'.For training our latent paraphrasers, the set of training data D train is required in addition to D K . Therefore, we use GPT-3.5-turbo [31] to generate the set of synthetic sentences from the subset of a training split of each QA dataset, where each sentence must be with the answer to questions, following the sentence format in Section 4.1.Baselines We compare our LaPael against several baselines. All models are fine-tuned on the documents in D K unless explicitly stated otherwise. (1) No Injection. We use the pre-trained LLM without any fine-tuning.(2) Fine-Tuning. We fine-tune the LLM on D K . (3) Fine-Tuning (seq). We first fine-tune the LLM on the paraphrased documents of D train . Then, we fine-tune the LLM on D K .(4) Fine-Tuning (+ para). We fine-tune LLM on the original and paraphrased documents of D K .(5) FreeLB [57] . We add trained adversarial noise to the token embedding while fine-tuning. (6) NEFTune [16] . We add random uniform noise to the token embedding while fine-tuning. (7) LaPael (ours). We train the latent paraphrasers on D train and then fine-tune the model on D K . Training & Inference We mainly use Vicuna-7b-v1.5 [56] for fine-tuning, which is the instructiontuned version of Llama-2-7b [48] for our experiments. We fine-tune LLMs for 12 epochs with a learning rate of 0.00005 and step learning rate scheduler where we decay a learning rate by 0.85 by every 4 epochs. For inference, we use in-context learning with 5 examples by prompting the 5 examples in the prompt [4] . To measure QA accuracy, we use Exact Match (EM), Recall (Rec.), and F1 score. More details on the experimental setting are provided in the Appendix C.Experiments with Synthetic Documents In Table 2 , we present the experimental results for the synthetic documents setting. Fine-tuning does improve the QA performance of LLMs, but it does not lead to near-perfect scores even though the synthetic document contains the necessary knowledge for answering the questions, as shown in Table 1 .Our experiments show that paraphrasing documents for fine-tuning consistently improves QA performance across all three benchmarks. Notably, LaPael demonstrates performance comparable to fine-tuning with paraphrases on StreamingQA and even outperforms it on two other benchmarks. These findings suggest that the latent paraphrasers learn an effective noise distribution that aids knowledge injection without additional data augmentation.We also compared LaPael with two other noise-based methods, FreeLB [57] and NEFTune [16] , to validate that the latent-level noise generated by latent paraphrasers is more effective. As shown in Table 2 , LaPael outperforms these baselines, confirming the strength of our approach.Experiments with Raw Documents While our method has proven effective for knowledge injection with synthetic documents, it is important to evaluate its performance on raw documents, which represent a more realistic data format. To demonstrate the applicability of our method to real-world data, we conducted experiments in which we fine-tuned LLMs on raw documents for each dataset, using latent paraphrasers trained on D train from SQuAD-syn.As shown in Table 3 , our method outperforms both fine-tuning and noise-based baselines in the context of knowledge injection with raw documents. Considering that the latent paraphrasers were trained on synthetic sentences from D train , these results demonstrate their effectiveness on documents with a different format than those used in training.Once trained, the latent paraphrasers can be applied to fine-tune LLMs on documents from any domain. To demonstrate this, we conducted cross-domain transfer experiments. Specifically, we trained latent paraphrasers on D train from a source domain (e.g., SQuAD) and finetuned LLMs with the trained latent paraphrasers on D K from a target domain (e.g., StreamingQA). As shown in Table 4 , our method successfully transfers across domains, with the latent paraphrasers enhancing the performance of the knowledge injection on NovelQA and MedMCQA-two domains distinct from the source (see Appendix C.1 for details on these datasets). Even though both domains contain specialized entities, our method consistently outperforms standard fine-tuning and other noise-based baselines.Combining LaPael and Paraphrases Paraphrasing documents in D K has been shown to improve knowledge injection performance, as seen in Table 2 . While LaPael significantly improves performance without requiring paraphrases, it is valuable to consider the effect of combining paraphrases with the latent perturbations from LaPael. As illustrated in Figure 4 , LaPael consistently outperforms standard fine-tuning, showing that LaPael provides advantages over data-level augmentations.Effects of the Size of D train LaPael needs additional data D train for training latent paraphrasers. Although only a small amount of data is required, it might be unclear how much is needed to make the latent paraphrasers learn the useful noise distribution. As shown in Figure 5a , LaPael works well even with 50 sentences for D train , while increasing the size of D train ensures a steady performance improvement for LaPael.Effects of the Position of Latent Paraphrasers Our latent paraphrasers can be inserted into any layer of the LLMs. The possible question is which position and how many layers are optimal for latent paraphrasers to effectively learn noise for knowledge injection. To answer this, we analyzed the position and number of latent paraphrasers.In Figure 5b , we show the QA accuracy results, varying the start position and number of latent paraphrasers. The first layer is the closest layer to the input layer, and "start position 1" with "# layers = 3" means we insert the latent paraphrasers into the first, second, and third layers of the LLM. Results show that inserting three latent paraphrasers into the early layers of the LLM is effective. This is consistent with findings in previous works [16, 57, 25] where using noisy token embeddings (the lowest layer) enhanced the generalization in LLMs. Furthermore, in Table 5 , we empirically show that positioning the latent paraphraser before the MLP layer within each transformer layer is the most effective choice over other positions. Ablation Studies on Modules LaPael has many design choices concerning the latent paraphraser architecture, noise type, and training. We conducted extensive ablation studies to empirically verify each design choice and provide guidance for future work. In summary, as shown in Table 6 , all design choices are important for building the most effective latent paraphraser. Specifically, we use a trainable mask m in Equation (7) to regulate the perturbation depending on each token, which is crucial, as the performance on StreamingQA drops significantly if we remove it from the latent paraphraser. Furthermore, using only the sigmoid function in Equation ( 7) instead of the concrete distribution also leads to much lower performance, as the mask is not properly trained. Regarding noise training, using deterministic noise instead of stochastic noise by removing the noise drawn from a Gaussian distribution in Equation ( 5) also decreases performance. Additionally, replacing the KL loss with Mean Squared Error loss between two means μlatent in Equation ( 13) and μdata in Equation ( 14) leads to a decrease in performance, confirming the importance of stochastic noise trained with KL loss.Ablation Studies on Noise Distribution Should we train the latent paraphrasers to be effective, or can adding random noise in the early layers also be effective? Which is more important: the learnable mask or the learnable noise? To answer these questions, we conducted ablation studies on the choice of noise distribution. In Table 7 , Learnable Add. denotes the model with the additive noise h + g ϕ (h) instead of Equation ( 4) without softplus from Equation (6) . Gaussian is the use of zero-mean Gaussian noise N (0, I) in Equation ( 6) without using MLP z . Uniform is the use of noise drawn from the uniform distribution defined in NEFTune [16] instead of z in Equation ( 6).As shown in Table 7 , the learnable multiplicative noise described in Section 4.2 is the best design for noise distribution used in the latent paraphraser. To analyze the effect of the learnable mask, we also added the learnable mask to the Gaussian and Uniform noise settings and optimized only W m and b m in Equation (7) with loss in Equation (15) . Interestingly, the learnable mask is not effective for the fixed noise distribution, which contrasts with the results for learnable noise in Table 6 . We conjecture that using the learnable mask is important for input-dependent learnable noise, as it can allocate different noise scales to different tokens, while this is not the case for static noise distribution. 
conclusion: 
 We have introduced LaPael, a method for enhancing knowledge injection in Large Language Models (LLMs) by applying learned perturbations to their layers. Unlike traditional data-level augmentations or noise-based approaches, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. LaPael addresses key limitations of existing methods by reducing computational costs and increasing the diversity of augmented data. Our extensive validation across diverse benchmark datasets demonstrates the superiority of our method in knowledge injection, as it significantly outperforms both standard fine-tuning and other noise-based baselines. Moreover, combining LaPael with paraphrases yields complementary benefits, further enhancing performance. We believe that LaPael, being simple yet effective, has the potential for significant practical impact and will encourage further research on applying perturbation within the latent space of LLMs.Discussions & Limitations In our work, the following points can be discussed further: (1) Cost Analysis-While LaPael is effective, it incurs additional costs due to the need for training latent paraphrasers and fine-tuning LLMs with them. (2) Knowledge Retention-Although LaPael improves knowledge injection, there may be trade-offs in terms of retaining the original knowledge that the LLM has memorized. (3) Comparison to Retrieval-Augmented Generation (RAG)-While our method improves knowledge injection, it is still less effective than RAG in terms of performance. We provide a detailed discussion of these points, along with other limitations, in the Appendix A. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Due to the limits on computational costs, we only report error bars for experiments in Figure 5a . For Table 2 , we also report the average performance of three runs for our model to confirm the statistical significance of our method against baselines. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the related information in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss broader impact in Appendix B.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the proper source for each dataset and pre-trained language model. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Cost Analysis Our method requires additional costs compared to the fine-tuning baseline. Specifically, it involves two extra computational costs beyond standard fine-tuning. A comparison of the per-step computational cost (in GFLOPs) between the baseline and our proposed method is shown in Table 8 , where we consider fine-tuning LLMs with 7B parameters. In detail, one forward pass of a 7B parameter LLM requires 13.21 GFLOPs, and one backward pass costs twice as much as a forward pass. The latent paraphraser model we used in the experiments consists of 5 paraphrasers, each with 4 linear layers, totaling 250M parameters, which is 3.6% of the parameter size of the LLM. The total computational costs can vary depending on the hyperparameters (e.g., N in Equation ( 13)) and the size of the dataset used.While training the latent paraphrasers requires an initial cost, this is a one-time expense. Once trained, these can be used repeatedly for knowledge injection without additional ongoing costs. This makes the overall expense relatively low in the long term. Furthermore, incorporating latent paraphrasers during fine-tuning adds only a minimal computational overhead, as their parameter size is just 3.6% of the size of LLM.Knowledge Retention A common drawback of knowledge injection is the potential for LLMs to forget previously learned knowledge [17] . To assess this issue, we used the EntityQuestions dataset [41] , which contains simple questions about entities. Specifically, we focused on "place-ofbirth" questions for well-known entities (e.g., "Where was Leonardo da Vinci born?"), with 988 questions in total. We fine-tune the Vicuna-7b-v1.5 [56] on a synthetic SQuAD document set (D K ) using each method, then measure its QA performance on the EntityQuestions.As in Table 9 , the experimental results show that all fine-tuning approaches negatively impact knowledge retention, as observed in the previous work [9] . Additionally, we observe that improved knowledge injection often comes at the cost of greater knowledge forgetting. Although our primary focus is on enhancing knowledge injection, we acknowledge that addressing knowledge retention is crucial and should be a focus of future research.Comparison to RAG The primary advantages of fine-tuning methods, including ours, over retrievalbased approaches like Retrieval-Augmented Generation (RAG) [26] , lie in their simplicity and reduced computational cost on the inference [6] . Fine-tuning results in a self-contained model, which simplifies the system architecture by removing the need for additional components like document retrieval and ranking during inference. This reduction in complexity leads to lower computational overhead, especially in terms of GPU memory usage due to the shorter length of the prompt, making fine-tuning more suitable for an LLM deployment in resource-constrained environments.However, it is important to check the performance gap between them. Therefore, we experiment with RAG on the Events 2024 dataset with Vicuna-7b. For ours, we follow the same experimental setting with Table 3 . For RAG, we use the bge-large-en-v1.5 [53] model for document and query embedding for retrieval. In Table 10 , our experimental results indicate that the RAG approach outperforms fine-tuning methods including ours, as previously observed by de Luis Balaguer et al. [6] . However, our LaPael method narrows the gap between the two approaches, suggesting that there is potential for further improvements in fine-tuning strategies. Reversal curse. The proposed method is unable to address the reversal curse, where the Large Language Models (LLMs) trained on "A is B" fail to answer "What is B?" [3] . As outlined in Berglund et al. [3] , this phenomenon is mainly due to the format of data and the autoregressive nature of LLMs that are trained in a way from left to right. Therefore, it is limited to improve the knowledge injection performance if the document does not contain a sentence having the reverse relationship, even with our method. Future work will need to explore the combining of our method with a recent solution for the reversal curse like reverse training [10] . Otherwise, we can seek a solution that addresses the reversal curse at the latent level similar to LaPael, which can be an interesting direction for future work.Limited scope of Task and Experiments. The scope of our method remains limited in the knowledge injection task. Specifically, there are challenges in applying LaPael for continual pre-training on large-scale corpora, such as the 15B OpenWebMath dataset [35] , or for instruction tuning with datasets like Alpaca [46] . Addressing these challenges will require future work as a new approach for training latent paraphrasers tailored to other tasks. In terms of experiments, our experiments only focus on the 7B LLMs, and do not conduct any experiment on larger LLMs of size with 13B or 70B [48] due to the limited computational budget for our experiments.This work explores the knowledge injection in Large Language Models (LLMs), which are highly related to hallucinations [15] . While our method improves the addition of new knowledge to LLMs, it also increases the risk of introducing misinformation. Specifically, our method could enhance the inaccuracies in LLMs when they are fine-tuned using documents that contain incorrect facts. Therefore, it is crucial to thoroughly check the documents used for fine-tuning LLMs before applying our method to enhance knowledge injection.C.1 Dataset As briefly mentioned in Section 5.1, we generate the synthetic document from each question-answer pair using GPT-4-turbo model [32] . To generate the documents from the question and answer pairs, we use the prompt in Table 13 . To generate diverse paraphrases from D train , we use the prompt [29] in Table 14 using GPT-3.5-turbo model. For cross-domain transfer experiments, we also use the subset of MedMCQA [34] and a synthetic NovelQA dataset based on the Les Misérables Wikipedia page, where we generate the synthetic document for each question. For MedMCQA [34] , we use the subset of the dataset where the domain of question corresponds to the anatomy.We summarize the statistics of the synthetic dataset used in our experiments in Table 11 . We also plot the distributions of token counts in documents, questions, and answers for each dataset used in our experiments in Figure 6 . We present the example of each dataset in Table 12 .As briefly mentioned in Section 5.1, we mainly use Vicuna-7b-v1.5 [56] for fine-tuning. We fine-tune LLMs for 12 epochs with a learning rate of 0.00005 and step learning rate scheduler where we decay a learning rate by 0.85 by every 4 epochs. For experiments in Figure 4 , we fine-tune for 3 epochs with a decaying period as 1 epoch. For optimizer, we use AdamW [28] . For all experiments, we only update the parameters corresponding to the MLP layer of transformer [50] . For Llama model [47, 48] , it corresponds to linear layers named up_proj, gate_proj, and down_proj. We use 4 A100 GPUs for fine-tuning LLMs. For inference, we use in-context learning with 5 examples by prompting the 5 examples in the prompt [4] .For training latent paraphrasers, we train them for 10 epochs with a learning rate of 1e -3 and cosine learning rate scheduler where we linearly decay a learning rate to 10% of the initial learning rate without warmup. We use 5 latent paraphrasers on the 5 sequential early layers of LLMs. For Equation ( 13), we use N = 4. For Equation ( 14), we use K = 10. For Equation ( 15), we set r = 0.5.For gold mask mt , we use a similar method to Agrawal et al. [2] to find the named entities from each document using GPT-3.5-turbo. For fine-tuning with latent paraphrases (Equation ( 17)), we use N = 4. (i) US Events in Table 3 Figure 6 : The distributions of token counts in documents, questions, and answers for each dataset used in our experiments.Table 13 : Prompt for Synthetic Document Generation. An 1-shot prompt for generating the synthetic document from the question.Write a concise informative background sentence, that is directly helpful to answer the following question. The background sentence is the sentence that ends with a suffix. In other words, the answer entity should be followed by the entities used in the question.### Question Question: Who replaced Tim Sloan as CEO of Wells Fargo? Answer: Charles Scharf ### Suffix Charles Scharf ### Background Sentence Tim Sloan was succeeded as CEO of Wells Fargo by Charles Scharf.### Question [question] ### Suffix [answer] ### Background SentenceTable 14 : Prompt for Paraphrasing. A 2-shot prompt for paraphrasing. y indicates the answer for the question and x denotes the remaining part of sentence, as introduced in Section 4.1.For the following prefix, give me 2 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not inclue numbering. Maintain the sentence structure.In infrainguinal bypass surgery, the preferred type of graft for optimal outcomes is an Autologous vein.In infrainguinal bypass surgery, the preferred type of graft for optimal outcomes is an # Suffix (PRESERVE AND KEEP LETTER CASE) Autologous vein. # Paraphrases (Prefix + Suffix) In infrainguinal bypass procedures, the graft type most recommended for the best results is an Autologous vein.During infrainguinal bypass operations, the optimal choice for a graft to achieve the best outcomes is an Autologous vein.For the following prefix, give me 2 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not include numbering. Maintain the sentence structure. # Sentence During the embryonic development of the gastrointestinal tract, proper rotation of the gut is necessary for the correct placement of the caecum; an abnormality in this process can lead to Mixed rotation. # Prefix During the embryonic development of the gastrointestinal tract, proper rotation of the gut is necessary for the correct placement of the caecum; an abnormality in this process can lead to # Suffix (PRESERVE AND KEEP LETTER CASE) Mixed rotation. # Paraphrases (Prefix + Suffix) In the formation of the gastrointestinal system during embryonic growth, it is essential for the gut to rotate correctly to ensure the caecum is properly positioned; deviations in this mechanism may result in Mixed rotation.Throughout the development of the gastrointestinal tract in the embryo, the accurate rotation of the gut is crucial for the appropriate localization of the caecum; any irregularities in this rotation can result in Mixed rotation.For the following prefix, give me 10 highly diverse paraphrases of the same in high-quality English language as in sentences on Wikipedia. Ensure that the suffix is followed by a paraphrased prefix. Do not include numbering. Maintain the sentence structure.# Sentence (x, y) # Prefix x # Suffix y # Paraphrases (Prefix + Suffix)Table 15 : Experimental results on datasets with synthetic documents from diverse LLMs. We present results from Llama2-7B [48] , Mistral-7B-Instruct-v0.2 [18] , and Phi3-mini-4k-instruct [1] . Verifying whether the proposed method can be transferred to other Language Models (LMs) is important. First, we validate our LaPael with Llama-2-7B [48] , a non-instruction-tuned version of the Vicuna-7B we used in experiments. In Table 15 , we present the experimental results with Llama-2-7B.The results show that our LaPael is effective even in the LM that is not instruction-tuned. In Table 15 , we also present the experimental results with Mistral-7B-Instruct-v0.2, which is an instruction-tuned model based on a different LLM Mistral-7B [18] . The results indicate that our LaPael is applicable not only to Llama-based models but also to LMs with different bases. Furthermore, in Table 15 , we present the experimental results with Phi3-mini-4k-instruction, which is a pre-trained LLM with 3.8 billion parameters [1] . The results indicate that our LaPael is highly effective when applied to the Phi3-mini model, which has fewer parameters than other LLMs.Parameter-efficient fine-tuning is a method that fine-tunes LLMs with minimal updates to their parameters. It is of interest that our LaPael can be effective even with parameter-efficient fine-tuning.LoRA [13] is a well-known method for parameter-efficient fine-tuning, which updates trainable rank decomposition matrices injected into the parameters of LLMs. In Table 16 , we present the experimental results with LoRA on Vicuna-7b-v1.5 where we update only the low-rank matrices of up_proj, gate_proj, and down_proj layers. The results demonstrate that LaPael is also effective in LoRA fine-tuning, highlighting its flexible applicability in diverse fine-tuning scenarios.In Figure 7 , we display the latent features from the final layers of large language models (LLMs) with and without latent paraphrases, where we reduce the dimension using t-SNE [49] . Crosses ('x') mark the embeddings from LLMs with latent paraphrasers. As illustrated in Figure 7 , latent paraphrasers enable the generation of diverse data samples, enhancing the diversity compared to data-level paraphrases. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: Our research conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).One possible way is to prompt the LLM (e.g., gpt-3.5-turbo[31]) with instruction "For the following paragraph give me a paraphrase of the same in high-quality English language as in sentences on Wikipedia"[29]randomly sampled data from ArchivalQA dataset. Each color denotes the different data, circles denote the original sentences, triangles denote the paraphrases, diamonds denote the questions, and crosses ('x') denote the original sentence with latent paraphrasing. 


Decision: The paper noticed that during knowledge injection in LLMs, data augmentation by paraphrasing is very important. And inspired by this, they proposed Latent Paraphrasing (LaPael) as a novel approach by introducing perturbations at the latent feature level, rather than relying solely on data-level paraphrasing. When injecting new knowledge into an LLM, LaPael first insert a learnable noisy mask before the MLP layers in Transformers and train the mask distribution on an existing paraphrase dataset. Then during the fine-tuning process, this mask distribution induces variability similar to paraphrased data but operates directly within the model's latent space. The paper presents strong experimental results for LaPael, supported by improvements over data augmentation in performance across various benchmarks, including question-answering tasks.

The reviewers have pointed out some concerns that should be addressed. A major concern is the lack of detailed analysis regarding the computational cost of training the latent paraphraser, which could potentially limit the practical applicability of the method. Furthermore, the paper would benefit from additional qualitative analysis to illustrate how the perturbations at the latent level differ from traditional paraphrasing at the data level. There are also questions regarding the generalizability of LaPael across different task, but not just question-answering.

However, the reviewers generally agree that LaPael offers a novel approach by targeting the latent feature space, which is a promising direction that could reduce dependency on external paraphrasing models and improve the knowledge injection process. The paper also provides a potentially new viewpoint of the training dynamics of LLMs, where at the end of training, the model needs some proper weight perturbation to learn new knowledge. Overall the Pros trumps the Cons, which lead to a recommendation to acceptance, but I still recommend authors to put significant efforts to resolve the concerns.

Reviews: review: paper_summary: paper_summary: This paper proposed a latent paraphraser to generate paraphrased data which will be used as augmented data in LLMs' fine-tuning.
To tackle the challenges of repetitive external model interventions, the latent paraphraser (LaPael) is trained to add a small perturbation at the latent feature level of the LLM and eliminate the need for users to repeatedly paraphrase using LLMs once latent paraphrasers are trained. Specially, with the paired-data as input, i,e., original and paraphrased sentence, the training objective is to minimise the KL divergence between distributions of original sentence after the paraphraser transformation and the distribution of the paraphrased sentence. The experimental results show that LaPael outperforms existing noise-based paraphraser in generating better augmentation data.
strength_weakness: strength_weakness: 1. The cost of training the latent paraphraser is not clear, as expensive training will impedes the efficiency of the proposed method.
2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.
questions: questions: 1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y.
2. Any explainations or case study showing the generated paraphase sentences from LaPael?
3. The computation cost comparsion with noise-based paraphraser.
limitations: limitations: The trade-off between task performance and computation cost.
paper_summary: paper_summary: This paper presents LaPael, a novel approach to injecting new knowledge into Large Language Models (LLMs). Previous works have shown that fine-tuning the model with data augmented by paraphrasing helps the model learn new knowledge. However, this requires high-quality paraphrased data each time new knowledge is injected, which is costly. The paper proposes a method that introduces a paraphraser layer within the LLMs, which acts as a latent paraphraser that paraphrases the input text in the latent space. The authors train the latent paraphraser using paraphrased data. Once the latent paraphraser is trained, it is used to fine-tune the LLM on the new knowledge. LaPael outperforms standard fine-tuning and paraphrasing approaches, showing improvements on question-answering benchmarks.
strength_weakness: strength_weakness: **The major weakness of the paper lies in the experiment section and the details provided regarding the experiment. Below are my concerns and questions related to this section:**

1. Line 190: "We use the subset of questions in the test set of each dataset for $D_{QA}$" -> Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper.

2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection. It would have been more suitable to use datasets with available documents, inject the knowledge from these documents, and then test the model on the QA. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM.

3. There are no statistics available for the created documents, such as their size. Creating small paragraphs from question-and-answer pairs, fine-tuning the model on these paragraphs, and then testing it on the same question-and-answer pairs does not constitute a correct and fair evaluation.

4. Line 212:  Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge?

5. Lines 184-185: It is not clear what the authors meant by "We sample N noise."
6. Line 182: "From Equation 10" should be "from Equation 15."

**The writing of the paper is also very weak. Here are a few examples:**

a. The acronym "LM" is used in the experiment section without being defined. The paper uses "LLM" for Large Language Models and sometimes "LM."

b. The paper uses "$D_{knowledge}$" and "$D_{K}$" interchangeably. Please ensure consistency.

c. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.
questions: questions: Please refer to the weaknesses section.
limitations: limitations: NA
paper_summary: paper_summary: This paper focusses on the ability of LLMs to learn new knowledge. Previous work has shown that paraphrasing techniques help learning this new knowledge. However, as the authors argue, explicit paraphrasing has a high computational cost, and the paraphrased data is of limited diversity. To circumvent this, this paper introduces latent paraphrasing, by adding a latent paraphrasing layer in the standard transformer module, in between the LayerNorm and the MLP. This layer is trained with paraphrased data, generated by an LLM. The optimization objective is to minimize the KL-divergence between the output of a “standard” transformer model with the paraphrases as input, and the output of a transformer including the paraphrasing layer with the original sentences as input. 

The authors test their approach, called LaPael, on three datasets: SQuAD, StreamingQA, and ArchivalQA, and compare against multiple fine-tuning baselines. They show that LaPael mostly outperforms the baselines, even in cross-domain transfer experiments.
strength_weakness: strength_weakness: * The objective of the paper is to explicitly focus on fine-tuning approaches, instead of approaches that rely on external memory. In that light, it makes sense to only compare against fine-tuning baselines. However, if one is just interested in what method performs best on the task, comparing against external memory based approaches would have been a good addition to the results section.
* As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection. For example, SQuAD is based on Wikipedia, which the models have probably seen during pretraining. Also see my related question below.
* I also have a few other questions about the approach, which I have added in the question section below.
questions: questions: * The authors mention that they use Vicuna as the base model. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM?

Some clarifying questions:

* Line 177: regarding the gold mask for tokens that correspond to the named entity. How is determined what the named entities are in the sentence? Do the authors deploy some sort of NER tagger?
* Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”?
* Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K?
* Line 238: regarding “Combining LaPael and Paraphrases”. I understand the objective of this experiment, and I agree with the authors that it is insightful. I am not entirely sure if I understand the experimental setup here. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning?
* Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?
limitations: limitations: The limitations are addressed in the appendix.
paper_summary: paper_summary: The paper proposes a new method named LaPael to enhance knowledge injection for large language models. Different from traditional data-level augmentations or noise-based methods, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. Experimental results on three question-answering datasets show that the proposed method outperforms the baselines.
strength_weakness: strength_weakness: 1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly.

2. Why is the output distribution of Transformer assumed to be Gaussian in Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method.

3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.
questions: questions: 1. In Table 7, why was it not compared with the Fine Tuning (+para) method?

2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?
limitations: limitations: Please see above Weaknesses and Questions.

Comments: comment: We do appreciate your valuable suggestions on our work and will gladly continue improving our paper. Thank the reviewer once again for your time and effort in helping us enhance our work.
Thank you! I think it would be valuable to add some additional discussion around this topic in the paper, as it clarifies the motivation for your approach.
Thank the reviewer for clarifying the questions. 

> Q1: I am asking for the computation cost comparison between noise-based and the proposed method, as the author mentioned "We have clearly stated the cost of training the latent paraphraser compared to noise-based fine-tuning baselines in the revised manuscript.", But I couldn't find it. 

We apologize for any confusion. Regarding the revised manuscript, we have consistently addressed all comments in the revised version of the manuscript, which unfortunately we cannot upload at this moment. 

Here, we provide the computational cost comparison between noise-based baselines and the proposed method in terms of per-step FLOPs for training:

### Noise-based baselines: 
* Fine-tuning LLM: $3F$ FLOPs 
### The proposed method: 
* Fine-tuning LLM: $3.036F$ FLOPs 
* Additional training for latent pararphasers: $1.108F$ FLOPs. 

We believe that our method offers a viable option to **enhance the effectiveness of knowledge injection** into LLMs, despite requiring some initial training cost. This cost leads to superior results compared to noise-based baselines. 

> Q2: Thanks for your results and the acknowledgment of "this degradation of math reasoning performance". I keep my original score mostly because the inefficiency in keeping the (other) basic reasoning capability, e.g., math of the LLMs after inserting such paraphrase layer, as one of the important principle of knowledge injecting is not to destroy the existing knowledge capacity. 

Thank the reviewer for sharing valuable thoughts. 
While we understand the concern regarding fine-tuning and catastrophic forgetting, we would like to emphasize that **this is a common challenge across all fine-tuning methods, not just our approach**, as evidenced by the previous work [1].
Our method does not significantly exacerbate this issue while improving the knowledge injection performance. From this perspective, we believe our method offers a valuable contribution.

We are very grateful for the opportunity to discuss these points with the reviewer. The reviewer's engagement in this discussion is highly valuable to us. Please let us know before the discussion period ends if there are any further concerns or questions. 

**Reference**

[1] Jang et al., Towards Continual Knowledge Learning of Language Models
Thank the reviewer for engaging in this discussion. We are happy to share our response to the question below:

> I'm wondering about the computational cost though, because you also need to train and use the latent paraphraser. I'm assuming the computational cost in the end boils down to certain architectural choices you make. Or would you say LaPael is always computationally more cost efficient than a RAG type of method?

To clarify, **our comparison focused on the computational cost during inference, not training.** In our previous response, we emphasized that fine-tuning methods are more cost-efficient than RAG-based methods in terms of inference cost. 

When considering the computational cost for training, fine-tuning methods including LaPael do indeed incur additional costs, particularly due to the fine-tuning of LLMs for knowledge injection. Moreover, LaPael requires further costs for training and utilizing the latent paraphraser, as we mentioned in our responses to Reviewer Zuau (`1-2`) and Reviewer (`4-3`). 

We hope our answer clarifies the inquiry. If there are any further questions, please let us know.
We would like to thank the reviewer for their engagement during this phase. We hope we have addressed the issues raised. It would be very helpful if the reviewer could provide any further concerns or suggestions that remain.
Thank the reviewer once again for your time and effort in this process.
Thanks to the authors for the detailed responses. After reading the other reviews and the rebuttals, I maintain the original rating.
Q1: I am asking for the **computation cost comparison** between noise-based and the proposed method, as the author mentioned "We have clearly stated the cost of training the latent paraphraser compared to noise-based fine-tuning baselines in the revised manuscript.", But I couldn't find it. 

Q2: Thanks for your results and the acknowledgment of "**this degradation of math reasoning performance**".

I keep my original score mostly because the inefficiency in keeping the (other) basic reasoning capability, e.g., math of the LLMs after inserting such paraphrase layer, as one of the important principle of knowledge injecting is not to destroy the existing knowledge capacity.
Thanks for your elaboration. I appreciate your work for trying a new approach. I'm wondering about the computational cost though, because you also need to train and use the latent paraphraser. I'm assuming the computational cost in the end boils down to certain architectural choices you make. Or would you say LaPael is always computationally more cost efficient than a RAG type of method?
Thank you to the authors for their rebuttal. I have read the authors' responses and the comments from other reviewers.
Thank you for reviewing our response and providing additional comments. We greatly appreciate your feedback and will address your questions below.

---
> Can you give a comparison here between noise-based paraphraser (such as FreeLB) and Lapael?

Here is the comparison between two noise-based baselines and Lapael. The noise-based baselines, NEFTune and FreeLB, do not require a preliminary step prior to fine-tuning LLMs:

* **NEFTune:** During fine-tuning, NEFTune adds random noise to token embeddings during fine-tuning to improve model robustness. [1]  
* **FreeLB:** During fine-tuning, FreeLB introduces adversarial perturbations to token embeddings, aiming to minimize adversarial risk by optimizing the model's performances across various perturbations. [2]  
* **LaPael (ours):** Uses an *input-dependent noise generator* (latent paraphrasers) that applies noise to latent features, learning the noise distribution directly from given paraphrases. LaPael optimizes its latent pararphrasers by aligning the output distributions between the perturbed model with the original sentence and the model with paraphrased sentences, as detailed in Section 4.2 of the main paper. This approach enables LaPael to generate noise that is more contextually appropriate and effective. 

We would like to emphasize that while LaPael requires an additional step of training latent paraphrasers before fine-tuning the LLM, this overhead becomes negligible (previous response `1-2`) and significant performance gains can be achieved compared to the baselines (Tables 2 and 3). We have added this detailed comparison in addition to the content on Lines 105-107 of the main manuscript.

---
> I am curious, after adding such a latent Paraphraser layer, what are the inference performance changes on Unrelated but Basic Reasoning datasets? such as math reasoning, code generation. As the implementation is expected to not degrader the basic model capacity.

Thank you for the insightful question. Fine-tuning LLMs for knowledge injection can indeed degrade the model's capacity due to *catastrophic forgetting* induced by the fine-tuning process. To illustrate this, we provide the math reasoning performance of the LLM (fine-tuned on the synthetic documents of the SQuAD dataset) on the GSM8K [3] test set.

* Performance on GSM8K test set after knowledge injection

| GSM8K    | Accuracy |  
| -------- | -------- |  
| No adapt.| 23.05    |  
| Fine-Tuning| 18.88  |  
| Ours (Fine-Tuning with LaPael) | 18.04  |

The results show that fine-tuning itself largely contributes to the degradation of math reasoning performance. As discussed in our response to Reviewer NFcZ (`4-5`), maintaining reasoning performance after fine-tuning is beyond the scope of our work, which primarily focuses on improving knowledge injection. Nonetheless, we acknowledge the importance of addressing this problem and its potential to guide future research. We have included this point and the associated results in the limitations section of our revised manuscript.

---
We hope that our response sufficiently addresses your questions. Should you have any further inquiries or comments, please don’t hesitate to let us know so that we can resolve them. If you find that most of your concerns have been addressed, we kindly ask that you consider adjusting the score accordingly. Thank you so much for your effort and time in reviewing our work.

**Reference**

[1] Jain et al., NEFTune: Noisy Embeddings Improve Instruction Fine-tuning.

[2] Zhu et al., FreeLB: Enhanced Adversarial Training for Natural Language Understanding.

[3] Cobbe et al., Training Verifiers to Solve Math Word Problems
Thanks very much for your detailed clarification!

Follow up questions:
(1) Can you give a comparison here between noise-based paraphraser (such as FreeLB) and LaPael? 
(2) I am curious, after adding such a latent Paraphraser layer, what are the inference performances changes on Unrelated but Basic Reasoning datasets?  such as math reasoning, code generation. As the inplementation is expected to not degrader the basic model capacity.
Thank the reviewer for your reply and follow-up question. We are happy to hear that your concerns are largely addressed by our response.

Regarding your follow-up question, the main advantage of fine-tuning approaches including ours, over retrieval-based approaches such as RAG lies in simplicity, inference efficiency, and computational cost. Fine-tuning approaches result in a self-contained model that *simplifies* the overall system architecture by eliminating the need for additional infrastructure for document retrieval and ranking during inference. This approach *avoids the extra computational costs* associated with query embedding and extended prompts with retrieved documents, making it more *cost-efficient*, especially in terms of GPU memory, and suitable for deployment in resource-constrained environments.

Due to the complexity-performance trade-off, the choice between two approaches depends on the deployment environment. Fine-tuning approaches are specifically suited for settings that require low-latency and resource-efficient deployment. We hope this explanation clarifies your question. If you have any further questions, please let us know.
Thank you for your response, and clarifications. They largely address my concerns, and I appreciate the additional results for the two new datasets.

I have one follow-up question about the results for the new RAG baseline. First of all, thank you for running these experiments, and including the numbers in the rebuttal, and for updating the manuscript with these numbers. Currently, the RAG numbers are quite far from the LaPael numbers. What do you think is the main advantage of using a method like LaPael over the retrieval based method?