FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models (2024)

Abstract: August 11-16, 2024 ©2024 Association for Computational Linguistics FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models Kaixin Lan* Tao Fang1∗ Derek F. Wong1† Yabo Xu Lidia S. Chao Cecilia Guanfang Zhao NLPCT Lab, Department of Computer and Information Science, University of Macau nlp2ct.{Kaixin,taofang}@gmail.com, {derekfw,lidiasc}@um.edu.mo Guangdong Hengqin DataStory Information Technology Ltd. arber@datastory.com.cn Department of English, Faculty of Arts and Humanities, University of Macau czhao@um.edu.mo Abstract

heading: 1 Introduction
text: Pre-trained language models (PLMs) have gained widespread recognition for their unparalleled performance in numerous downstream natural language processing (NLP) tasks (Clinchant et al., 2019; Li et al., 2022; Fang et al., 2023a,b,c; Zhang et al., 2023; Pang et al., 2024), especially in text generation (Hua and Wang, 2020; Zhang et al.,
* Equal Contribution † Corresponding Author
2020; Guan et al., 2021; Wang et al., 2024). With the emergence of advanced PLMs, there is an intensifying debate over the distinctiveness of texts produced by these models as opposed to those written by humans, a sentiment highlighted by (McCoy et al., 2023). Within this landscape, generative PLMs primarily fall into two architectural paradigms. On the one hand, BART (Lewis et al., 2020), T5 (Raffel et al., 2020), and other such pretrained language models (Song et al., 2019) serve as quintessential representatives of the encoderdecoder approach. On the other hand, OpenAI’s GPT (Generative pre-trained Transformer) series (Radford et al., 2019; Brown et al., 2020; Ouyang et al., 2022) and Meta AI’s LLaMA series pretrained language models (Touvron et al., 2023a,b), leveraging a decoder-only design, has carved out a unique niche for itself. Both these architectural categories of pre-trained language models have exhibited impressive capabilities in natural language understanding (Ebrahimi et al., 2022), and natural language generation (Rothe et al., 2021; Jiao et al., 2023), solidifying their reputation as premier commercial offerings. Because of their efficiency and adaptability (Zhan et al., 2024), these models have been widely adopted across diverse sectors, including writing, and academic research. Nonetheless, the sophistication these models showcase in text generation has ignited concerns regarding academic integrity, prompting many educational establishments to restrict their utilization in scholarly activities.
We believe that the aforementioned issues mainly occur because the content generated by language models often lacks originality, as their outputs heavily rely on their training data. This might lead the models to replicate or mimic the information or patterns they encountered during training. In areas where originality is highly valued, such as academic writing or story generation, the outputs from these models can sometimes be seen as pla-
14432
giaristic. In fact, earlier research has already highlighted the potential risks of intentional or unintentional leakage of sensitive information within the training sets of language models (Zanella-Béguelin et al., 2020; Carlini et al., 2021; Brown et al., 2022). This concern persists even during the fine-tuning phases, as evidenced by Mireshghallah et al. (2022). To further investigate the potential lack of originality in PLMs outputs, we conduct fine-tuning on two prominent generative PLMs, LLaMA2 and GPT2 pre-trained language models, using an academic paper AASC1 dataset from the NLP domain. Testing their outputs with Turnitin2 revealed significant instances where the models reproduced segments from the training set, as illustrated in Figure 1.
Currently, a considerable portion of research is dedicated to determining whether the outputs from PLMs display plagiarism or retain their originality (Ferrero et al., 2017; Wahle et al., 2022; Lee et al., 2023; Wu et al., 2024). However, there is a noticeable lack of research focused directly on the innate originality of content generated by these PLMs. To address this void, we introduce a novel generation approach named the "self-plagiarism" (SP) contrastive decoding strategy, aimed at bolstering the inherent originality of text generated by PLMs. This strategy builds upon the principles delineated by Schick et al. (2021) and Chuang et al. (2023), yet diverges from their methodologies. Initially, our approach shifts its focus from mitigating model bias at the token level to accentuating origi-
1https://github.com/KMCS-NII/AASC 2https://www.turnitin.com/
nality at the paragraph level. A distinctive feature of our method lies in the strategic emphasis on the topmost layer, enriched with high-level knowledge. Coupled with the extension of generation length, our methodology encompasses a broader spectrum of content uniqueness. This proves particularly advantageous in domains like storytelling and academic writing, where the nuances of high-level knowledge hold significant importance.
Secondly, we achieve an amateur model and a professional model by adjusting the prompts of the language models. For the amateur model, we innovatively introduce three prompts, which originate from the three most common research categories in plagiarized literature: verbatim plagiarism, paraphrase plagiarism, and idea plagiarism. The purpose of these prompts is to guide the model to replicate the training data according to specific plagiarism standards. On the other hand, the professional model uses conventional prompts to encourage the model to generate normal text. On this basis, we subtract the probability distribution of the last layer of the two models and impose penalties on tokens that show a higher probability in a regulatory factor function, ensuring a balance between following the prompts and maintaining originality. Finally, we use our method on various language models, including T5, GPT-2, LLaMA1, and LLaMA2, on the academic dataset AASC and ROCStories dataset. The results show a significant reduction in non-original sequences of more than three words generated by PLMs.
Our primary contributions are as follows:
• We reconfirm that even during the fine-tuning phase, pre-trained language models still manifest tendencies of plagiarism. This propensity for un-originality is particularly evident in the domain of academic writing.
• We innovatively introduce “self-plagiarism” contrastive decoding strategy by adjusting the prompts of the language models to achieve an amateur model and professional model, and subsequently penalizes the plagiarized tokens. This approach significantly reduces the plagiarism rate and enhances originality.
• We showcase the efficacy of our approach in augmenting the originality of the content produced by the models on the widely-used academic writing AASC dataset and ROCStories
dataset, providing valuable guidelines for subsequent text generation endeavors.

heading: 2 Methodology

heading: 2.1 Definition of Language Modeling
text: Pre-trained autoregressive language model pθ(y|x) is usually built based on the Transformer framework and parametrized by θ. The pre-trained model computes hi as a function of zi and the past activations in its left context:
hi = PLMθ(zi), (1)
where hi is the last layer which is used to compute the distribution for the next token:
pθ(zi+1|h≤i) = Softmax(Wθhi), (2)
where Wθ is the pre-trained parameters matrix. During the finetuning stage, we use the pretrained parameters θ to initialize the model where pθ is a trainable language model distribution. The finetuning performs gradient updates on the loglikelihood objective:
arg θ
max p(y|x; θ) = ∑ log pθ(zi|h<i), (3)

heading: 2.2 Plagiarism and Self-plagiarism Prompts
text: In the academic domain, plagiarism involves using someone else’s work, ideas, or expressions and presenting them as one’s own without proper acknowledgment. However, our work focuses on addressing plagiarism in the context of large models—specifically, reducing the straightforward replication of training data by these models. We refer to this as enhancing model’s originality.
Inspired by Lee et al. (2023), they introduce three most commonly studied categories in plagiarism literature: Verbatim, Paraphrase, and Idea plagiarism. They evaluate both pre-trained and fine-tuned models’ plagiarism tendencies in these categories. Their findings suggest language models do not just mimic training samples but can also rephrase or borrow ideas from original texts. Informed by their insights, we craft Self-plagiarizing prompts, tailor to guide the model in replicating training data based on the designated plagiarism criteria. Figure 2 illustrates our specific prompt templates.
Formally, let the input text be represented as X . Our self-plagiarizing templates encompass text components denoted by P , which can manifest in one or more of the ways listed below:
• Exact copies of words or phrases without transformation.
• Synonymous substitution, word reordering, and back translation.
• Reuse of the core idea by shortening or summarizing the original content.

heading: 2.3 Amateur LM and Expert LM
text: Upon finalizing the design of the plagiarism prompting template, our primary objective is to construct a robustly flawed amateur model, alongside a proficient expert model. In particular, we denote p(Y |X) as the original predictive probability distribution of the input X, and p(Y |sp(X,P )) signifies the probability of the subsequent word given input X in self-plagiarizing prompt templates. This selfplagiarizing input urges the language model to manifest plagiarized behavior, laying the foundation for both the amateur and expert models we have developed. The amateur model is established based on p(Y |X), and it emulates self-plagiarizing behavior to formulate a predictive probability distribution pAMA(Y |sp(X,P )) for input X. Conversely, we provide default system prompts for the expert model, or no prompt at all for models that do not support system prompts. The expert model draws insights from the original predictive probability distribution pEXP (Y |X) to deliver more precise predictions on input X.

heading: 2.4 Contrastive Decoding
text: Figure 3 displays the detailed framework of our methodology. The created amateur model tends
to produce plagiarized words or fragments more than the expert model. When both the expert model and the amateur model assign a higher probability score to a "repeating token", the expert model is more likely to also assign high scores to other good tokens with low repetitiveness (Li et al., 2023). However, the amateur plagiarism model does not behave this way, which means that it is more prone to be influenced by plagiarism prompts.
To make the text generated more original, it is necessary to resist the occurrence of plagiarism. Therefore, we propose a contrastive objective function:
∆(Y,X, P ) = pEXP (Y |X)−pAMA(Y |sp(X,P )), (4)
In cases of multiple prompts used simultaneously, we keep the largest difference.
∆(Y,X, P ) = min P ∆(Y,X, P ), (5)
A plagiarized sentence is more likely to receive a higher probability from pAMA(Y |sp(X,P )) than pEXP (Y |X). Hence, for those exact copies of words or phrases, ∆(Y,X, P ) will be lower than zero. For those original expressions, ∆(Y,X, P ) will be greater than zero.
However, it’s not necessary to dismiss all results produced by amateur models. To address this, we follow Schick et al. (2021) to design a regulatory
factor, denoted as α, a scale function is used to scale those differences to a number between 0 to 1.
α(x) =
{ 1, x > 0
eλ·x, otherwise (6)
where λ is a hyperparameter, x is contrastive objective. For those original expressions (∆(Y,X, P ) > 0), their probability will be kept the same, while for those exact copies of words or phrases , eλ·∆(Y,X,P ) will be lower than 1. The result will be used to adjust the original predict the probability distribution p(Y |X). Finally, model will generate output based on p̃(Y |X):
p̃(Y |X) ∝ α(∆(Y,X, P )) · p(Y |X). (7)

heading: 3 Experiments
text: Considering the vast amount of training data used for pre-trained models and the lack of transparency regarding the specific datasets utilized, it is challenging to detect plagiarize on models that are either not fine-tuned or proprietary. To effectively address the issue, we opt to fine-tune open-source models on publicly available datasets and subsequently assess their plagiarism on these fine-tuned datasets. This strategy allows us to rigorously validate our method and its efficacy in detecting and mitigating plagiarism in PLMs.

heading: 3.1 Dataset
text: We utilize the ROCStories training dataset (Mostafazadeh et al., 2016) and the ACL Anthology
Sentence Corpus (AASC)3 during the fine-tuning phase of the language models. For the ROCStories dataset, we strictly follow the partition outlined in (Mostafazadeh et al., 2016), dividing it into training, testing, and development sets. The AASC dataset constitutes a curated compilation of text excerpts extracted from scientific papers in the field of natural language processing. Drawn from PDFformat papers published within the ACL Anthology between 2000 and 2018, each paper is segmented into individual sentences, categorized according to their respective originating sections. To construct our training set, we chose sections including Abstract, Introduction, Background, Method, Result, and Discussion. For evaluation and testing, a chosen 1% of sentences were randomly extracted from the corpus. A detailed data statistics can be found in Table 1.

heading: 3.2 Model and Training
text: With due consideration for performance optimization and the judicious allocation of computational resources, our methodological framework revolves around the utilization of LLaMA-7b series pretrained language models (Touvron et al., 2023a,b), GPT-2 large (Radford et al., 2019) and T5 large (Raffel et al., 2020) as baseline models. Our training process leverages the computational power of four A100 or V100 GPUs. Both GPT-2 and T5 models undergo fine-tuning over a span of five epochs, employing two distinct datasets. Because of limitation of computing resource, LLaMA series are fine-tuned over 3 epochs with Alpaca-LoRA.4 The foundational pre-trained models and training scripts are sourced from the Huggingface repository.5 Appendix A presents a detailed account of the hyper-parameters used in the training process.

heading: 3.3 Evaluation
text: We employ three distinct evaluation ways: Generation Originality Test (GOT) (Brooks and Youssef, 2021), Turnitin , and human evaluation. GOT is
3https://github.com/KMCS-NII/AASC 4https://github.com/tloen/alpaca-lora 5https://github.com/huggingface/transformers
an n-gram automated test for assessing originality. It constructs an original set by extracting unique n-gram fragments from the training set and subsequently examines whether the fragments generated in the test output are contained within this original set (see Appendix B). We apply GOT to all models on both datasets. To further evaluate if our approach effectively mitigates plagiarism in academic writing and enhances the model’s originality, we utilize Turnitin, a widely used academic plagiarism detection software, specifically on the AASC academic dataset. For human evaluation, we enlist the feedback of two volunteers to assess the impact of our method on the coherence (the logical connection and content association between sentences) and fluency (grammar error, naturalness, and writing style) of the model’s output. Appendix C presents a detailed definition of coherence and fluency, as well as evaluation template and examples.

heading: 4 Results

heading: 4.1 Evaluation on GOT
text: Figures 4 illustrate the comparative analysis of output dissimilarity between the four pivotal models, evaluated using the GOT metric. Appendix D presents illustrative instances of input and output. In terms of overarching trends, a discernible pattern emerges: the resemblance between modelgenerated texts diminishes in opposite proportion to the length of the identified segments. The occurrence of text displaying similarity, spanning more than seven consecutive words, becomes exceedingly rare. Moreover, the similarity reduction due to the SP contrastive decoding strategy decreases with increasing segment length. All of the four models show a lower similarity rate on AASC than on ROCStories. We think the reason lies on ROCStories has a more common used vocabulary, while the inherent limitations of the SP contrastive decoding strategy inadvertently constrain the range of potential candidates for model predictions, prompting a selection bias towards words that are more prone to detection by the GOT algorithm.
Within the context of the dataset evaluated through ROCStories analysis, LLaMA series show an overall lower similarity, indicates a better originality of large model. GPT-2 large consistently exhibits a reduction of over one percent in similarity across all segment lengths up to five words. In contrast, T5 large deviates from this trajectory,
Fluency-1 3 4 5 6 7 8 default 9.26% 4.65% 1.36% 0.36% 0.09% 0.03% Our Method 9.00% 3.64% 0.94% 0.26% 0.06% 0.01%
Fluency-1-1 3 4 5 6 7 8 default 16.72% 10.11% 4.92% 2.16% 0.89% 0.03% Our Method 15.61% 8.85% 3.77% 1.51% 0.59% 0.01%
Fluency-1-1-1
3
default 19.80% Our Method 18.02%
default Our Method
exhibiting less similarity decreases when fragment length is higher than five, but maitain a lower similarity rate at the same time, which means T5 takes an advantage on similarity when generating a long sentence, also makes a discount on the performance of SP contrastive decoding strategy.

heading: 4.2 Evaluation by Turnitin Check on AASC test set
text: We collate the output texts generated by LLaMA2, GPT-2 and T5 on the AASC dataset. Subsequently, these generated texts are systematically uploaded onto the Turnitin platform to undergo rigorous originality assessments. Finally, we analyze the outcomes of these Turnitin evaluations. Table 2 showcases the assessment outcomes rendered by Turnitin. For all of the three fine-tuned pre-trained models, our proposed method yields a at least 3%
decrease in similarity rates. This empirical validation through Turnitin lends robust support to the practical efficacy of our proposed SP contrastive decoding strategy, confirming its effectiveness within real-world applications.

heading: 4.3 Human Evaluation
text: We average the scores of the two volunteers for the model evaluation. The results, outlined in Table
14437
Coherence Fluency
default SPCD default SPCD
A_T5 83.5% 88.5% 87.3% 87.2% A_LLaMA2 86.5% 88.0% 82.0% 83.0% R_GPT2 76.8% 76.8% 74.5% 75.3% R_LLaMA2 92.3% 87.5% 87.5% 85.3%
Table 3: Human evaluation of coherence and fluency on LLaMA2, GPT2-Large and T5-Large. The notation “A_model” refers to the fine-tuned model using the AASC dataset, while “R_model” denotes the fine-tuned model using the ROCStories dataset, SPCD means SP contrastive decoding strategy.
3, demonstrate that, in the majority of cases, our method yields comparable results in coherence and fluency. Notably, it even exhibits higher coherence on the AASC dataset. We attribute this to the robustness of pre-trained language models (PLMs), which can generate coherent and fluent text even when certain tokens are penalized. This suggests that our method has minimal negative effects on the coherence and fluency of the model’s output, successfully preserving the natural flow and understandability of the generated content.

heading: 5 Ablation study

heading: 5.1 Impact of Self-plagiarism Prompts
text: To independently verify the impact of selfplagiarizing prompts, we design ablation experiments using LLaMA2 with both default system prompts and self-plagiarizing prompts, specifically:
• The following text contains exact copies of words or phrases without transformation of language:
We then compare the similarity of the output results. The findings, depicted in Figure 5, indicate that the similarity of the outputs generated with selfplagiarizing prompts is higher across all fragment lengths compared to those generated with default prompts. This demonstrates the effectiveness of the contrastive decoding strategy.

heading: 5.2 Impact of Varying Prompt Numbers
text: To comprehensively comprehend the influence of prompt quantity on the SP contrastive decoding strategy’s efficacy, we conduct a meticulous evaluation using GPT-2 large PLM on the ROCStories dataset, modulating the number of prompts as the
Fluency-1 3 4 5 6 7 8 default 9.26% 4.65% 1.36% 0.36% 0.09% 0.03% Our Method 9.00% 3.64% 0.94% 0.26% 0.06% 0.01% Si m ila ri ty 0.0% 3.7% 7.3% 11.0% 3 4 5 6 7 0.06%0.26% 0.94% 3.64% 9% 0.09%0.36% 1.36% 4.65% 9.26% Default Our Method (a) GPT-2 large on AASC Dataset
Fluency-1-1 3 4 5 6 7 8 default 16.72% 10.11% 4.92% 2.16% 0.89% 0.03% Our Method 15.61% 8.85% 3.77% 1.51% 0.59% 0.01% Si m ila ri ty 0.0% 6.3% 12.7% 19.0% 3 4 5 6 7 0.59% 1.51% 3.77% 8.85% 15.61% 0.89% 2.16% 4.92% 10.11% 16.72% (b) GPT-2 large on ROCStories Dataset Fluency-1-2 3 4 5 6 7 8 default 10.32% 5.47% 1.97% 0.66% 0.27% 0.03% Our Method 10.14% 4.36% 1.23% 0.35% 0.11% 0.01% Si m ila ri ty 0.0% 4.0% 8.0% 12.0% 3 4 5 6 7 0.11%0.35% 1.23% 4.36% 10.14% 0.27%0.66% 1.97% 5.47% 10.32% (c) T5 large on AASC Dataset Fluency-1-1-1 3 4 5 6 7 8 default 19.80% 10.65% 4.26% 1.56% 0.40% 0.03% Our Method 18.02% 9.15% 3.45% 1.13% 0.31% 0.01% Si m ila ri ty 0.0% 7.3% 14.7% 22.0% 3 4 5 6 7 0.31%1.13% 3.45% 9.15% 18.02% 0.40% 1.56% 4.26% 10.65% 19.80% (d) T5 large on ROCStories Dataset Fluency-1-3 3 4 5 6 7 8 default 16.72% 10.11% 4.92% 2.16% 0.89% 1 prompt 15.41% 8.86% 4.06% 1.60% 0.65% 2 prompts 15.25% 8.79% 4.00% 1.66% 0.70% 3 prompts 15.61% 8.85% 3.77% 1.51% 0.59% Si m ila ri ty 0.0% 6.0% 12.0% 18.0% 3 4 5 6 7 Default 3 prompts 2 prompts 1 prompt
GPT-2 large on ROC Dataset
Fluency-1-3-2
3 4 5 6 7
default 8.76% 4.25% 1.36% 0.39% 0.18% 1 prompt 9.80% 5.27% 2.07% 1.14% 0.74%
Si m
ila ri
ty
0.0%
4.0%
8.0%
12.0%
3 4 5 6 7
Default SP prompt
Fragment Length
Figure 5: Impact of Self-plagiarism Prompts. When adding only self-plagiarism prompt, the similarity score consistently increase.
Fluency-1 3 4 5 6 7 8 default 9.26% 4.65% 1.36% 0.36% 0.09% 0.03% Our Method 9.00% 3.64% 0.94% 0.26% 0.06% 0.01% Si m ila ri ty 0.0% 3.7% 7.3% 11.0% 3 4 5 6 7 0.06%0.26% 0.94% 3.64% 9% 0.09%0.36% 1.36% 4.65% 9.26% Default Our Method (a) GPT-2 large on AASC Dataset
Fluency-1-1 3 4 5 6 7 8 default 16.72% 10.11% 4.92% 2.16% 0.89% 0.03% Our Method 15.61% 8.85% 3.77% 1.51% 0.59% 0.01% Si m ila ri ty 0.0% 6.3% 12.7% 19.0% 3 4 5 6 7 0.59% 1.51% 3.77% 8.85% 15.61% 0.89% 2.16% 4.92% 10.11% 16.72% (b) GPT-2 large on ROC Dataset Fluency-1-2 3 4 5 6 7 8 default 10.32% 5.47% 1.97% 0.66% 0.27% 0.03% Our Method 10.14% 4.36% 1.23% 0.35% 0.11% 0.01% Si m ila ri ty 0.0% 4.0% 8.0% 12.0% 3 4 5 6 7 0.11%0.35% 1.23% 4.36% 10.14% 0.27%0.66% 1.97% 5.47% 10.32% (c)T5 large on AASC Dataset Fluency-1-1-1 3 4 5 6 7 8 default 19.80% 10.65% 4.26% 1.56% 0.40% 0.03% Our Method 18.02% 9.15% 3.45% 1.13% 0.31% 0.01% Si m ila ri ty 0.0% 7.3% 14.7% 22.0% 3 4 5 6 7 0.31%1.13% 3.45% 9.15% 18.02% 0.40% 1.56% 4.26% 10.65% 19.80% (d) T5 large on ROC Dataset
Fluency-1-3
3 4 5 6 7 8
default 16.72% 10.11% 4.92% 2.16% 0.89% 1 prompt 15.41% 8.86% 4.06% 1.60% 0.65% 2 prompts 15.25% 8.79% 4.00% 1.66% 0.70% 3 prompts 15.61% 8.85% 3.77% 1.51% 0.59%
1
independent variable. As depicted in Figure 6, our findings reveal a consistent trend: the incorporation of the SP contrastive decoding strategy invariably leads to a reduction in the similarity of the generated texts across all test instances. This demonstrates that our approach enhances the originality of content produced by PLMs.
Upon augmenting the prompts to three, a slight augmentation in performance arrives. This enhancement emerges when the fragment length exceeds three words. This empirical observation underscores the pivotal role that sufficiently comprehensive prompts play in adeptly guiding the model’s generation process. This nexus between prompt comprehensiveness and algorithmic efficacy underscores that the SP contrastive decoding strategy can achieve heightened performance levels with a more elaborate array of prompts. This augmentation manifests as a reduction in similarity across an array of test cases, reinforcing the integral role of robust and all-encompassing prompts in enhancing the overall efficacy of the SP contrastive decoding strategy.

heading: 5.3 Impact of Prompt Template
text: We evaluate the performance of LLaMA2 on the AASC dataset using various prompts, as depicted in Figure 7. In the "Name Only" template, we streamline the detailed definitions of the three types of
Fluency-1-3-1
3 4 5 6 7 8
default 10.54% 6.56% 2.94% 1.49% 1.03% Names Only-multi 8.34% 3.64% 1.23% 0.63% 0.46% Detail Difinition 8.38% 4.13% 1.49% 0.62% 0.44%
multi-names 8.59% 4.42% 1.43% 0.58% 0.36%Si m
ila ri
ty
0.0%
4.0%
8.0%
12.0%
3 4 5 6 7
Default Names Only Detail Definition multi
Fragment Length
Figure 7: Impact of Prompt Template. A more specific prompt, along with a higher quantity of prompts, contributes to a more significant improvement in performance.
plagiarism to include only their names, while the "Detail Definition" template retains the same format as in the previous experiment. Importantly, the multi-prompt condition, which includes both templates, outperformes other control groups. These findings highlight that a more specific prompt and a higher quantity of prompts contribute to a more substantial improvement in performance.

heading: 6 Related Work

heading: 6.1 Plagiarism and Memorization in PLMs
text: Numerous studies have consistently shown that pre-trained language models have a tendency to memorize and plagiarize content from their training data. The study by Brown et al. (2022) underscores the potential risk of intentional or unintentional disclosure of sensitive information from a model’s training set. Pioneering research, such as that by Zanella-Béguelin et al. (2020) and Carlini et al. (2021), has revealed the extensive ability of large-scale models to internalize training samples during pre-training, making them especially susceptible to membership inference and data extraction attacks. Importantly, a significant portion of the training datasets used for language model training are culled from the Internet, often without obtaining clear consent from the original content creators Brown et al. (2022). Lee et al. (2023) evidence that PLMs do reproduce content from training samples, encompassing all three classifications of plagiarism. More recently, McCoy et al. (2023) explore the novelty of machine-produced texts and conclude that neural language models have the knack for weaving familiar components into fresh content, rather than merely echoing training samples. Collectively, these studies illuminate the undeniable fact that the outputs of PLMs do draw heavily from their training samples. This sheds light on the deep reliance of pre-trained language models on
their training data, underscoring their limitations in terms of innovation and originality.

heading: 6.2 Plagiarism Detection for LMs
text: Plagiarism detection in language models refers to determining whether the model’s output replicates content from its training data. To identify if machine-generated texts directly “plagiarize" from its training set, researchers have developed a multitude of detection techniques. For instance, Bensalem et al. (2014) introduce an innovative, language-agnostic plagiarism detection approach, dubbed the n-gram class method, which relies on a novel text representation. Küppers and Conrad (2012) gauge the Dice coefficient for 250-character blocks between passage pairs, while Shrestha and Solorio (2013) employ n-grams to compute the Jaccard similarity. Relying on Convolutional Neural Networks, Agarwal et al. (2018) extract regional information from n-grams and use Recurrent Neural Networks to grasp long-term dependencies. Alzahrani (2015) identify candidate documents by searching for exact duplicated sequences and analyze the similarity of overlapping 8-grams. In the context of generation tasks lacking standard automatic measures, Brooks and Youssef (2021) propose an automated originality testing method. Recently, Lee et al. (2023) embrace a combination of traditional similarity metrics and cutting-edge models, aiming to enhance the efficacy of plagiarism detection. While these studies focus on pinpointing models’ plagiaristic behaviors, the question of how to amplify the originality of the generated content has remained unexplored.

heading: 6.3 Contrastive Decoding
text: Li et al. (2023) use an amateur language model (LM) to aid an expert LM in generating coherent text. The work of O’Brien and Lewis (2023) additionally supports the efficacy of contrastive decoding in text generation. In our approach, we integrate their concept into downstream tasks. We use prompts to encourage model plagiarism. This allows us to utilize a single PLM, comparing its predictions when conditioned by opposing prompts, thereby discerning and refining the model’s responses. In a related vein, Chuang et al. (2023) propose a contrastive decoding approach, emphasizing the disparity in logits between a higher layer and a lower layer to derive the output probability over the next word. Additionally, Dai et al. (2022) discover the presence of “knowledge neurons” dis-
14439
tributed predominantly in the topmost layers of pre-trained BERT models. Building upon these insights, our work adopts a similar idea, but just leveraging knowledge from the topmost layer to penalize plagiarized tokens. This will contribute to the improving of originality of model’s generation.
Conclusion
In this study, we emphasize once again the susceptibility of fine-tuned PLMs to display plagiarism tendencies and replicate content from their training sets. This observation applies to both encoder-decoder and decoder-only architectures. These findings highlight the potential risks linked to the utilization of PLMs in sensitive domains like academic writing and storytelling. To tackle the problem of model plagiarism and boost its originality, we introduce an approach involving the introduction of three plagiarism prompts. These prompts guide the model to initially replicate training data in accordance with plagiarism prompts, effectively functioning as an amateur model. Simultaneously, penalties are applied to tokens that show increased probabilities in a designated function, thereby enhancing the overall originality of the generated text. Implementing the proposed approach results in a significant reduction in the generation of non-original sequences, particularly those with over three-word fragments, across both the AASC and ROCStories datasets using PLMs. The alignment of these outcomes offers a robust evaluation of our proposed SP contrastive decoding strategy’s ability to ensure the originality of generated text. Furthermore, human evaluation suggests that our method has minimal adverse effects on the coherence and fluency of the output from large pre-trained models. This comprehensive validation highlights the strength and practical utility of our proposed approach in addressing plagiarism risks in NLP tasks.
Limitations
Due to computational limitations, we can not test SP contrastive decoding strategy on larger pretrained models like GPT-3 (Brown et al., 2020). This will be a focus of our future work. Given the opacity of the training data and the high complexity of the GOT algorithm, we opt not to validate our method on base models without fine-tuning. This is an area we plan to address and improve upon in future work. The primary constraint inherent
to our approach lies in its theoretical inability to entirely eradicate plagiarism. Furthermore, given its prompt-based nature, the efficacy of the SP contrastive decoding strategy is contingent upon the model’s inherent comprehension capacity. This dependence implies that its performance might be less optimal when applied to models outside the realm of PLMs. Another potential concern with the SP contrastive decoding strategy is its inherent need to prompt the model to generate predictions contrary to the intended outcome. This complicates the algorithm’s use for purposes other than counterplagiarism unless a consistently opposing directive is employed.
Ethics Statement
This research endeavor rigorously adheres to ethical guidelines and principles. We exclusively employ publicly accessible models, datasets, and tools, thereby precluding any involvement or collection of sensitive or private data. All datasets and model parameters are obtained strictly for research purposes from public repositories. Central to this study is the objective to chart a course for effectively guiding the behavior of models, aiming to enhance their capacity to assist users ethically. Our ambition is to foster the ethical advancement of language models, championing their deployment in ways that respect ethical standards and promote societal well-being.

heading: Acknowledgements
text: This work was supported in part by the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/060/2022/AFJ, the mainland China collaboration project, National Natural Science Foundation of China Grant No. 62261160648), the Science and Technology Development Fund, Macau SAR (Grant No. FDCT/0070/2022/AMJ, the mainland China collaboration project, China Strategic Scientific and Technological Innovation Cooperation Project Grant No. 2022YFE0204900), the Multi-year Research Grant from the University of Macau (Grant No. MYRGGRG2023-00006-FST-UMDF), and the Tencent AI Lab Rhino-Bird Gift Fund (Grant No. EF202300151-FST). This work was performed in part at SICC which is supported by SKL-IOTSC, and HPCC supported by ICTO of the University of Macau. We would like to thank the anonymous reviewers for their insightful comments.

heading: A Hyper-parameters for training
text: As shown in table 5, in the fine-tuning process of both LLaMA1 and LLaMA2 using both the AASC and ROCStories datasets, specific hyperparameters were employed. These include a learning rate of 1e-4, a batch size of 4, a LORA-R of 8, a LORAAlpha of 16, and a LORA-Dropout of 0.05. For the fine-tuning of GPT-2 with the AASC dataset, a learning rate of 5e-4 is adopted. The block size is configured at 128, and a batch size of 16 per device is instituted, further compounded by a gradient accumulation step count of 2. Conversely, for the fine-tuning of GPT-2 using the ROCStories dataset, default learning rate settings are maintained. The block size is established at 60, with a batch size of 128 per device and a gradient accumulation step count of 2. As for the T5 model, the batch size is set to 24 per device. During the fine-tuning phase on the AASC dataset, a learning rate of 0.001 is employed, coupled with a gradient accumulation step count of 4. Correspondingly, for the ROCStories dataset, the learning rate is maintained at 1e-4, and the gradient accumulation step count is retained at 2.

heading: B GOT Algorithm
text: Table 4 provides the pseudo code of the Generation Originality Test (GOT) algorithm. We utilized this as the basis for implementing the Python script of the GOT algorithm for conducting similarity testing.

heading: C Human Evaluation
text: C.1 Evaluation Metric In this section, we provide definitions for coherence and fluency, breaking them down into specific components:
Coherence:
• Content Association: Indicates whether the output sentence is related to the content of the source sentence.
• Logical Coherence: Rates the natural degree of logical connection between the output sentence and the source sentence.
Fluency:
• Grammar Errors: Indicates whether the output sentence contains grammar errors.
• Naturalness and Vividness: Assesses whether the language of the output sentence is natural and vivid, as opposed to being stiff or verbose.
• Writing Style: Examines whether the output sentence aligns with the style of academic papers or narrative storytelling.
C.2 Template of Human Evaluation Figure 8 illustrates the template utilized for manual evaluation alongside four concrete examples. It is pertinent to note that due to the recruitment of two volunteer assessors who are native Chinesespeaking doctoral candidates, certain rule explanations are presented in Chinese. The test sentences were randomly sampled from model outputs. Additionally, the identities of the three models have been concealed to uphold the integrity and fairness of the evaluation process. This methodology ensures an unbiased assessment of model performance, safeguarding against potential predispositions or biases associated with prior knowledge of specific model identities.

heading: D Example outputs
text: Table 6 and 7 elucidates a collection of exemplars encompassing input texts and corresponding outputs, both with and without the integration of the SP contrastive decoding strategy, derived from both fine-tuned LLaMA2, GPT-2 large and T5 large pre-trained models across the two distinct datasets (ROCStories and AASC datasets). Evidently, all
instances of SP contrastive decoding strategy generated outputs aptly maintain logical coherence and textual fluency while effectively circumventing plagiarism. This substantiates the inherent compatibility of the SP contrastive decoding strategy with models underpinning disparate architectures, as well as its efficacy within divergent text generation domains. This robustness further underscores the algorithm’s commendable generalization potential. However, it is imperative to acknowledge a potential drawback: the proposed SP contrastive decoding strategy’s direct influence on altering the content and conceptual structure of the output text. Consequently, the algorithm may exhibit suboptimal performance within certain generation contexts where the content or conceptual direction is explic-
itly specified or mandated.

heading: Future Work
text: This will be a focus of our future work. Given the opacity of the training data and the high complexity of the GOT algorithm, we opt not to validate our method on base models without fine-tuning. This is an area we plan to address and improve upon in future work. The primary constraint inherent
to our approach lies in its theoretical inability to entirely eradicate plagiarism. Furthermore, given its prompt-based nature, the efficacy of the SP contrastive decoding strategy is contingent upon the model’s inherent comprehension capacity. This dependence implies that its performance might be less optimal when applied to models outside the realm of PLMs. Another potential concern with the SP contrastive decoding strategy is its inherent need to prompt the model to generate predictions contrary to the intended outcome. This complicates the algorithm’s use for purposes other than counterplagiarism unless a consistently opposing directive is employed