InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification

Authors: yujia  hu
zhiqiang  hu
chun-wei  seah
ka-wei  roy
  lee
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 04:31:44

Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in a wide range of NLP tasks. However, when it comes to authorship verification (AV) tasks, which involve determining whether two given texts share the same authorship, even advanced models like ChatGPT exhibit notable limitations. This paper introduces a novel approach, termed  InstructAV, for authorship verification. This approach utilizes LLMs in conjunction with a parameter-efficient fine-tuning (PEFT) method to simultaneously improve accuracy and explainability. The distinctiveness of InstructAV lies in its ability to align classification decisions with transparent and understandable explanations, representing a significant progression in the field of authorship verification. Through comprehensive experiments conducted across various datasets, InstructAV demonstrates its state-of-the-art performance on the AV task, offering high classification accuracy coupled with enhanced explanation reliability.

Main paper: 
 introduction: 
 Authorship Verification (AV) is a task aimed at determining if two texts were written by the same author, with significant implications across forensics, literature, and digital security domains (Halvani et al., 2019; Stamatatos, 2016) . Traditionally, AV relied on stylometric analysis, utilizing linguistic and stylistic features, such as word and sentence lengths, and function word frequencies, to distinguish between authors (Seroussi et al., 2011; Bevendorff et al., 2019) . However, the advent of machine learning, particularly deep learning models like BERT (Devlin et al., 2018) and RoBERTa (Jones et al., 2022) , has revolutionized this field. These modern approaches, leveraging complex patterns in text, have shown superior performance over conventional stylometric techniques in identifying authorship (Saedi & Dras, 2021; Konstantinou et al., 2022; Valdez-Valenzuela et al., 2023) . This paradigm shift underscores a significant evolution in AV methodologies, emphasizing the increasing effectiveness of machine learning in text analysis (Zheng & Jin, 2023) .While current AV models have made notable advancements, they predominantly focus on binary classification and notably lack in providing explanatory insights. Explainability is not only of academic interest. It is fundamental to understanding a model's decision-making logic, and it also enhances trust and reliability in the model's output. Additionally, the lack of clear explanations makes it hard to find and fix any biases that may be hidden inside these models, creating a significant problem for ensuring they are fair and unbiased. Therefore, it's critical for AI models not only to be accurate but also to provide transparency and interpretability. This paper presents the InstructAV framework, an innovative approach tailored for AV tasks. Unlike existing models, InstructAV is designed to accurately verify authorship across texts while concurrently furnishing detailed linguistic explanations for its determinations.A key feature of InstructAV is its unique capacity to integrate explainability directly into the classification process, thereby creating a direct pathway between making accurate predictions and offering deep explanations. Through rigorous testing across three diverse AV datasets, the InstructAV framework has demonstrated not only outstanding accuracy in authorship verification but has also set a new benchmark by producing coherent and substantiated explanations for its findings. This dual capability of InstructAV-merging high predictive performance with actionable insights-marks a significant leap forward in the AV domain, contributing both to the enhancement of model transparency and the advancement of explainable artificial intelligence.Our contributions can be summarized as follows: (i) We propose the InstructAV framework for AV tasks to accurately determine whether two texts share the same author and to furnish robust linguistic explanations for the AV outcomes. (ii) We have curated three instructiontuning datasets, each accompanied by dependable linguistic explanations for AV tasks. These datasets are intended to serve as valuable resources for advancing research in this field1 . (iii) Both automated and human evaluation results demonstrate the effectiveness of the InstructAV in providing precise AV predictions and reliable linguistic explanations. 
authorship verification: 
 In the last two decades, AV has evolved significantly, transitioning from traditional methods focusing on linguistic features like spelling and style to machine learning techniques (Boenninghoff et al., 2019) . However, traditional machine learning, such as support vector machines, showed limited effectiveness (Konstantinou et al., 2022) . Recent advancements involve contextual embeddings from language models like BERT, T5, and MPNET (Devlin et al., 2018; Raffel et al., 2020; Song et al., 2020) , and further studies have explored graph convolutional networks and BiLSTM with attention mechanisms (Valdez-Valenzuela et al., 2023; Sun et al., 2023) . Moreover, Huang et al. (2024) explored different representations of authorship to verify their effectiveness in encoding writing styles. They concluded that authorship representations might be expected to remain robust against certain types of data shifts.Previous neural network-based methods, such as BERT and MPNET, have been crucial in advancing classification tasks, including AV. However, these models inherently lack mechanisms to elucidate their decision-making processes, a critical gap as the demand for explainability in AI grows Hung et al. (2023) . Addressing this, Hung et al. (2023) introduced PromptAV, an innovative technique that leverages the capabilities of large language models (LLMs) for AV, employing step-by-step stylometric explanation prompts to enhance interpretability. Their findings demonstrate that PromptAV significantly outperforms traditional approaches like chain of thought (CoT) prompting (Wei et al., 2022) and PS+ prompting (Wang et al., 2023) in both accuracy and interpretability, marking a noteworthy advancement in the application of LLMs to AV tasks. Huang et al. (2024) utilized LLMs with the Linguistically Informed Prompting (LIP) technique for authorship verification, revealing that even without domain-specific fine-tuning, the LIP method guides LLMs to satisfactory performance.Despite the progress, these methods' reliance on a few-shot demonstration model poses challenges in ensuring the consistency and relevance of their explanations. This limitation underscores a critical need for a more robust solution that can deliver both precise classification and meaningful explanations across a broader range of scenarios. To address this need, we introduce the InstructAV framework, which adopts an instruction fine-tuning approach to significantly enhance the classification accuracy and explanation quality in AV tasks. By refining the model's ability to generate relevant and consistent explanations, InstructAV not only builds on the foundation established by PromptAV but also addresses its primary limitations, offering a comprehensive solution that advances the field of explainable AV. Text 1: Mama's homemade remedy for laughs! Mama Harper ....... Text 2: Not so Bad! Remember that this show was one of the first ......The correct answer is yes/no. Analyzing Text 1 and Text 2 based on the listed writing style characteristics, we find the following similarities/differences: ...... The emergence of LLMs like GPT-3 represents a significant advancement in AI, expanding the capabilities of machine learning technologies. However, deploying these models poses challenges due to their high computational and memory demands. Parameter-Efficient Fine-Tuning (PEFT) addresses these issues. It selectively adjusts a small subset of model parameters, customizing it for specific downstream tasks with minimal resource usage Ding et al. (2023) ; Liu et al. (2022) .Within the PEFT paradigm, adapters are notable. These compact modules integrate into pre-trained networks, providing a resource-efficient way to customize models for specific tasks or datasets (Hu et al., 2023) . By enabling targeted specialization without extensive retraining, adapters expand the accessibility of LLMs for various applications, enhancing their utility and flexibility.The Low-Rank Adaptation (LoRA) adapter, a notable PEFT method, fine-tunes model weight matrices with low-rank modifications to boost task-specific trainability while retaining model strengths (Hu et al., 2021) . Another method, Prefix-Tuning, adds task-specific parameters to transformer layers for alignment (Li & Liang, 2021) . These methods merge adaptability with computational efficiency, enabling advanced LLMs to operate in resourcelimited settings. LoRA, especially, preserves LLMs' general capabilities while enabling task-specific fine-tuning, marking a significant advancement in the field (Ding et al., 2023) . In our study, we outperform AV tasks with LoRA fine-tuning.In this setting, we augment the classification data with linguistic analysis to empower the model to generate robust explanations for the AV predictions. The LLMs are fine-tuned to not only predict the classification labels, but also provide an analysis of eleven linguistic features of the two texts. This added layer of analysis offers a reasoned explanation behind the classification decision, thereby enhancing the model's interpretability and reliability.For the Classification setting, We randomly sampled 11,000 balanced samples from the IMDB62, Twitter, and Yelp datasets, respectively. The data was then divided into a training set comprising 10,000 samples and a test set comprising 1,000 samples, with both sets maintaining balanced class distributions.For the Classification and Explanation setting, we initially sampled 20,000 examples from each dataset and used the GPT-3.5-turbo API for linguistic analysis. To ensure the high quality of the generated explanations, samples with incorrect linguistic analysis were dropped during the initial consistency verification phase. For a comprehensive evaluation, balanced subsets consisting of 10,000 training samples and 1,000 testing samples were randomly selected from the verified examples within each subset. These samples formed instruction-tuning data, including text and corresponding linguistic explanations (see Section 3.1). Details regarding dataset statistics and characteristics, including the splits between training and test sets, are presented in Table 1 .These two dataset settings enable a thorough investigation into how the addition of explanatory components influences the performance of the classification task, providing insights into the efficacy and adaptability of the InstructAV framework.Baselines. For the Classification task, our baseline models comprised BERT (Devlin et al., 2018) and its variants, DistilBERT (Sanh et al., 2019) , and AlBERT (Lan et al., 2019) . These models were selected for their widespread usage and proven effectiveness in AV classification tasks (Brad et al., 2021; Tyo et al., 2022; Fabien et al., 2020) . However, these models are not inherently designed for text generation tasks. Another baseline related to LLMs is the application of the LIP method proposed by Huang et al. (2024) on GPT-4-turbo, LLaMA-2-70B, and Mistral-7B.For tasks involving Classification and Explanation, autoregressive models like GPT, which generate text sequences based on preceding text, are suitable. We adopted the PromptAV methodology on GPT-3.5-Turbo as a baseline for the ChatGPT model, as proposed by (Hung et al., 2023) . Although PromptAV may be slightly less effective than BERT in classification accuracy, it offers advantages such as bypassing extensive training and generating explanations. Additionally, we integrated PromptAV, using GPT-4 as the foundational model, to establish a more competitive baseline. Our comparative analysis evaluates both classification accuracy and the quality of explanations from InstructAV and PromptAV. This comprehensive assessment aims to understand the strengths and potential limitations of each approach in AV tasks.Implementation. In our research, we employed three different LLMs to validate the efficacy of the InstructAV framework. These models include OPT-6.7B (Zhang et al., 2022) , LLaMA-1-7B (Touvron et al., 2023a) , and LLaMA-2-7B (Touvron et al., 2023b) . Our experiments were conducted using two NVIDIA A6000 GPUs, encompassing both the fine-tuning and inference phases.For fine-tuning, we applied the LoRA approach with a rank set to 8. This introduced 4.1 million parameters, representing only 0.06% of the total 7 billion parameters of the base models. This parameter increase highlights LoRA's effectiveness in offering an efficient fine-tuning method. Fine-tuning was performed over 3 epochs using the LLM-Adapters Toolkit (Hu et al., 2023) , tailored for integrating adapters into LLMs.During inference, all operations were performed deterministically with a fixed temperature of 0.1. This maintains consistency and reliability in the model's performance, crucial for accurately assessing the InstructAV framework's capabilities. The selection of diverse LLMs and the strategic application of LoRA for fine-tuning effectively showcase the capability of the InstructAV framework across various model architectures. To ensure the reproducibility of the experiments conducted using the InstructAV framework, each experiment was repeated three times with different random seeds. The mean and standard deviation were calculated from these repetitions. 
methodology: 
 The overview of the InstructAV framework, as depicted in Figure 1 , comprises three primary steps: data collection, consistency verification, and the fine-tuning of LLMs using the LoRA method (Hu et al., 2021) . Initially, the framework focuses on the aggregation of explanatory data for AV samples. This approach uses the binary classification labels available in existing AV datasets. Following this, a strict quality check is implemented, aimed at verifying the alignment and consistency of the explanations with the corresponding classification labels. The final stage involves the synthesis of instruction-tuning data, which is a fusion of the gathered classification labels and their associated explanations. This composite data then serves as the foundation for fine-tuning LLMs in conjunction with the LoRA adaptation technique. This approach ensures that the LLMs are not only accurately fine-tuned for the AV task but also enhanced in their capacity to provide coherent and reliable explanations for their predictions. The subsequent sections present the details of each component in InstructAV.To augment the explanatory capabilities of the InstructAV model, particularly in generating dependable explanations for AV predictions, we initiated a comprehensive collection of explanations using ChatGPT, with the crucial step of informing ChatGPT about the classification labels beforehand. This process involved three datasets widely used in AV studies: the IMDB dataset (Seroussi et al., 2014) , the Twitter dataset (Schwartz et al., 2013) , and the Yelp Reviews dataset2 . This selection strategically covers various dimensions of textual data, thereby ensuring a diverse and comprehensive analysis.The IMDB dataset, characterized by its longer text length (averaging 303 words), exemplifies long-form content. In contrast, the Twitter dataset, with an average text length of 16 words, epitomizes short-form content. The Yelp Reviews dataset, averaging 154 words per text, represents medium-length content. These datasets, initially curated for authorship attribution tasks, were adapted for our study. We extracted 11,000 samples from each dataset to facilitate a robust AV evaluation. Each sample comprises two texts, either written by the same author or by different authors.For generating explanations, we employed ChatGPT with true classification labels and few-shot prompts, focusing on 11 linguistic features for each AV sample. These features, identified in the research by (Boenninghoff et al., 2019) , are crucial for analyzing writing styles in textual content. The linguistic features generated by known-label ChatGPT were recorded as labels for the AV explanations in our dataset if they accurately predict and meet the consistency check.To link classification labels closely with linguistic explanations during the data collection phase, we crafted prompts that incorporate the classification labels. For example, a prompt might state "The following Text1 and Text2 are written by different authors. Please analyze their writing styles and explain why they are written by different authors." to guide the explanation process. A detailed example of the explanation generation prompt is shown in Appendix B.2 Table 8 . These prompts enhance the relevance of the collected explanations, thereby improving the explanatory capacity of the InstructAV model. Consistency Verification. Alignment between classification labels and their linguistic explanations is essential for explanation data integrity and trustworthiness. Models like known-label ChatGPT are skilled at generating explanations but face challenges in align explanations with classification labels. For instance, despite being informed that "The following Text1 and Text2 are written by the same author," ChatGPT might incorrectly respond with, "The correct answer is no." Instances like this create a mismatch between explanations and classification labels, reducing trust in the model's explanations. To enhance user trust in automated decisions, it is important to guarantee the consistency and reliability of both classifications and explanations. As shown in Figure 1 , we employ a consistency verification method to verify the alignment between the model's analytical explanations and its classification decisions. We have instituted a comprehensive verification process to ensure the explanations are consistent with the classification. This process leverages demonstration templates that inherently guide the model to incorporate specific expressions-such as 'same/similarities' or 'different/differences'-into its outputs. During the consistency verification stage, keyword searching was performed on terms within the generated text. Matching the phrases 'written by the same author' (resp. 'written by different authors') with classification labels allows us to assess the quality of ChatGPT's linguistic explanations.The key to this process is the construction of instruction-tuning data, which serves a dual purpose: facilitating AV classification prediction and supporting the generation of explanations. An example of the instruction-tuning data is shown in Appendix B.3 verification step is essential for ensuring that the InstructAV model is not only accurate in its predictions but also capable of generating explanations that are coherent, relevant, and aligned with the classification outcomes, thereby enhancing the overall efficacy and reliability of the InstructAV framework.Adapting LLMs for AV tasks can be a demanding process, particularly due to the significant computational resources and extensive labeled data typically required for fine-tuning. To address this challenge, we have incorporated a PEFT method known as LoRA (Hu et al., 2021) into our approach for adapting LLMs to AV tasks.LoRA presents a novel approach to updating weight matrix parameters. It decomposes the matrix into two low-rank matrices, reducing trainable parameters while processing high-dimensional matrices effectively. During the forward pass, LoRA matrices compute rank-decomposed weights, used in attention or feed-forward activations. Importantly, this method keeps the LLM's core parameters unchanged, enabling adaptation to new data distributions or tasks without overhauling the model completely.The LoRA architecture is encapsulated in the following equation:h = W 0 x + ∆W 0 x = W o x + BAx (1)where W 0 ∈ R d×d is a pre-trained weight matrix in LLMs, which is associated with two adjustable low-rank matrices, B ∈ R d×r and A ∈ R r×d , d corresponds to the hidden dimension of the attention layers, and r is the adaptation rank, which is selected such that r ≪ d. Within the training phase, W 0 remains frozen, exempt from gradient updates, whereas A and B are dynamic, containing the parameters subject to training. 
experiment settings: 
 Datasets. The InstructAV framework evaluation utilized three distinct Authorship Verification (AV) datasets: IMDB62 (Seroussi et al., 2014) , Twitter (Schwartz et al., 2013) , and Yelp Reviews3 , chosen for diversity and relevance.To evaluate the AV classification and explanation tasks performance, we constructed two distinct types of dataset settings, each incorporating varying levels of information:1. Classification: This dataset involves integrating a question alongside two texts as the input and employs the LoRA method to fine-tune the model for classification tasks.The expected output is a straightforward binary classification indicating whether the two texts are written by the same author, formatted as "The correct answer is yes/no".The evaluation of the InstructAV framework involves both AV classification and explanation generation tasks. This dual-focused approach enables an in-depth assessment of the framework's capabilities in key areas of AV.For AV classification tasks, accuracy serves as the primary metric. This metric effectively measures the models' proficiency in determining whether two texts have been written by the same author. To evaluate the framework's performance in generating explanations for AV tasks, a distinct approach is required. Here, the focus shifts towards assessing the quality of the linguistic analysis generated by the models. Unlike accuracy, which is quantitatively measurable, the quality of linguistic explanations presents a more complex evaluation challenge, as it involves subjective elements related to the coherence, relevance, and clarity of the explanations. Given the complex nature of evaluating explanation quality, we adopt a dual evaluation approach, employing both automatic and human evaluation metrics.Automatic Evaluation for Explanations. In our automated assessment of explanation quality for both InstructAV models and baseline models, we measure the similarity between generated and labeled explanations, using ChatGPT-generated texts as standards. Through consistency verification, we ensure explanatory labels within test sets consistently align with classification labels, affirming their accuracy. Consequently, we consider ChatGPT's linguistic analysis as explanation labels. We employ metrics like ROUGE-1, ROUGE-2, ROUGE-L (for content coverage and structural fluency), and BERT Score (for semantic quality). Higher scores indicate better explanation quality, ensuring a comprehensive evaluation based on content accuracy, logical coherence, and contextual relevance.Human Evaluation for Explanations. To supplement the automated metrics for evaluating explanation quality in the InstructAV framework, we also implemented a human evaluation. This approach involved randomly selecting 100 examples from each of the IMDB, Twitter, and Yelp Reviews datasets for qualitative assessment by human evaluators. These evaluations included not only explanations generated by InstructAV and baseline models, but also those explanation labels. For the human evaluations focusing on explanations, we selected InstrctAV (LLaMA-2-7B) along with the two most robust baselines, namely PromptAV-2shot equipped with GPT-3.5 and GPT-4 as the base models. Three evaluators were enlisted to independently evaluate each explanation across four criteria:Coverage: Evaluating how many of the anticipated linguistic features are present in the explanations. Evaluators checked the number of features covered in each explanation, with baseline models like PromptAV covering 0 to 7 features, and InstructAV, along with explanation labels, extending from 0 to 11 features.Relevance: Evaluators scored the relevance of the explanations to the original texts on a 5-point Likert scale, ranging from 1 (completely irrelevant) to 5 (perfectly relevant).Reasonableness: This involved assessing the logical soundness of each feature's explanation, using a 5-point Likert scale, from 1 (totally unreasonable) to 5 (completely reasonable).Persuasiveness: Evaluators rated the persuasiveness of the explanations on a scale from 1 (completely not persuasive) to 5 (highly persuasive).Classification Results. We evaluate the InstructAV framework for AV tasks, utilizing both classification and explanatory dataset settings. The primary goal is to explore how linguistic analysis can improve the model's performance in AV classification tasks. The results, detailed in Table 2 , include the accuracy of InstructAV with various LLMs and baseline models.A key finding is that InstructAV, particularly when paired with LLaMA-1-7B and LLaMA-2-7B, outperforms baseline models in all datasets when using only classification data. Notably, with LLaMA-2-7B, InstructAV achieves a 25.2% improvement over the highestperforming baseline, BERT, on the IMDB dataset.It's imperative to underline that in our experimental setup, models were evaluated on their ability to concurrently generate classification predictions and conduct linguistic analysis without incorporating explanation data in the input. This methodology ensured fair comparison with approaches that focus only on classification. Our results clearly demonstrate that all variants of the InstructAV framework, utilizing different LLMs, significantly benefit from training on explanatory labels. Notably, InstructAV with LLaMA-2-7B showcased a remarkable 27.1% improvement in classification accuracy over the PromptAV approach using 2-shot prompts based on GPT-3.5. Furthermore, when compared to PromptAV-2shot employing GPT-4, InstructAV exhibited superior performance across all three evaluated datasets. These findings highlight the substantial benefits of incorporating explanatory training, indicating that providing InstructAV with the dual function of AV classification and generating linguistic explanations significantly boosts its classification precision.Automatic Evaluation Results on Explanations. Our InstructAV framework demonstrates that training in explanation labels can significantly improve AV classification performance. Additionally, we aim to assess whether InstructAV can generate high-quality explanations.For this purpose, we selected PromptAV-2shot (GPT-3.5), the best-performing variation of the framework, alongside GPT-4, to benchmark against all variations of InstructAV. we subjected them to both automatic and human evaluations focused on the quality of their generated linguistic explanations. The results of the automatic evaluation are presented in Table 3 . These results reveal that InstructAV consistently surpasses PromptAV models (GPT-3.5 and GPT-4) across all datasets and all evaluation metrics. Notably, ROUGE-1 and ROUGE-2 scores highlight InstructAV's superior performance in achieving content overlap at both the word and phrase levels. Moreover, the ROUGE-L metric indicates that InstructAV is more proficient in maintaining sentence-level structure and fluency. Furthermore, the BERT Score supports the observation that the explanations generated by InstructAV are semantically closer to the explanation labels. This comprehensive evaluation underscores In-structAV's capability not only in improving AV classification accuracy but also in generating linguistically coherent and contextually relevant explanations.To comprehensively evaluate the generated explanations and to assess the quality of the linguistic analysis produced by ChatGPT, which serves as our explanation labels, we have conducted a human evaluation using four key metrics: Coverage, Relevance, Reasonableness, and Persuasiveness. The results of this human evaluation are presented in Table 4 . Our findings from this human evaluation process show that our explanation labels achieves the highest scores. This result validates our methodological choice of using explanations generated by known-label ChatGPT as both a source for training data and a benchmark for explanation labels in our testing scenarios. Importantly, the results also reveal that InstructAV, particularly with the LLaMA-2-7B model, not only surpasses the performance of PromptAV-2shot models (GPT-3.5 and GPT-4) but also attains a level of explanation quality that is comparable to known-label ChatGPT. This outcome is significant as it demonstrates that InstructAV can produce explanations that are not only accurate but also contextually relevant, logically sound, and convincing to human evaluators. Such a capability is essential for applications where understanding the rationale behind model predictions is as important as the predictions themselves.To explore the relationship between explanation quality and classification accuracy, we have selected two distinct subsets of InstructAV samples: the top 25% with the highest average human evaluation scores and the bottom 25% with the lowest average human evaluation scores. We then calculated the classification accuracy for each of these subsets, and the results are presented in Table 5 .Our analysis clearly indicates that samples associated with higher quality explanations consistently achieve superior classification accuracy. This finding underscores the effectiveness of training InstructAV to not only generate AV classification predictions but also provide linguistic explanations. Such training not only enhances the model's performance in AV classification tasks but also enhances its ability to produce valuable linguistic analysis. 
ablation study: 
 Ablation experiments were conducted to investigate potential data contamination issues, specifically whether the data might have been included in the training corpora of LLMs. We performed 0-shot, 2-shot, 4-shot, and 8-shot in-context tests on each dataset using the original, untuned LLaMA-2-7B, LLaMA-1-7B, and OPT-6.7B models. Each experiment was replicated three times, with the mean results presented in Table 6 . The results indicate that the original models struggle to perform AV tests effectively on our dataset, especially in the 0-shot setting where the outputs are highly random. Both LLaMA-1-7B and OPT-6.7B failed to make correct judgments, and LLaMA-2-7B showed weak judgment capabilities. Consequently, we postulate that it is unlikely that our dataset was included in the training data of the original models. 
conclusion: 
 This research presents InstructAV, an innovative approach to AV tasks that leverages LLMs with a PEFT method. Our study establishes InstructAV as a significant advancement in the AV domain, showcasing its ability to enhance classification accuracy and provide clear and coherent explanations for its decisions. The contributions of this paper, including the development of the InstructAV framework, the creation of three instruction-tuning datasets with reliable linguistic explanations, and the demonstration of the framework's effectiveness through both automated and human evaluations, mark a crucial progress in AV research.InstructAV, with its dual priority on high accuracy and the ability to provide high quality explanations, positions it as a state-of-the-art AV solution. 
 
 Appendix: The InstructAV framework, while achieving state-of-the-art performance in AV classification and explanation tasks, does face a notable limitation in its current form. The human evaluations of model-generated explanations differ from typical evaluations in other tasks. Specifically, our evaluations are intended for high-level verification to ensure that the explanations produced by ChatGPT are coherent and align with the text , not as a full assessment of authorial features. Author analysis involves critical frequency and nuanced features better captured by computational methods then humans (Stamatatos et al., 1999) .Theoretically, an ideal human evaluation would ask human participants to justify why two texts are written by the same author, but such research is costly and problematic, as humans do not perform well in tasks that require analysis of writing styles. Considering these limitations, we consider ChatGPT-generated explanations to be a reasonable and cost-effective alternative. Future work will consider a more suitable evaluation method that incorporates computational features.On the other hand, when tasked with generating both AV classification predictions and linguistic explanations, InstructAV is constrained to produce a maximum of 512 new tokens. This restriction can lead to significantly longer inference times. For example, generating explanations for 1,000 samples in a long-text dataset like IMDB can take around 4-5 days, while in shorter-text datasets such as Twitter, this time is reduced to approximately 2 days. Addressing this challenge, future work will focus on developing methods to enhance the efficiency of the inference process within InstructAV, particularly in the context of generating explanations.We have included randomly selected examples to showcase the capabilities of InstructAV in generating both classification predictions and language feature-based explanations. One such example from the IMDB dataset is detailed in Table 7 . This example demonstrates how InstructAV first provides a classification prediction and then follows it with detailed explanations anchored in specific language features pertinent to the classification.These examples highlight InstructAV's proficiency in delivering accurate classifications paired with clear and credible explanations. Notably, the explanations are tailored to each specific sample, rather than being generic or broad. This customization enhances the explanations' effectiveness, as they are directly correlated with the content of the original text. For instance, the model explicates features such as writing style and use of expressions by incorporating words and phrases directly from the text in question. This approach not only bolsters the relevance of the explanations but also their trustworthiness, providing users with insightful and contextually grounded rationale behind each classification decision made by InstructAV.Instruction: Please decide if the following Text1 and Text2 are written by the same authors. 'yes' means from the same author, 'no' means not from the same author. Then, provide an analysis based on writing styles. Upon analyzing Text 1 and Text 2 based on the listed writing style characteristics, we find the following similarities that could suggest that both texts were written by the same author: Writing Style: Both texts seem to be reminiscing about older television shows. The author uses a mixture of narrative and opinion in both texts, indicating a personal connection or memory attached to the subject matter.Expressions and Idioms: The expressions "I miss this show" and "I don't recall ever not watching it" showcase a personal touch and sentimentality in both texts. Tone and Mood: Both texts have a nostalgic and somewhat informal tone. The author shares memories and impressions about shows from the past, using phrases like "I was a kid so I didn't know better" and "Still, I miss this show." Sentence Structure and Syntax: Both texts employ a mix of compound and simple sentences. For instance, "When a sitcom or drama could not get a network, they went with syndication" and "Mama Harper develops her mother's concoction to help her family get over an illness like the flu." Punctuation Style: Both texts employ spaces before and after commas inconsistently, such as in "$180 , 000" and "Marla Pennington , Edie McClurg." Special Characters Style, Capitalization Style: Both texts capitalize show titles like "Mama's Family" and show proper nouns. However, both texts sometimes lack proper capitalization at the start of sentences, such as "Still" in Text 1.Compound and Separate Spelling: No significant differences in spelling patterns are observed between the two texts. Acronyms and Abbreviations: The second text uses "1980s" to denote the decade, showcasing a preference for this kind of abbreviation. Characters Style: The character portrayal in both texts has a casual undertone. For example, "bimbo-in-law Naomi" and "the guy who played her father." Diatopic Variations and Foreign Languages: Both texts appear to be written in American English and reference American television shows. Other Relevant Aspects:Both texts contain anecdotal or subjective experiences like "I was a kid so I didn't know better" and "Still, I miss this show." The author provides specific details about show characters, indicating a personal familiarity with the subject. There's a recurring theme of syndication and older television programming in both texts, further suggesting the same author might have written them.In conclusion, based on the aforementioned similarities in writing style, expressions, tone, and other characteristics, it is plausible that Text 1 and Text 2 were written by the same author.Table 8 : Prompt used for generating explanation labelsThe code and datasets can be found at https://github.com/Social-AI-Studio/InstructAV.https://www.yelp.com/datasethttps://www.yelp.com/dataset 


Decision: The authors propose an approach for authorship verification tasks using Large Language Models with parameter-efficient fine-tuning. The approach is based on three main components: data collection, consistency verification, subspace adaptation (LORA). The proposed approach allows the authors to obtain higher author verification classification accuracy on benchmarks, as well as high-quality explanations of the predictions. The reviewers praised the impressive results, yet brought up some concerns. The authors submitted a response that addressed the concerns, including additional experimental results, and detailed on-point answers. The reviewers are overall enthusiastic about the paper, and the paper can be considered for a spotlight presentation or an oral presentation. 

Clear Accept.

Reviews: review: paper_summary: paper_summary: This paper introduces InstructAV, a novel approach for authorship verification (AV) tasks using Large Language Models (LLMs) with parameter-efficient fine-tuning (PEFT). InstructAV aims to improve both accuracy and explainability in determining whether two given texts are written by the same author. The framework consists of three main steps: data collection, consistency verification, and LLM fine-tuning using the LoRA method. The authors curate three instruction-tuning datasets with reliable linguistic explanations and demonstrate the effectiveness of InstructAV through automated and human evaluations.
strength_weakness: strength_weakness: Cons:
1. Table 4 hard to understand. It is strongly suggested that the authors should explain the evaluation metrics for “Coverage Relevance Reasonableness Persuasiveness” clearly in the table. Otherwise, it is hard for readers to understand.
2. The current baselines are old. It is strongly suggested that the authors include LLMs as baselines considering LLMs have achieve a good performance in Authorship Attribution[2,3]. Otherwise, it is hard to evaluate the effectiveness of the proposed method.
3. No code is provided. It is strongly encouraged that the authors release the code for reproduction.
4. Not sure whether the authors run the baselines multiple times to ensure the convincingness. If the authors run multiple times, it is suggested the authors provide more details.
5. It is encouraged the authors conduct statistical testing for the experiments.
6. Related works are incomplete [1,3,4,5]. It is strongly suggested that the authors conduct a comprehensive literature review.

[1] VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web https://arxiv.org/abs/2207.03477

[2] Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification https://arxiv.org/abs/2310.08123

[3] Can Large Language Models Identify Authorship? https://arxiv.org/abs/2403.08213

[4] Can Authorship Representation Learning Capture Stylistic Features? https://arxiv.org/abs/2308.11490

[5] Contrastive Disentanglement for Authorship Attribution https://dl.acm.org/doi/10.1145/3589335.3652501
questions: questions: How do the authors address the potential data contamination problem?

The potential risk of data leakage is not discussed. The data might have been included in the LLMs' training corpus. I wonder if there are some tests showing the data memorization possibilities and how the authors address the potential data contamination problem.
paper_summary: paper_summary: ## Summary
The paper proposes a new dataset that can be used for instruction fine-tuning for LLMs. As per my understanding, the process (briefly) goes as follows. 
1) Collecting samples from three AV datasets.
2) Providing ChatGPT with a true label and ask it to generate explanation based on a specific template that includes 11 dimensions.
3) Verification of alignment between model answer about the similarity and the provided explanation. (**Not sure what happens with examples that are not-aligned).
4) Using the resulting dataset to fine-tune models (With LoRA)
5) Evaluation of the generated output (classification + explanation) using automatic metric and human evaluations. 

### Quality
The authors investigate different aspects of evaluations for the performance. This includes both the quantitative and the qualitative evaluations.  

### Clarity
There are a few details in the methodology that are not clear. Additionally, there are a few details that, IMO, would greatly enhance the readability of the paper:
1) Using the correct citation format across the paper. (Author's names should be inside the parentheses, except for cases when we are referring to the author's and not the work. E.g. _More details can be found in (Author et al 2020)_ **vs**. _Author (2020) discussed this_. 
2) Giving more informative table captions.  Tables 2-5 all start with **performance of InstructAV on ...*. Instead, relevant captions such as Classification Accuracy, Token-overlap, ... etc.  
3) In page 1, the last line: "*accuracy in authorship identification but ..*". This is probably a typo, authorship identification is a different task. Authors should avoid mixing up the two. 
4) CoT and PS+ need citations. 
5) Table~2: The caption mentions 2 datasets but the table shows 3.

### Originality
To me, the originality is in the instruction-tuning dataset. The use of this dataset results in an increase in the verification performance. Though, I have questions about the evaluation experiments. 

### Significance
The field of AV would greatly benefit from new datasets that aim at understanding the problem. I think the presented dataset is helpful, but the evaluation details have a few unclear issues that require clarification. (Please see questions below) 

P.S. The answer of the questions below is important to my review. I highly encourage the authors to answer the questions and not focus only on the preliminary decision.

# Scores have been updated after reading the responses
strength_weakness: strength_weakness: 1- The analysis of the experiments is very limited and misses many interesting discussion points, e.g. the effect of the LM on the results. 

1.1. In section 3.2, PromptAV and InstructAV are fine-tuned using different language models which brings up the effect of the LM itself on the results. This can be seen for InstructAV where it performs worse then BERT on two datasets when Llama-1 is used. 

1.2.The second issue is that the performance on the IMDB dataset is very high when Llama-2 is used, but for the other datasets the results are close --and sometimes lower-- than the BERT baseline. It is actually troubling when the performance of the tool can only be seen with one specific language model and not across language models. 

2. *Evaluation/Human Evaluation with ChatGPT*. 

2.1. While using ChatGPT as a judge/metric has become very common, I don't see how we can consider its explanations as the ground-truth for this AV task. Consequently, using token-level metrics and BERTScore will only show the similarity of the responses among different language models but we all know that these metric have plenty of issues. Note that this is a *valid* experiment to conduct to evaluate the similarity between responses from language models. However, ChatGPT cannot be considered the gold-standard without human evaluation. 

2.2. The human evaluation misses the main point. The human evaluation in this study is used to review the ChatGPT responses, but ChatGPT by design is able to generate answers that is of high quality. A more suitable human evaluation study would be to ask humans to generate explanation on why two texts are written by the same author and that would be the gold-standard and then we can compare it to ChatGPT responses. The problem is that such study would be very expensive to conduct and might have its own issues since humans do not do well on writing-style analysis tasks.
questions: questions: . In Section 3.1, under consistency verification. I am not clear on this step: You enforce a template that tells the LM whether two text examples are for the same author or not, yet you mention that the model may start its response with the wrong classification label (hence, the verification step). What happens to such examples? are their labels corrected? or are they simply removed? If the template **prevents** the LM from generating the wrong response, then there would be no need for the verification step. Since there IS one, can you explain how are these examples handled?

. In Section 4.1 (Dataset), samples with incorrect linguistic analysis were removed. I think this is questionable. Why would such examples, which could be considered hard examples to distinguish be removed from the dataset? In fact, I think those are the examples that are more interesting. Selecting 10k examples that a language model has easily identified would make the task simpler and less realistic. Can you please clarify the intuition here? 

. How sensitive is InstructAV to the choice of language model? 

. Why is the results on IMDB very high compared to other datasets/or when other LMs are used?

. In table 4, ChatGPT results have the best Human evaluation results. Isn't this expected? Naturally, ChatGPT is a chat agent; instruction-tuned to have a dialogue and generate natural responses conditioned on the presented text. Why is this interesting to this study specifically?
paper_summary: paper_summary: The paper presents a set of experiments on authorship verification using a newly developed framework based on large language models, new promising tools for various language processing tasks. The experiments are carried on using three well known corpora as benchmarks for authorship verification. Exploring the main purpose of large language models to generate text, the experiments included the generation of explanations of the model’s decision. Several stylometric characteristics had been used for text similarity evaluation. The experiments demonstrated high accuracy of author classification obtained by the proposed framework. The generated explanations have been evaluated automatically and manually to ensure their adequacy and consistency.
strength_weakness: strength_weakness: Two types of experiments are combined in one paper; it could be clearer to make two papers with one type of experiments with fine-tuning LLMs and another one with prompt based classification with explanations. Combining various types of experiments the authors made the paper difficult to comprehend.
It is hard to say that automatic evaluation of explanations are effective. 
The paper would be better with more detailed error analysis.
questions: questions: Why do you use ChatGPT explanations as the ground truth?
Why the accuracy of classification & explanation is higher than for classification in Table 2?
paper_summary: paper_summary: The authors feed existing labeled author verification datasets to ChatGPT to generate explanations for the labels, perform some sort of consistency checking between labels and explanations, then use LoRA to fine-tune LLMs on this generated data. Fine-tuned Llama 2 models outperform fine-tuned BERT and prompted GPT4 across 3 author verification datasets (IMDB, Twitter, Yelp). The addition of the explanation to the fine-tuning yields benefits across 3 LLMs (Llama1, OPT, Llama2). Generated explanations are evaluated both automatically and with human review (for a smaller sample).

Minor issues:
* There are many places where the citation style is wrong, e.g., "domains Halvani et al. (2019); Stamatatos (2016)" should be "domains (Halvani et al. 2019; Stamatatos 2016)". The "Author (Year)" form should only be used when the citation takes the place of a noun phrase.
* "A d key feature" => "A key feature"
* Define "PS+ prompting"
* "decisions. we have" => "decisions. We have"
* There's no need to spend space on the LoRA equation. Since you do not make any innovations over this formulation, just saying you're using LoRA for fine-tuning and citing the LoRA paper is enough.
* Tables are often far from where they are discussed. For example, Table 2 is first mentioned on page 8 but appears on page 5, and Table 3 is first mentioned on page 9 but appears on page 7.
* "InstructAV. we" => "InstructAV. We"
* I don't think Table 5 convincingly shows a correlation. Instead, show a scatter plot of explanation quality vs. classification accuracy for each dataset.
strength_weakness: strength_weakness: * The IMDB, Twitter, and Yelp Reviews datasets are "widely used in AV studies" but the authors extract "11,000 samples from each dataset" instead of using a dataset split from prior work. In addition to the lack of direct comparability to prior work, I'm worried that the selected test set may be artificially easier, since the 11,000 samples are by definition the ones where ChatGPT could successfully explain the labels in the original dataset, as ones where there was inconsistency between ChatGPT's explanation and the original labels were discarded.
* The method used to check for consistency between labels and explanations is not described. There needs to be a formal description of this algorithm. As it is, I can only guess that it involves searching for some keywords like "similarities" or "differences", and cannot guess the extent of the keyword list nor what is done once those are found.
* Much of the first 3 pages reads like a sales brochure, describing the task being solved and talking up the approach, but not actually explaining what the technical approach is. The first real details about what exactly is being proposed don't appear until the 4th page.
questions: questions: * What is the background of the human evaluators? Were they blinded to which model's output they were evaluating?

Comments: title: Brief summary
Detailed comment
How do the authors address the potential data contamination problem?
comment: Thank you to all the reviewers for your time and effort in reviewing our paper. Your suggestions are very useful for improving the quality of our paper. Many thanks!
Dear Reviewer GRur,

We hope that you've had a chance to read our comment. We would really appreciate a reply to let us know if our response and clarifications have addressed the issues raised in your previous comment, or if there is anything else we can address.

Sincerely,

Authors
Thank you for your detailed comments.

**[Q1]**

Yes, we will include these details in the updated paper.

**[Q2]**

Thank you for your suggestions. In the updated paper, we will adopt your recommendation to divide the classification and Classification+Explanation sections into two parts. For the classification section, we will use the new test set without discarding any samples. For the Classification+Explanation section, we will use the previously selected samples and provide additional explanations.

**[Q3 & W1.1]**

The performance of InstructAV fine-tuning is correlated with the capabilities of the base model. These capabilities are influenced by factors such as the model size, the size and quality of the pretraining corpus. For instance, we employed 7B models for both LLaMA-2 and LLaMA-1, but LLaMA-2 performed better due to being trained on more tokens and having longer context lengths. Therefore, if we choose a base model with a larger size and a more extensive pretraining corpus, we can expect better performance.

**[Q4 & W1.2]**

Thank you for your insightful comment. I appreciate your perspective and the opportunity to clarify our approach. We understand your point that computational approaches often outperform humans in authorship analysis tasks, as evidenced by works such as Stamatatos (1999). Our intent in using human evaluation was indeed to perform a high-level check to ensure that the explanations generated by ChatGPT were coherent and aligned with the content of the text, rather than to serve as a comprehensive evaluation of authorship features. Specifically, our primary goal was to verify that these explanations were sensible and relevant to the texts being analyzed.
We acknowledge that frequency and other nuanced features play a crucial role in authorship analysis, which computational methods are better suited to capture. As per your earlier comments, a more suitable human evaluation study would involve asking humans to generate explanations on why two texts are written by the same author, which could then serve as a gold standard for comparison with ChatGPT's responses. However, such a study would be very expensive to conduct and might have its own issues, given that humans do not perform well on writing-style analysis tasks.
Given these limitations and constraints, we considered ChatGPT-generated explanations to be a reasonable and cost-effective alternative; minimally, we have verified that the explanations generated by ChatGPT were coherent and aligned with the text content.

In our updated paper, we will make these distinctions clearer and provide more context around the purpose and scope of our human evaluations. We will also highlight the limitations of our approach and elaborate on the role of computational methods in capturing the finer details of authorship features.

**[Brief summary]**

Once again, thank you for the valuable questions, which have greatly refined our paper. We will revise our paper to address your comments and questions, including those about clarity, to improve overall readability.
Thank you for your reply and comment.

To investigate potential data contamination, we conducted 0-shot, 2-shot, 4-shot, and 8-shot tests on each dataset using the original, un-tuned LLaMA-2-7B, LLaMA-1-7B, and OPT-6.7B models. Each experiment was performed three times, and the average results are presented below:

| |Model | IMDB | Twitter | Yelp |
|----|----|----|----|----|
0-shot |LLaMA-1-7B | 0.000 | 0.000 | 0.000
| | OPT-6.7B | 0.000 | 0.000 | 0.000
| |LLaMA-2-7B | 0.006 | 0.010 | 0.021
2-shot |LLaMA-1-7B | 0.189 | 0.097 | 0.226
| | OPT-6.7B | 0.258 | 0.179 | 0.297
| |LLaMA-2-7B | 0.309 | 0.336 | 0.397
4-shot  |LLaMA-1-7B | 0.014 | 0.020 | 0.127
| | OPT-6.7B | 0.191 | 0.220 | 0.217
| |LLaMA-2-7B | 0.279 | 0.350 | 0.375
8-shot |LLaMA-1-7B | 0.003 | 0.007 | 0.002
| | OPT-6.7B | 0.005 | 0.008 | 0.003
| |LLaMA-2-7B | 0.020 | 0.033 | 0.025

The results indicate that the original models struggle to perform AV tests effectively on our dataset, especially in the 0-shot setting where the outputs are highly random. Both LLaMA-1-7B and OPT-6.7B failed to make correct judgments, and LLaMA-2-7B showed weak judgment capabilities. The best performance was observed with the 2-shot models, which could produce outputs following the template format, "The correct answer is yes/no...[Analysis]." However, even when the format was correct, the accuracy was still very low, with the best result (LLaMA-2 on the Yelp dataset) being lower than a random guess. Consequently, we postulate that it is unlikely that our dataset was included in the training data of the original models.
Dear authors, 

Thank you for the response. I did read your reply to my comments and to the other reviewers' as well. I believe that the experiment on 1000 examples (response qkT2 below) is very important AND is the correct setup to conduct the experiments. It also shows that with this setup similar observations about the performance of baselines and proposed methods still hold.
 
Additionally, I think that most of the issues can be solved either by rewording and making careful claims (specifically about human evaluation being the ground truth), and some doable reorganization of the experimental sections (if you choose to do so).  

Finally, please consider the comments that I've provided about clarity above.
Thank you for the detailed response. 

Q1) I see. Would be great if you can clarify this in the paper as well.

Q2) Thank you for the clarification. I see an issue here where certain samples were dropped because chatGPT did not do well on them. To me, this is what happened: "we have a test set and some examples are problematic to ChatGPT so we are going to remove them and consider the remaining samples as our new test set". Perhaps, this would work if all the models are of the same family, but comparing with the baselines (which are very competitive even after subsampling) would not be fair. 

You address this point in "response to W1 of Reviewer qkT2" which is very important and in fact, I think that's how your main experiment should be. This should make the separation between classification (the typical verification task) experiment and explanation experiment much clearer. In the classification experiments below, the baseline BERT outperforms the other methods including yours except when Llama-2 is used. Conditioning the effectiveness of the proposed approach on the use of one specific language model is problematic. 

IMO, a much stronger case for your paper would be either: 
1) propose InstructAV as a data collection method where you show that using the selected samples, fine-tuned models will result in better performance. This way, you are not claiming a new method and therefore don't have to explain why one language model performs better than the other.  Or, 
2) Have a stronger separation between classification only and clarification with explanation. I would suggest splitting the experiments (and table 2) into two parts: Classification where you show the performance on ALL the test samples like you did in the response below, and Classification+Explanation on the selected samples where you compare only with methods that provide explanation. In step two, you can justify the drop in performance when other language models by offering an explanation to the results. In Authorship Analysis, it is a well-known phenomena that explainable methods have lower performance and SOTA methods are harder to explain (especially to the layman person).  

Q3 & W1.1) Thank you. This is clear from the results, but why? is it because of the model size? or the characteristics of the model? If I want to use your method with another model, how do I choose that model? just the biggest model available? I am interested in more than describing what the table shows. 

[Q4 & W1.2] That's a strong claim. You cannot infer that without sharing descriptive stats of the data, e.g. avg length in words for example. 

[W2: ChatGPT can correctly determine the label, it can also provide a possible explanation.] I respectfully disagree. Please let me clarify that for authorship analysis tasks "humans" do not do well compared to computational approaches. (see Stamatatos  99, for example). Therefore, when you generate an explanation such as "uses passive voice", or "passionate about the topic" or some generic explanation, humans would tend to agree unless such note is completely wrong. What matters more in authorship analysis is the frequency of such features. What I was trying to say that, human evaluation in your case is a high-level check that whatever ChatGPT generated is actually in the text but not more than that. It is very different from the typical "human evaluation" for other tasks. Even BERTScore, it is just showing that similar models generated similar text and that on its own has its problems.  

Stamatatos, E., Fakotakis, N., & Kokkinakis, G. (1999, June). Automatic authorship attribution. In Ninth conference of the European Chapter of the Association for Computational Linguistics (pp. 158-164)
Dear Reviewer upBC,

We hope that you've had a chance to read our response. We would really appreciate a reply to let us know if our response and clarifications have addressed the issues raised in your review, or if there is anything else we can address.

Sincerely,

Authors
The potential risk of data leakage is not discussed. The data might have been included in the LLMs' training corpus. I wonder if there are some tests showing the data memorization possibilities and how the authors address the potential data contamination problem.