InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification
Authors: Yujia Hu
Zhiqiang Hu
Chun Wei Seah
Roy Ka-Wei Lee
Venue: COLM (2024)
Keywords: Authorship Verification, Parameter-Efficient Fine-Tuning (PEFT), InstructAV
TL;DR: We propose the InstructAV framework for Authorship Verification (AV) tasks to accurately determine whether two texts share the same author and to furnish robust linguistic explanations for the AV outcomes.

Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in a wide range of NLP tasks. However, when it comes to authorship verification (AV) tasks, which involve determining whether two given texts share the same authorship, even advanced models like ChatGPT exhibit notable limitations. This paper introduces a novel approach, termed  InstructAV, for authorship verification. This approach utilizes LLMs in conjunction with a parameter-efficient fine-tuning (PEFT) method to simultaneously improve accuracy and explainability. The distinctiveness of InstructAV lies in its ability to align classification decisions with transparent and understandable explanations, representing a significant progression in the field of authorship verification. Through comprehensive experiments conducted across various datasets, InstructAV demonstrates its state-of-the-art performance on the AV task, offering high classification accuracy coupled with enhanced explanation reliability.

Published as a conference paper at COLM 2024
InstructAV: Instruction Fine-tuning Large Language Models for Authorship Verification
Yujia Hu ; Zhiqiang Hu Chun-Wei Seah, Roy Ka-Wei Lee Information Systems Technology and Design Singapore University of Technology and Design {yujia hu, chunwei seah, roy 1ee ] edg. (zhiqiang hu huj@mymail. sutd. outd.edu.sg... edu.sg
Abstract
Large Language Models (LLMs) have demonstrated remarkable proficiency in a wide range of NLP tasks. However, when it comes to authorship verifi cation (AV) tasks, which involve determining whether two given texts share the same authorship, even advanced models like ChatGPT exhibit notable limitations. This paper introduces novel approach, termed InstructAV, for authorship verification. This approach utilizes LLMs in conjunction with a parameter-efficient fine-tuning (PEFT) method to simultaneously improve accuracy and explainability. The distinctiveness of InstructAV lies in its ability to align classification decisions with transparent and understandable explanations, representing significant progression in the field of author- ship verification. Through comprehensive experiments conducted across various datasets, InstructAV demonstrates its state-of-the-att performance on the AV task, offering high classification accuracy coupled with enhanced explanation reliability.
1 Introduction
Authorship Verification (AV) is a task aimed at determining if two texts were written by the same author, with significant implications across forensics, literature, and digital security domains (Halvani et al. 2019: Stamatatos] 2016). Traditionally, AV relied on stylometric analysis, utilizing linguistic and stylistic features, such word and sentence lengths, and
function word frequencies, to distinguish between authors (Seroussi et al. 2011 Bevendorif al. 2019 However, the advent of machine learning, particularly deep learning models like BERT Devlin et al., 2018) and RoBERTa (Jones et al., 2022), has revolutionized this field. These modern approaches, leveraging complex patterns in text, have shown superior performance over conventional stylometric techniques in identifying authorship (Saedi & Drras 2021 Konstantinou et al. 2022 Middez-Vleezuela et al. 2023). This paradigm
shift underscores significant evolution AV methodologies, em emphasizing the increasing effectiveness of machine learning in text analysis (Zheng & Jin 2023).
While current AV models have made notable advancements, they predominantly focus on binary classification and notably lack in providing explanatory insights. Explainability is not only of academic interest. It is fundamental to understanding model's mecision-making logic, and it also enhances trust and reliability in the model's output. Additionally, the lack of clear explanations makes hard to find and fix any biases that may be hidden inside these models, creating a significant problem for ensuring they are fair and unbiased. Therefore, it's critical for Al models not only to be accurate but also to provide transparency and interpretability.
This paper presents the InstructAV framework, an innovative approach tailored for AV tasks. Unlike existing models, InstructAV is designed to accurately verify authorship across texts while concurrently furnishing detailed linguistic explanations for its determinations.
Equul contribution

---

Published as a conference paper at COLM 2024
A key feature of InstructAV is its unique capacity to integrate explainability directly into the classification process, thereby creating a direct pathway between making accurate predictions and offering deep explanations. Through rigorous testing across three diverse AV datasets, the InstructAV framework has demonstrated not only outstanding accuracy in authorship verification but has also set new benchmark by producing coherent and substantiated explanations for its findings. This dual capability of InstructAV-merriinn high predictive performance with actionable imsights-marks a significant leap forward in the AV domain, contributing both to the enhancement of model transparency and the advancement of explainable artificial intelligence.
Our contributions can be summarized as follows: (i) We propose the InstructAV framework for AV tasks to accurately determine whether two texts share the same author and to furnish robust linguistic explanations for the AV outcomes. (ii) We have curated three instruction- tuning datasets, each accompanied by dependable linguistic explanations for AV tasks. These datasets are intended to serve as valuable resources for advancing research in this field"] (iii) Both automated and human evaluation results demonstrate the effectiveness of the InstructAV in providing precise AV predictions and reliable linguistic explanations.
2 Related Work
2.1 Authorship Verification
In the last two decades, AV has evolved significantly, transitioning from traditional methods focusing on linguistic features like spelling and style to machine learning techniques (Boen- ninghoff al. 2019). However, traditional machine learning, such as support vector machines, showed limited effectiveness (Konstantinou et al. 2022). Recent advancements involve contextual embeddings from language models like BERT, 15, and MPNET (Devlin eet al. 2018 Raffel et al. |020 Song et al. 22020), and further studies have explored graph convolutional networks and BIL.STM with attention mechanisms (Valdez- Valenzuela al. 2023 Sun et al. 2023). Moreover, Huang et al. (2024) explored different representations of authorship to verity their effectiveness in encoding writing styles. They concluded that authorship representations might be expected to remain robust against certain types of data
shifts.
Previous neural network-based methods, such as BERT and MPNET, have been crucial in advancing classification tasks, including AV. However, these models inherently lack mechanisms to elucidate their decision decision-making processes, critical gap as the demand for explainability in AI grows Hung et al. (2023). Addressing this, Hung et al. (2023) intro- duced PromptAV, an innovative technique that leverages the ca capabilities of large language models (LLMs) for AV, employing step-by-step stylometric explanation prompts to en- hance interpretability. Their findings demonstrate that PromptAV significantly outperforms traditional approaches like chain of thought (CoT) prompting (Wei et al.) (022) and PS+ prompting (Wang et al., (202)) in both accuracy and interpretability, marking a noteworthy advancement in the application of LLMs to AV tasks. [Huang et al. (2024) utilized LLMs with the Linguistically Informed Prompting (LIP) technique for authorship verification, revealing that even without domain-specific fine-tuning, the LIP method guides LLMs to satisfactory performance.
Despite the progress, these methods' reliance on few-shot demonstration model poses challenges in ensuring the consistency and relevance of their explanations. This limitation underscores a critical need for more robust solution that can deliver both precise classifica- tion and meaningful explanations across a broader range of scenarios. To address this need, we introduce the InstructAV framework, which adopts instruction fine-tuning approach to significantly enhance the classification accuracy and explanation quality in AV tasks. By refining the model's ability to generate relevant and consistent explanations, InstructAV not only builds on the foundation established by PromptAV but also addresses its primary limitations, offering a comprehensive solution that advances the field of explainable AV.
1 The code and datasets can be found at athttpa: htttpaa://gicubb.co///Siiiii.
(ttddio/InttructV
2

---

Published as a conference paper at COLM 2024
InstructAV
STEP 1 DATA COLLECTION
STEP 2 CONSISTENCY VERIFICATION
FINETUNING MODELS
Classitication + Explanation:
Output Consistency:
The following Texti and Tex2 are written by the Please anstyze their styles
anower is yes.
Pre-Trained LLMs
expain why
Explanation; Analyzing Text and Text based on the listed wrtting style characteristios, wo tnd the rolloing simlianttess
LORA
Texd 1: Marra's i. homemade remedy for laughal Mamn Harper....
Texd 2:
Ball Rementer his
willien by the same sutPor.
show was ona of the Trat...
Output Layer
Please follow the tormat of demostrations
method
Clesse Ficationn The conres
[Demo
2]
Explanation: Analycing Text mentioned witting style characterisdes, we find the following differences:
besed on
Classification: yes writing
Output:
Hyles Text 1 and Text
Explanation: Andlyting
The correct answer is yeaino. Andlycinn isted within
condusiion, evident that Text Text different authors.
Text
style charactericties cimfont enttrorneen .....
folloiing
Exomples
True
(Irput) (Oupul))
[treruction) )
Datasets:
['Oupuf:yes same. no difereett]
Consistent Dutasets:
[Inoutt:]
[Instrurion:]
Automatic Evaluation
Figure 1: An illustration InstructAV's architectures.
2.2 Parameter-Efficien Fine-Tuning of Large Language Models
The emergence of LLMs like GPT-3 represents significant advancement in AI, expanding the capabilities of machine learning technologies. However, deploying these models poses challenges due to their high computational and memory demands. Parameter-Efficciet Fine-Tuning (PEFT) addresses these issues. It selectively adjusts a small subset of model parameters, customizing it for specific downstream tasks with minimal resource usage Ding
et al. 2023) Liu et al. (0022).
Within the PEFT paradigm, adapters are notable. These compact modules integrate into pre-trained networks, providing a resource-efficient way to customize models for specific tasks datasets (Hu et al., 2023). By enabling targeted specialization without extensive retraining, adapters expand the accessibility of LLMs for various applications, enhancing their utility and flexibility.
The Low-Rank Adaptation (LoRA) adapter, a notable PEFT method, fine-tunes model weight matrices with low-rank modifications to boost task-specific trainability while retain- ing model strengths (Hu et al. [2021). Another method, Prefix- Tuning, adds task-specific parameters to transformer layers for alignment (Li & Liang 2021). These methods merge adaptability with computational efficiency, enabling advanced LLMS to operate in resource- limited settings. LoRA, especially, preserves LLMs'' general capabilities while enabling task-specific fine-tuning, marking a significant advancement in the field (Ding et al., 2023). In our study, we outperform AV tasks with LoRA fine-tuning.
3 Methodology
The overview of the InstructAV framework, as depicted in Figure 1 comprises three primary steps: data collection, consistency verification, and the fine-tuning of LLMs using the LoRÁ method (Hu et al. 2021). Initially, the framework focuses on the aggregation of explanatory data for AV samples. This approach uses the binary classification labels available in existing AV datasets. Following this, strict quality check is implemented, aimed at verifying the alignment and consistency of the explanations with the corresponding classification labels. The final stage involves the synthesis of instruction-tuning data, which is fusion of the gathered classification labels and their associated explanations. This composite data then serves as the foundation for fine-tuning L.LMs in conjunction with the LoRA adaptation
3

---

Published as a conference paper at COLM 2024
technique. This approach ensures that the LLMs are not only accurately fine-tuned for the AV task but also enhanced in their capacity to provide coherent and reliable explanations for their predictions. The subsequent sections present the details of each component in InstructAV.
3.1 Explanation Data Collection
To augment the explanatory capabilities of the InstructAV model, particularly in generating dependable explanations for AV predictions, we initiated a comprehensive collection of explanations using ChatGPT, with the crucial step of informing ChatGPT about the classifi- cation labels beforehand. This process involved three datasets widely used in AV studies: the IMDB dataset (Seroussi et al. 2014), the Twitter dataset (Schwartz et al. 2013), and the Yelp Reviews datasel This selection strategically covers various dimensions of textual data, thereby ensuring a diverse and comprehensive analysis.
The IMDB dataset, characterized by its longer text length (averaging 303 words), exemplifies long-form content. In contrast, the Twitter dataset, with an average text length of 16 words, epitomizes short-form content. The Yelp Reviews dataset, averaging 154 words per text, represents medium-length content. These datasets, initially curated for authorship attribution tasks, were adapted for our study. We extracted 11,000 samples from each dataset to facilitate a robust AV evaluation. Each sample comprises two texts, either written by the same author or by different authors.
For generating explanations, we employed ChatGPT with true classification labels and few-shot prompts, focusing on 11 linguistic features for each AV sample. These features, identified in the research by (Boenninghoff et al. 2019), are crucial for analyzing writing styles in textual content. The linguistic teatures generated by known-label ChatGPT were recorded as labels for the AV explanations in our dataset if they accurately predict and meet the consistency check.
To link classification labels closely with linguistic explanations during the data collection phase, we crafted prompts that incorporate the classification labels. For example, a prompt might state "The following Textl and Text2 are written by different authors. Please analyze their writing styles and explain why they are written by different authors." " to guide the explanation process. A detailed example of the explanation generation prompt is shown in Appendix [B.2] Table 8 These prompts enhance the relevance of the collected explanations, thereby improv- ing the explanatory capacity of the InstructAV model.
Consistency Verification. Alignment between classification labels and their linguistic explanations is essential for explanation data integrity and trustworthiness. Models like known-label ChatGPT are skilled at generating explanations but face challenges in align explanations with classification labels. For instance, despite being informed that "The following Textl and Text2 are written by the same author," ChatGPT might incorrectly respond with, "The correct answer is no." Instances like this create a mismatch between explanations and classification labels, reducing trust in the model's explanations. To enhance user trust in automated decisions, it is important to guarantee the consistency and reliability of both classifications and explanations. As shown in Figure 1 we employ a consistency verification method to verify the alignment between the model's analytical explanations and its classification decisions. We have instituted a comprehensive verification process to ensure the explanations are consistent with the classification. This process leverages demonstration templates that inherently guide the model to incorporate specific expressions -ucchnnssssch as 'samesiimilarities or different/äiffrrncee'' into its outputs. During the consistency verification stage, keyword searching was performed on terms within the generated text. Matching the phrases 'written by the same author' (resp. 'written by different authors') with classification labels allows us to assess the quality of ChatGPT's linguistic explanations.
The key to this process is the construction of instruction-tume purpose: facilitating AV classification prediction and supporting the generation of explana-
data, which serves a dual
tions. An example of the instruction-tuning data is shown in Appendix B.3 Table 9 This
2https:/wwww.yllp.co dataset
4

---

Published as a conference paper at COLM 2024
Dataset
#Authors
#Train
#Test
Avg length
IMDB
64
10,000
1,000
303
Twitter
100
10,000
1,000
16
Yelp
1,000
10,000
1,000
154
Table 1: Dataset statistics for IMDB, Twitter and Yelp.
Dataset Setting
Model
IMDB
Twitter
Yelp
BERT
0.677 (0.0124)
0.702 (0.002T)
0.622 (0.0020)
DistilBERT
0.526 (0.0125)
0.575 (0.0065)
0.543 (0.0036)
ALBERT
0.642 (0.0040)
0.701 (0.0023)
0.601 (0.0020)
LIP (GPT-4-turbo)
0.732 (0.0070)
0.612 (0.0040)
0.632 (0.0036)
Classification
(LLaMA-2-70B)
0,533 (0.0080)
0.554 (0.0042)
0.528 (0.0040)
LIP (Mistral-7B)
0,507 (0.0038)
0.539 (0.0025)
0.527 (0.0032)
InstructAV (LLaMA-1-7B)
0.648 (0.0236)
0.610 (0. 0062)
0.542 (0.0031)
InstructAV (OPT-6.7B)
0.590 (0.0050)
0.524 (0.0110)
0.527 (0.0060)
InstructAV (LLaMA-2-7BB
0.914 (0.0046)
0.740 (0.0070)
0.689 (0.0025)
PromptAV-2shot (GPT-3.55
0.623 (0.0397)
0.628 (0.0147)
0.534 (0.0064)
PromptAV-4shot (GPT-33.5)
0.635 (0.0265)
0.667 (0.0163)
0.544 (0.0080)
PromptAV-8she (GPT-3.55)
0.601 (0.0070)
0.648 (0.0U75)
0.564 (0.0081)
Classification
PromptAV (GPI-4-Turboo)
0.755 (0.0075)
0.729 (0.0070)
0.597 (0.0065)
Explanation
InstructAV (L--MA-)-7BB
0.825 (0.0289)
0.625 (0.0065)
0.596 (0.0104)
InstructAV (OPT-6.7B)
0.744 (0.0095)
0.714 (0.0070)
0.575 (0.0140)
InstructAV (LLaMA-2-7B)
0.937 (0.0017)
0.745 (0.0063)
0.693 (0.0442)
Table 2: Classification Accuracy of InstructAV and the baselines on different dataset settings. Highest Acc are bolded. All experiments were repeated three times. The table lists the average values of the three repetitions, while the standard deviations of accuracy from the experiments are provided in brackets.
verification step is essential for ensuring that the InstructAV model is not only accurate in its predictions but also capable of generating explanations that are coherent, relevant, and aligned with the classification outcomes, thereby enhancing the overall efficacy and reliability of the InstructAV framework.
3.2 Fine-tuning with LoRA
Adapting LLMs for AV tasks can demanding process, particularly due to the significant computational resources and extensive labeled data typically required for fine-tuning. To address this challenge, we have incorporated a PEFT method known as LoRA (Hu et al., 2021 into our approach for adapting LLMs to AV tasks.
LoRA presents a novel approach to updating weight matrix parameters. decomposes the matrix into two low-rank matrices, reducing trainable parameters while processing high-dimensional matrices effectively. During the forward pass, LoRA matrices compute rank-deco weights, used in attention or feed-forward activations. Importantly, this method keeps the LLM's core parameters unchanged, enabling adaptation to new data distributions or tasks without overhauling the model completely.
The LoRA architecture is encapsulated in the following equation:
= Wox + AW,x Wax + BAx
(1)
where Wa E R is pre-trained weight matrix in LLMs, which is associated with two adjustable low-rank matrices, E daxr and A E Rixd, d corresponds to the hidden dimension of the attention layers, and is the adaptation rank, which is selected such that & d. Within the training phase, Wj remains frozen, exempt from gradient updates, whereas A and B are dynamic, containing the parameters subject to training.
5

---

Published as a conference paper at COLM 2024
4 Experiments
4.1 Experiment Settings
Datasets. The InstructAV framework evaluation utilized three distinct Authorship Verifica- tion (AV) datasets: IMDB62 (Seroussi et al. 2014), Twitter (Schwartz et al. 2013), and Yelp Reviews chosen for diversity and relevance.
To evaluate the AV classification and explanation tasks performance, we constructed two distinct types of dataset settings, each incorporating varying levels of information:
1. Classification: This dataset involves integrating a question alongside two texts as the input and employs the LoRA method to fine-tune the model for classification tasks. The expected output a straightforward binary classification indicating whether the two texts are written by the same author, formatted as "The correct answer is yes/no".
2. Classification and Explanation: In this setting, we augment the classification data with linguistic analysis to empower the model to generate robust explanations for the AV predictions. The LLMs are fine-tuned to not only predict the classification labels, but also provide an analysis of eleven linguistic features of the two texts. This added layer of analysis offers reasoned explanation behind the classification decision, thereby enhancing the model's interpretability and reliability.
For the Classification setting, We randomly sampled 11,000 balanced samples from the IMDB62, Twitter, and Yelp datasets, respectively. The data was then divided into a training set comprising 10,000 samples and test set comprising 1,000 samples, with both sets maintaining balanced class distributions.
For the Classification and Explanation setting, we initially sampled 20,000 examples from each dataset and used the GPT-3.5-turbo API for linguistic analysis. To ensure the high quality of the generated explanations, samples with incorrect linguistic analysis were dropped during the initial consistency verification phase. For comprehensive evaluation, balanced subsets consisting of 10,000 training samples and 1,000 testing samples were randomly selected from the verified examples within each subset. These samples formed instruction-tuninn data, including text and corresponding linguistic explanations (see Section 3.1). Details regarding dataset statistics and characteristics, including the splits between training and test sets, are presented Table 1
These two dataset settings enable a thorough investigation into how the addition of explana- tory components influences the performance of the classification task, providing insights into the efficacy and adaptability of the InstructAV framework.
Baselines. For the Classification task, our baseline models comprised BERT (Dv lin et al. 2018) and its variants, DistilBERT (Sanh et al. 2019), and AIBERT (Lan et al. 2019). These models were selected for their widespread usage and proven effectiveness in AV classifi- cation tasks (Brad et al. 2021 Tyo et al. 2022 Fabien et al. 2020). However, these models are not inherently designed for text generation tasks. Another baseline related to LLMs is the application of the LIP method proposed by Huang et al. (2024) on GPT-4-turbo, LLaMA-2-7UB, and Mistral-7B.
For tasks involving Classification and Explanation, autoregressive models like GPT, which generate text sequences based on preceding text, are suitable. We adopted the PromptAV methodology on GPT-3.5-Turbo as baseline for the ChatGPT model, as proposed by (Hung et al. 2023). Although PromptAV may be slightly less effective than BERT in classification accuracy, it offers advantages such as bypassing extensive training and generating explana- tions. Additionally, we integrated PromptAV, using GPT-4 as the foundational model, to establish more competitive baseline.
Phttps//wwwwwyell..cmm dataset

---

Published as a conference paper at COLM 2024
IMDB
Model
ROUGE-1
ROUGE-2
ROUGE-L
Bert_Score
PromptAV-2shot (GPT-3.5)
0.379
0.147
0.227
0.844
PromptAV (GPT-4)
0.496
0.193
0.237
0.861
InstructAV (LLaMA-1)
0.677
0.412
0.496
0.898
InstructAV (OPT)
0.656
0.403
0,482
0.893
InstructAV (LLaMA-2)
0.689
0.434
0.515
0.907
Twitter
PromptAV-2shot (GPT-3.55
0.445
0.203
0.287
0.856
PromptAV (GPT-4)
0.510
0.307
0.466
0.860
InstructAV (LLaMA-1)
0.670
0.406
0.522
0.899
InstructAV (OPT)
0.644
0.398
0.476
0.897
InstructAV (LLaMA-2)
0.689
0.420
0.542
0.904
Yelp
PromptAV-2shot (GPT-3.5)
0.377
0.142
0.243
0.856
PromptAV (GPT-4)
0.431
0.162
0.402
0.867
InstructAV (LLaMA-1)
0.666
0.407
0.524
0.906
InstructAV (OPT)
0.629
0.377
0.486
0.900
InstructAV (LLaMA-2)
0.716
0.429
0.592
0.912
Table Automatic Metric Performance for InstructAV and PromptAV. Higher Acc are bolded.
Our comparative analysis evaluates both classification accuracy and the quality of explana- tions from InstructAV and PromptAV. This comprehensive assessment aims understand the strengths and potential limitations of each approach in AV tasks.
Implementation. In our research, we employed three different LLMs to validate the efficacy of the InstructAV framework. These models include OPT-6.7B (Zhang et al., 2022), LLaMA- 1-7B (Touvron et al. 2023a), and LLaMA-2-7B [Tourron et al., (0236). Our experiments were conducted using two NVIDIA A6000 GPUs, encompassing both the fine-tuning and inference phases.
For fine-tuning, applied the LoRA approach with rank set to 8. This introduced 4.1 million parameters, representing only 0.06% of the total billion parameters of the base models. This parameter increase highlights LoRA's effectiveness fine-tuning method. Fine-tuning was performed over 3 epochs using the LLM-Adapters Toolkit (Hu et al., 2023), tailored for integrating adapters into LLMs.
offering an efficient
During inference, all operations were performed deterministically with fixed temperature of 0.1. This maintains consistency and reliability in the model's performance, crucial for accurately assessing the InstructAV framework's capabilities. The selection of diverse LLMs and the strategic application of LoRA for fine-tuning effectively showcase the capability of the InstructAV framework across various model architectures. To ensure the reproducibil- ity of the experiments conducted using the InstructAV framework, each experiment was repeated three times with different random seeds. The mean and standard deviation were calculated from these repetitions.
4.2 Evaluation Metrics
The evaluation the InstructAV framework involves both AV classification and explana- tion generation tasks. This dual-focused approach enables an in-depth assessment of the framework's capabilities key areas of AV.
For AV classification tasks, accuracy serves as the primary metric. This metric effectively measures the models' proficiency determining whether two texts have been written by the same author. To evaluate the framework's performance in generating explanations for
7

---

Published as a conference paper at COLM 2024
IMDB
Model
Coverage
Relevance
Reasonableness
Persuasiveness
Explanation Label
11
4.42
4.12
4.22
PromptAV-2shot (GPT-3.5)
3.8
3.84
3.26
PromptAV (GPT-4)
3,92
3.87
3,56
InstructAV (LLaMA-2-7B)
11
4.24
3,98
4.03
Twitter
Explanation Label
4.66
4.42
4.65
PromptAV-2shot (GPT-3.5)
3.01
3.24
2.94
PromptAV (GPT-4)
3.67
3.22
3.27
InstructAV (LLaMA-2-7B)
11
4.48
4.25
4.28
Yelp
Explanation Label
4.56
4.44
4.65
PromptAV-2she (GIT--3.5)
7
2.87
3.02
3.06
PromptAV (GPT-4)
7
3.75
3.41
3.50
InstructAV (LLaMA-2-7B)
4.3
4.32
4.36
Table 4: Results of the human evaluation on explanation labels and explanations generated by the InstructAV and PromptAV frameworks.
AV tasks, a distinct approach is required. Here, the focus shifts towards assessing the quality of the linguistic analysis generated by the models. Unlike accuracy, which is quantitatively measurable, the quality of linguistic explanations presents a more complex evaluation challenge, as it involves subjective elements related to the coherence, relevance, and clarity of the explanations. Given the complex nature of evaluating explanation quality, we adopt dual evaluation approach, employing both automatic and human evaluation metrics.
A Automatic Evaluation for Explanations. In our automated assessment of explanation quality for both InstructAV models and baseline models, we measure the similarity between generated and labeled explanations, using ChatGPT-gennrated texts standards. Through consistency verification, we ensure explanatory labels within test sets consistently align with classification labels, affirming their accuracy. Consequently, linguistic analysis as explanation labels. We employ metrics like ROUGE-1, ROUGE-2, ROUGE-L (for content coverage and structural fluency), and BERT Score (for semantic quality). Higher scores indicate better explanation quality, ensuring a comprehensive evaluation based content accuracy, logical coherence, and contextual relevance.
consider ChatGPT's
Human Evaluation for Explanations. To supplement the automated metrics for evaluating explanation quality in the InstructAV framework, also implemented a human evaluation. This approach involved randomly selecting 100 examples from each of the IMDB, Twitter, and Yelp Reviews datasets for qualitative assessment by human evaluators. These evalua- tions included not only explanations generated by InstructAV and baseline models, but also those explanation labels. For the human evaluations focusing on explanations, we selected InstrctAV (LLaMA-2-7B) along with the two most robust baselines, namely PromptAV-2sho equipped with GPT-3.5 and GPT-4 as the base models. Three evaluators were enlisted to independently evaluate each explanation across four criteria:
Coverage: Evaluating how many of the anticipated linguistic features are present the explanations. Evaluators checked the number of features covered in each explanation, with baseline models like PromptAV covering 0 to 7 features, and InstructAV, along with explanation labels, extending from 0 to 11 features.
Evaluators scored the relevance of the explanations to the original texts on 5-point Likert scale, ranging from 1 (completely irrelevant) to 5 (perfectly relevant).
Relevance:
This involved assessing the logical soundness of each feature's 's explanation, using 5-point Likert scale, from 1 (totally unreasonable) to 5 (completely reasonable).
Reasonableness:
Persuasiveness:
Evaluators rated the persuasiveness of the explanations on a scale from (completely not persuasive) to 5 (highly persuasive).
8

---

Published as a conference paper at COLM 2024
4.3 Experiment Results
Classification Results. We evaluate the InstructAV framework for AV tasks, utilizing both classification and explanatory dataset settings. The primary goal is to explore how linguistic analysis can improve the model's performance in AV classification tasks. The results, detailed in Table include the accuracy of InstructAV with various LLMs and baseline models. A key finding is that InstructAV, particularly when paired with LLaMA-1-7B and LLaMA-2-7B, outperforms baseline models in all datasets when using only classification data. Notably, with LLaMA-2-7B, InstructAV achieves 25.2% improvement over the highest- performing baseline, BERT, on the IMDB dataset.
It's imperative to underline that in our experimental setup, models were evaluated on their ability to concurrently generate classification predictions and conduct linguistic analysis without incorporating explanation data in the input. This methodology ensured fair com- parison with approaches that focus only classification. Our results clearly demonstrate that all variants of the InstructAV framework, utilizing different LLMs, significantly benefit from training on explanatory labels. Notably, InstructAV with LLaMA-2-7B showcased remarkable 27.1% improvement in classification accuracy over the PromptAV approach using 2-shot prompts based on GPT-3.55. Furthermore, when compared to PromptAV-2shot employing GPT-4, InstructAV exhibited superior performance across all three evaluated datasets. These findings highlight the substantial benefits of incorporating explanatory training, indicating that providing InstructAV with the dual function of AV classification and generating linguistic explanations significantly boosts its classification precision.
Automatic Evaluation Results on Explanations. Our InstructAV framework demonstrates that training in explanation labels can significantly improve AV classification performance. Additionally, we aim to assess whether InstructAV can generate high-quality explanations. For this purpose, we selected PromptAV-2shot (GPT-3.5), the best-performing variation of the framework, alongside GPTT-4, to benchmark against all variations of InstructAV. we subjected them to both automatic and human evaluations focused on the quality of their generated linguistic explanations. The results of the automatic evaluation are presented in Table 3 These results reveal that InstructAV consistently surpasses PromptAV models (GPT-3.5 and GPT-4) across all datasets and all evaluation metrics. Notably, ROUGE-1 and ROUGE-2 scores highlight InstructAV's superior performance in achieving content overlap at both the word and phrase levels. Moreover, the ROUGE-L metric indicates that InstructAV is more proficient in maintaining sentence-level structure and fluency. Furthermore, the BERT Score supports the observation that the explanations generated by InstructAV are semantically closer to the explanation labels. This comprehensive evaluation underscores In- structAV's capability not only in improving AV classification accuracy but also in generating linguistically coherent and contextually relevant explanations.
Human Evaluation Results on Explanations. To comprehensively evaluate the generated explanations and to assess the quality of the linguistic analysis produced by ChatGPT, which serves as our explanation labels, we have conducted a human evaluation using four key metrics: Coverage, Relevance, Reasonableness, and Persuasiveness. The results of this human evaluation are presented in Table Our findings from this human evaluation process show that our explanation labels achieves the highest scores. This result validates our methodological choice of using explanations generated by known-label ChatGPT as both source for training data and benchmark for explanation labels in our testing scenarios. Importantly, the results also reveal that InstructAV, particularly with the LLaMA-2-7D model, not only surpasses the performance of PromptAV-2shot models (GPT-3.5 and GPT-4) but also attains a level of explanation quality that is comparable to known-label ChatGPT. This outcome is significant as it demonstrates that InstructAV can produce explanations that are not only accurate but also contextually relevant, logically sound, and convincing to human evaluators. Such a capability is essential for applications where understanding the rationale behind model predictions is important the predictions themselves.
Correlation between Explanation and Classification. To explore the relationship between explanation quality and classification accuracy, we have selected two distinct subsets of InstructAV samples: the top 25% with the highest average human evaluation scores and the bottom 25% with the lowest average human evaluation scores. We then calculated the

---

Published as a conference paper at COLM 2024
IMDB
Twitter
Yelp
Top 25
0.92
0.8
0.84
Bottom 25
0.88
0.68
0.72
Table 5: Performance of InstructAV on Top 25 and Bottom 25 explanation. Higher accuracies bolded.
Model
IMDB
Twitter
Yelp
LLaMA-1-7B
0.000
0.000
0.000
0-shot
OPT-6.7B
0.000
0.000
0.000
LLaMA-2-7B
0.006
0.010
0.021
LLaMA-1-7B
0.189
0.097
0.226
2-shot
OPT-6.7B
0.258
0.179
0.297
LLaMA-2-7B
0.309
0.336
0.397
LLaMA-1-7B
0,014
0.020
0.127
4-shot
OPT-6.7B
0.191
0.220
0.217
LLaMA-2-7B
0.279
0.350
0.375
LLaMA-1-7B
0.003
0.007
0.002
8-shot
OPT-6.7B
0.005
0.008
0.003
LLaMA-2-7B)
0.020
0.033
0.025
InstructAV (LLaMA-1-7B)
0.648
0.610
0.542)
Finetuned
InstructAV (OPT-6.7B)
0.590
0.524
0.527
InstructAV (LLaMA-2-7B)
0.914
0.740
0.689
Table 6: Classification Performance of InstructAV Framework and LLMs in Few-Shot Con- texts Without Fine-Tuning, Highest Acc are bolded.
classification accuracy for each of these subsets, and the results are presented in Table 5 Our analysis clearly indicates that samples associated with higher quality explanations con- sistently achieve superior classification accuracy. This finding underscores the effectiveness of training InstructÁV to not only generate AV classification predictions but also provide linguistic explanations. Such training not only enhances the model's performance in AV classification tasks but also enhances its ability to produce valuable linguistic analysis.
5
Ablation Study
Ablation experiments were conducted to investigate potential data contamination issues, specifically whether the data might have been included in the training corpora of LLMs. We performed 0-shot, 2-shot, 4-shot, and 8-shot in-context tests on each dataset using the original, untuned LLaMA-2-7B, LLLMA-1-7B, and OPT-6.7B models. Each experiment was replicated three times, with the mean results presented in Table The results indicate that the original models struggle to perform AV tests effectively our dataset, especially in the 0-shot setting where the outputs are highly random. Both LLaMA-1-70 and OPT-6.7B failed to make correct judgments, and LLaMA-2-7B showed weak judgment capabilities. Consequently, we postulate that it is unlikely that our dataset was included the training data of the original models.
Conclusion
This research presents InstructAV, an innovative approach to AV tasks that leverages LLMs with a PEFT method. Our study establishes InstructAV as a significant advancement in the AV domain, showcasing its ability to enhance classification accuracy and provide clear and coherent explanations for its decisions. The contributions of this paper, including the development of the InstructAV framework, the creation of three instruction-tuning datasets with reliable linguistic explanations, and the demonstration of the framework's 's effectiveness through both automated and human evaluations, mark crucial progress in AV research. InstructAV, with its dual priority on high accuracy and the ability to provide high quality explanations, positions it as state-of-the-art AV solution.
10

---

Published as a conference paper at COLM 2024
References
Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Generalizing unmask- ing for short texts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Téchnologies, Volume (Long and Short Papers), pp. 654-659, 2019.
Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Robert M Nickel. Explainable authorship verification in social media via attention-based similarity learning. In 2019 IEEE International Conference on Big Data (Big Data), pp. 36-45. IEEE, 2019.
Florin Brad, Andrei Manolache, Elena Burceanu, Antonio Barbalau, Radu Ionescu, and Marius Popescu. Rethinking the authorship verification experimental setups. arviv preprint ar Xri::2212.05125, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding, arxiv preprint arXiv:1810.048 00.00000000 2018.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
Maël Fabien, Esaú Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. Bertaa: Bert fine-tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pp. 127-137, 2020.
Oren Halvani, Christian Winter, and Lukas Graner. Assessing the applicability of authorship verification methods. In Proceedings of the 14th International Conference Availability, Reliability and Security, pp. 1-10, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ar Xiv:2100 09685, 2021.
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. Lim-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arxiv preprint arXiv:2304.01993,, 2023.
Baixiang Huang, Canyu Chen, and Kai Shu. Can large language models identify authorship?, 2024.
Yu Hung, Zhiqiang Hu, Yujia Hu, and Roy Ka-Wei Lee. Who wrote it and why? prompt- ing large-language models for authorship verification. arviv preprint arXio=2310.08123, 2023.
Keenan Jones, Jason RC Nurse, and Shujun Li. Are you robert or roberta? deceiving online authorship attribution models using neural text generators. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pp. 429 440, 2022.
Stefanos Konstantinou, A Zinonos, and Li. Different encoding approaches for authorship verification. CLEF, 2022.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint ar XrXio:1909.11942, 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arv preprint ar arXiv:2101.00199, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Fho parameter-efficien fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
11

---

Published as a conference paper at COLM 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1),5485-5511 2020.
Chakaveh Saedi and Mark Dras. Siamese networks for large-scale author identification. Computer Speech & Language, 70:101241, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXio:1910.01108, 2019.
Roy Schwartz, Oren Tsur, Ari Rappoport, and Moshe Koppel. Authorship attribution of messoemessages. In Proceedings of the 2013 Conference on empirical methods in natural language processing, pp- 1880-1891, 2013.
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. Authorship attribution with latent dirichlet allocation. In Proceedings of the fifteenth conference on computational natural language learning, pp. 181-189, 2011.
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. Authorship attribution with topic models. Computational Linguistics, 40(2):269-110, 2014.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857-16877, 2020.
Efstathios Stamatatos. Authorship verification: a review of recent advances. Research in Computing Science, 123:9-25, 2016.
Efstathios Stamatatos, Nikos Fakotakis, and George Kokkinakis. Automatic authorship attribution. In Ninth conference of the European Chapter of the Association for Computational Linguistics, pp. 158-164, 1999.
Yitao Sun, Svetlana Afanaseva, and Kailash Patil. Stylometric and neural features combined deep bayesian classifier for authorship verification. Working Notes of CLEF, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar 1r21:21302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint XiX::2230 099880, 2023b.
Jacob Tyo, Bhuwan Dhingra, and Zachary C Lipton. On the state of the art in authorship attribution and authorship verification. arxiv preprint orXiv:2209.06899 2022.
Andric Faldee--alenzuela Jorge Alfonso Martinez-Galiche and Helena Gómez-Adorno. Heterogeneous- convolutional network for authorship verification. 2023.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot tfain-of-thought reasoning by large language models, 2023. http:ttps://aat www.//gg/////////oo///oooe.eee/t 2205 04091
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language 2022.
models. Advances in neural information processing systems, 35:24824-2444
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Xrv preprint arXivv2205.01068, 2022.
Wanwan Zheng and Mingzhe Jin. A review on authorship attribution in text mining. Wiley Interdisciplinary Reviews: Computational Statistics, 1582) 2023.
12

---

Published as a conference paper at COLM 2024
A Limitations
The InstructAV framework, while achieving state-of-the-art performance in AV classification and explanation tasks, does face a notable limitation in its current form. The human evaluations of model-generated explanations differ from typical evaluations in other tasks. Specifically, our evaluations are intended for high-level verification explanations produced by ChatGPT are coherent and align with the text, not as full assessment authorial features. Author analysis involves critical frequency and nuanced features better captured by computational methods then humans (Stamatatos et al. (1999). Theoretically, an ideal human evaluation would ask human participants to justity why two texts are written by the same author, but such research is costly and problematic, humans do not perform well in tasks that require analysis of writing styles. Considering these limitations, we consider ChatGPT-generas cost-effective alternative. Future work will consider a more suitable evaluation method that incorporates computational features.
ensure that the
explanations to be reasonable and
On the other hand, when tasked with generating both AV classification predictions and linguistic explanations, InstructAV is constrained produce a maximum of 512 new tokens. This restriction can lead to significantly longer inference times. For example, generating explanations for 1,000 samples in a long-text dataset like IMDB can take around 4-5 days, while in shorter-text datasets such as Twitter, this time is reduced to approximately 2 days. Addressing this challenge, future work will focus on developing methods to enhance the efficiency of the inference process within InstructAV, particularly in the context of generating explanations.
B
Appendix
B.1
Case Study for InstructAV
We have included randomly selected examples to showcase the capabilities InstructAV in generating both classification predictions and language feature-based explanations. One such example from the IMDB dataset is detailed in Table 7 This example demonstrates how InstructAV first provides a classification prediction and then follows it with detailed explanations anchored in specific language features pertinent to the classification.
These examples highlight InstructAV's proficiency in delivering accurate classifications paired with clear and credible explanations. Notably, the explanations are tailored to each specific sample, rather than being generic broad. This customization enhances the explanations' effectiveness, as they are directly correlated with the content of the original text. For instance, the model explicates features such as writing style and use of expressions by incorporating words and phrases directly from the text in question. This approach not only bolsters the relevance of the explanations but also their trustworthiness, providing users with insightful and contextually grounded rationale behind each classification decision made by InstructAV.
B.2 Prompt Templates
B.3 Collected Datasets
13

---

Published as a conference paper at COLM 2024
Instruction: Please decide if the following Text1 and Text2 are written by the same authors. 'yes' means from the same author, 'no' means not from the same author. Then, provide an analysis based on writing styles.
The British Soap Awards 2005 am a fan of only two soaps Coronation Street and EastEnders, , but am always going to watch this programme to see what the other soaps are like, if they deserve to win something and what may be missing out on always want to agree with the awards [Truncated due to length restriction]
Text
Text Father of the Bride As far as can work out,
after this film that the career of Steve Martin has had very few laughs, which a shame, he just needs to find the right film. Anyway this is the remake of the 1950 film and it sees middle aged father George Banks Martin ) with wife Nina ( Diane Keaton having to prepare the wedding for their 21 21-year-old daughter Annie Banks ( Kimberly Hilliams-Paisley ) ... [Truncated due length restriction]
Generated Output
The correct answer is yes. Explanation: Upon analyzing Text and Text based on the listed writing style characteristics, we find the following similarities that could suggest that both texts were written by the same author:
Classification:
Writing Style: Both texts provide personal opinions and evaluations of the subject matter. Text focuses on the British Soap Awards, while Text 2 provides review of the film \"Father of the Bride.)" ""
Both texts use colloquial expressions and idioms, such as \"if they deserve to win something)"" " in Text and \"he just needs to find the right film) " in Text 2. Tone and Mood: Both texts have an informal and conversational tone. The author expresses their personal preferences and opinions in both texts.
Expressions and Idioms:
Sentence Structure and Syntax: Both texts use a mix of compound and simple sentences. For example, ("I always want to agree with the awards that are given according to the clips elips/photos photos shown)"" in Text and ("Martin does do well, Keaton offers a good support, and Short adds giggles as the foreign wedding organiser\" in Text
Punctuation Style: Both texts consistent punctuation, including commas and full stops. Special Characters Style, Capitalization Style: Both texts capitalize proper nouns and show titles. However, Text uses unconventional capitalization for emphasis, such as \ \"Very good!! " Compound and Separate Spelling: No significant differences in spelling patterns are observed between the two texts.
Acronyms and Abbreviations: Both texts use acronyms and abbreviations, such as \"EE\" " for (EEstEEneers' " in Text and ".B.D.ongg" " for \BB.D. Wong "" in Text
Characters Style: Both texts mention specific actors and their roles, indicating a familiarity with the subject matter.
Diatopic Variations and Foreign Languages: Both texts appear to be written in standard English without diatopic variations foreign language references.
Other Relevant Aspects: Both texts provide specific details about the subject matter, indicating personal familiarity or interest in the topic. The author's 's opinions and evaluations are evident in both texts.
Table 7: A text pair example with the generated responses from InstructAV
14

---

Published as a conference paper at COLM 2024
Task: Textl and Text2 written by the same author. Please analyze their writing styles and explain why they are written by the same author. You can refer to the following characteristics of writingstyle style. 1. writing style. 2. expressions and Idioms. 3. tone and mood. 4. sentence structure and syntax. 5. punctuation style. 6. special characters style, capitalization style. 7. compound and separate spelling. 8, acronyms and abbreviations. 9, characters style. 10. Diatopic variations and foreign languages. 11. any other relevant aspect.
Text 1:
The British Soap Awards 2005 I am fan of only two soaps Coronation Street and East En- ders, but I am always going to wafch this programme to see what the other soaps are like, if they deserve to win something and what may be missing out on. always want to agree with the awards that are given according the clips / photos shown, and that is what also enjoy, watching back on the past year.. ... ['runccteed due to length restriction]
Text 2: Father of the Bride As far as I can work out it is after this film that the career of Strve Mar- tin has had very few laughs which is a shame, he just needs to find the right film. Anyway, , this is the remake of the 1950 film, and it sees middle aged father George Banks Martin with wife Nina Diane Keaton ) having to prepare the wedding for their 21 2i-year-ole old daughter Annie Banks Kimbeely Williams- Willians-Paisley )....
[Truncated due to length restriction]
Please follow the format of the analysis method in the demostrations, You will be given 2 demostra- tions.
### Demostration Start:
Text 1: Mama's 's homemade remedy for laughs! Mama Harper develops her mother's 's concoction to help her family get over an illness like the flu. After her son and bimbo-in-law Naomi who is now expecting their child realized that they can't afford it. They try a get rich scheme which falls on the unmentioned ingredient,
prroool.....fruncattl
due to length restriction]
Text 2:Not so Bad! Remember that this show was one of the first sitcoms in syndication in the 1980s. It was a new thing. When a sitcom or drama could not get a network, they went with syndication. I thought this show was alright in syndication because it reached a wide audience. I often saw it on Saturday evenings usually before Mama's 's Family. was a kid so didn't know better.... (Truncated due to length restriction]
Upon analyzing Text 1 and Text 2 based on the listed writing style characteristics, we find the following similarities that could suggest that both texts were written by the same author: Writing Style: Both texts seem to be reminiscing about older television shows. The author uses mixture of narrative and opinion in both texts, indicating a personal connection or memory attached to the subject matter.
Expressions and Idioms: The expressions "I miss this show" and "I don't 't recall ever not watching it" showcase a personal touch and sentimentality in both texts.
Tone and Mood: Both texts have a nostalgic and somewhat informal tone. The author shares memories and impressions about shows from the past, using phrases like "I was a kid so I didn't know better" and "Still, I miss this show."
Sentence Structure and Syntax: Both texts employ a mix of compound and simple sentences. For instance, "When sitcom or drama could not get network, they went with syndication" and "Mama Harper develops her mother's 's concoction to help her family get over an illness like the flu." " Punctuation Style: Both texts employ spaces before and after commas inconsistently. such as in "$180, 000" and "Marla Pennington, , Edie McClurg."
Special Characters Style, Capitalization Style: Both texts capitalize show titles like "Mama's Family" and show proper nouns. However, both texts sometimes lack proper capitalization at the start of sentences, such as "Still" in Text
Compound and Separate Spelling: No significant differences in spelling patterns are observed between the two texts. Acronyms and Abbreviations: The second text uses "1980s" to denote the decade, showcasing a preference for this kind of abbreviation. Characters Style: The character portrayal in both texts has a casual undertone. For example, "bimbo-in-law Naomi" and "the guy who played her father." Diatopic Variations and Foreign Languages: Both texts appear to be written in American English and reference American television shows.
Other Relevant Aspects: texts contain aneedotal or subjective experiences like "I was a kid so I didn't 't know better" and "Still, I miss this show." " The author provides specific details about show characters, indicating a personal familiarity with the subject. There's a recurring theme of syndication and older television programming in both texts, further suggesting the same author might have written them.
conclusion, based on the aforementioned similarities in writing style, expressions, tone, and other characteristics, it is plausible that Text 1 and Text 2 were written by the same author.
Table 8: Prompt used for generating explanation labels
15

---

Published as a conference paper at COLM 2024
Classification Datasets
Instruction: Please decide the following Text and Text 2 are written by the same authors. 'yes' means from the same author, 'no' means not from the same author.
Text 1: Send in the Clowns!!! 1 don't know why we haven't heard of Judy Cohen since this episode. She can break your heart singing Stephen Sondheim's s Send in the Clowns. She plays herself as an aspiring singer who beats herself down very much because she is overweight and not the classic Crouty....[Trunnatee due to length restriction]
Text 2: Could have been better! Columbo's Most Dangerous Match is about chess. An unlikely subject or motive for murder in my opinion. The star is Manchurian Candidate's Lawrence Harvey who plays deaf chess (larpion....TTruncateed
lTruncated due to length restriction]
Output: The correct answer is yes.
Classification & Explanation Datasets
Instruction: Please decide if the following Text and Text are written by the same authors. "yes' means from the same author, 'no' means not from the same author. Then, provide an analysis based on writing styles.
Text 1: Send in the Clowns!!! 1 don't know why we haven't 't heard of Judy Cohen since this episode. She can break your heart singing Stephen Sondheim's s Send in the Clowns, She plays herself as an aspiring singer who beats herself down very much because she overweight and not the classic beauty...... beauty.....Irunnatee [Truncated due to length restriction]
Text 2: Could have been better! Columbo's Most Dangerous Match is about chess. An unlikely subject or motive for murder in my opinion. The star is Manchurian Candidate's Laorence Harvey who plays deaf chess champion..... (Truncated (Trunccc.....Trunatteed due to length restriction]
The correct answer is yes. Analyzing Text and Text based on the listed writing style characteristics, we find the following similarities that could suggest that both texts were written by the same author:
Output:
Writing Style: Both texts provide detailed commentary and analysis of specific episodes of TV show. The author shares personal opinions and emotional responses the episodes. Expressions and Idioms: Both texts use phrases like "break your heart" and hell of game" to evoke emotional reactions and convey personal opinions.
Tone and Mood: Both texts exhibit enthusiasm and engagement with the subject matter. The author expresses admiration for certain performances and emphasizes their impact.
Sentence Structure and Syntax: Both texts use mix of simple and compound sentences. The sentences focus on describing specific events and interactions between characters. Punctuation Style: Both texts use consistent punctuation, including proper use of commas and ellipses. They enhance the flow and expressiveness of the narrative.
Special Characters Style, Capitalization Style: Both texts consistently capitalize proper nouns and show titles. The author pays attention to the details and presents them accurately. Compound and Separate Spelling: Both texts use standard spelling patterns without significant variations.
Acronyms and Abbreviations: Neither text employs acronyms or abbreviations. Characters Style: Both texts describe specific characters and their roles within the TV show episodes. The author demonstrates familiarity with the characters' interactions and motivations. Diatopic Variations and Foreign Languages: Both texts are written in English without reference to specific geographical locations or foreign languages.
Other Relevant Aspects: Both texts contain personal appeals to specific individuals, such "Where are you Judy Cohen?" and "You almost feel sorry for him." " The author shows deep connection to the subject matter and a desire for continued engagement.
In conclusion, based on the aforementioned similarities in writing style, expressions, tone, and other characteristics, it is likely that Text and Text were written by the same author.
Table 9; Example of collected Classification Datasets and Classification & Explanation Datasets
16


Reviews:
[{"review_id": "Uw8wojvPQH", "reviewer": "Reviewer_GRur", "summary": "This paper introduces InstructAV, a novel approach for authorship verification (AV) tasks using Large Language Models (LLMs) with parameter-efficient fine-tuning (PEFT). InstructAV aims to improve both accuracy and explainability in determining whether two given texts are written by the same author. The framework consists of three main steps: data collection, consistency verification, and LLM fine-tuning using the LoRA method. The authors curate three instruction-tuning datasets with reliable linguistic explanations and demonstrate the effectiveness of InstructAV through automated and human evaluations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback and questions.\n\n**[Q1]**\n\nSection 4.2 details \"Coverage, Relevance, Reasonableness, Persuasiveness.\"  To enhance clarity, we will include a summary of these metrics in the caption of Table 4.\n\n**[Q2]**\n\nOur baseline \"PromptAV\" is an LLM-based model referenced in [2]. We did not initially include the model in Huang et al. [3] since it was published after our initial experiments, but have now incorporated it. Utilizing the LIP prompting technique from [3], we performed additional tests:\n\n|BaselineModel|IMDB|twitter|yelp|\n|----|----|----|----|\nGPT-4Turbo+LIPprompt|0.739|0.616|0.633\nLLaMA-2-70B+LIPprompt|0.534|0.559|0.524\nMistral+LIPprompt|0.504|0.541|0.526\nOurs\nInstructAV(LLaMA-2-7B)|0.938|0.747|0.661\n\nInstructAV still outperforms all of these baselines. We will add these new results to our paper.\n\n\n**[Q3]**\n\nWe will include a GitHub link to our code and all datasets in the camera-ready version. \n\n**[Q4]**\n\nTo ensure our results' robustness, we repeated each experiment three times, calculating the standard deviation (SD) for each model. Below is a table showing average accuracies and SD (in brackets):\n\n|Model|imdb|twitter|yelp|\n|----|----|----|----|\nBert|0.677 (0.0124)|0.702 (0.0021)|0.622 (0.0020)\nDistilbert|0.526 (0.0125)|0.575 (0.0065)|0.543 (0.0036)\nAlbert|0.642 (0.0040)|0.701 (0.0023)|0.601 (0.0020)\nPromptAV-2shot (GPT-3.5)|0.623 (0.0397)|0.628 (0.0147)|0.534 (0.0064)\nPromptAV-4shot (GPT-3.5)|0.635 (0.0265)|0.667 (0.0163)|0.544 (0.0080)\nPromptAV-8shot (GPT-3.5)|0.601 (0.0070)|0.648 (0.0075)|0.564 (0.0081)\nPromptAV (GPT-4)|0.755 (0.0075)|0.729 (0.0070)|0.597 (0.0065)\nInstructAV (LLAMA-1-7B)|0.825 (0.0289)|0.625 (0.0065)|0.596 (0.0104)\nInstructAV (OPT-6.7B)|0.744 (0.0095)|0.714 (0.0070)|0.575 (0.0140)\nInstructAV (LLAMA-2-7B)|0.937 (0.0017)|0.745 (0.0063)|0.693 (0.0442)\n\nThe small SD values across all datasets indicate consistent experimental results.\n\n**[Q5]**\n\nWe tested statistical significance between the top-performing baseline model, PromptAV-2shot-GPT4, and our's InstructAV (LLaMA2-7B):\n\n||IMDB|Twitter|Yelp\n|----|----|----|----|\np-value|4.630e-8|1.597e-4|1.516e-4\n\nAll p-values are <0.05. This confirms significant improvements in our InstructAV model.\n\n**[Q6]**\n\nWe'll add the suggested references and conduct a detailed literature review in our updated paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pros:\n1. The InstructAV framework significantly improves AV classification accuracy compared to baseline models, particularly when paired with LLaMA-2-7B.\n2. The approach generates high-quality explanations for AV predictions, providing valuable insights into the model's decision-making process.\n3. The consistency verification step ensures alignment between classification labels and linguistic explanations, enhancing the reliability of the generated explanations.\n4. InstructAV demonstrates state-of-the-art performance on AV tasks across various datasets, offering both high classification accuracy and enhanced explanation quality.\n5. The curated instruction-tuning datasets with reliable linguistic explanations are valuable resources for advancing research in the AV field.", "reasons_to_reject": "Cons:\n1. Table 4 hard to understand. It is strongly suggested that the authors should explain the evaluation metrics for “Coverage Relevance Reasonableness Persuasiveness” clearly in the table. Otherwise, it is hard for readers to understand.\n2. The current baselines are old. It is strongly suggested that the authors include LLMs as baselines considering LLMs have achieve a good performance in Authorship Attribution[2,3]. Otherwise, it is hard to evaluate the effectiveness of the proposed method.\n3. No code is provided. It is strongly encouraged that the authors release the code for reproduction.\n4. Not sure whether the authors run the baselines multiple times to ensure the convincingness. If the authors run multiple times, it is suggested the authors provide more details.\n5. It is encouraged the authors conduct statistical testing for the experiments.\n6. Related works are incomplete [1,3,4,5]. It is strongly suggested that the authors conduct a comprehensive literature review.\n\n[1] VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Web https://arxiv.org/abs/2207.03477\n\n[2] Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification https://arxiv.org/abs/2310.08123\n\n[3] Can Large Language Models Identify Authorship? https://arxiv.org/abs/2403.08213\n\n[4] Can Authorship Representation Learning Capture Stylistic Features? https://arxiv.org/abs/2308.11490\n\n[5] Contrastive Disentanglement for Authorship Attribution https://dl.acm.org/doi/10.1145/3589335.3652501", "questions_to_authors": "How do the authors address the potential data contamination problem?\n\nThe potential risk of data leakage is not discussed. The data might have been included in the LLMs' training corpus. I wonder if there are some tests showing the data memorization possibilities and how the authors address the potential data contamination problem.", "extra_scores": {}}, {"review_id": "J7ERwU2rb1", "reviewer": "Reviewer_upBC", "summary": "## Summary\nThe paper proposes a new dataset that can be used for instruction fine-tuning for LLMs. As per my understanding, the process (briefly) goes as follows. \n1) Collecting samples from three AV datasets.\n2) Providing ChatGPT with a true label and ask it to generate explanation based on a specific template that includes 11 dimensions.\n3) Verification of alignment between model answer about the similarity and the provided explanation. (**Not sure what happens with examples that are not-aligned).\n4) Using the resulting dataset to fine-tune models (With LoRA)\n5) Evaluation of the generated output (classification + explanation) using automatic metric and human evaluations. \n\n### Quality\nThe authors investigate different aspects of evaluations for the performance. This includes both the quantitative and the qualitative evaluations.  \n\n### Clarity\nThere are a few details in the methodology that are not clear. Additionally, there are a few details that, IMO, would greatly enhance the readability of the paper:\n1) Using the correct citation format across the paper. (Author's names should be inside the parentheses, except for cases when we are referring to the author's and not the work. E.g. _More details can be found in (Author et al 2020)_ **vs**. _Author (2020) discussed this_. \n2) Giving more informative table captions.  Tables 2-5 all start with **performance of InstructAV on ...*. Instead, relevant captions such as Classification Accuracy, Token-overlap, ... etc.  \n3) In page 1, the last line: \"*accuracy in authorship identification but ..*\". This is probably a typo, authorship identification is a different task. Authors should avoid mixing up the two. \n4) CoT and PS+ need citations. \n5) Table~2: The caption mentions 2 datasets but the table shows 3.\n\n### Originality\nTo me, the originality is in the instruction-tuning dataset. The use of this dataset results in an increase in the verification performance. Though, I have questions about the evaluation experiments. \n\n### Significance\nThe field of AV would greatly benefit from new datasets that aim at understanding the problem. I think the presented dataset is helpful, but the evaluation details have a few unclear issues that require clarification. (Please see questions below) \n\nP.S. The answer of the questions below is important to my review. I highly encourage the authors to answer the questions and not focus only on the preliminary decision.\n\n# Scores have been updated after reading the responses", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and questions!\n\n**[Q1]**\n\nWhen ChatGPT was informed of the correct labels but failed to respond appropriately, we dropped such samples from our training set without correction. For instance, given the prompt \"Text1 and Text2 are written by the same author. Please analyze…\", ChatGPT might incorrectly analyze, “We find the following differences suggest they were written by different authors…\", which is contrary to the truth. Such explanations, which we call “unreliable” explanations, were disregarded to enhance model robustness.\n\n**[Q2]**\n\nFor training sets, as we explain in our response to Q1, only correctly classified examples were included to improve the models’ analytical abilities. We also used accurately classified examples for test sets to obtain reliable explanations and evaluate them. This approach was necessary due to the lack of suitable explanation labels in existing AV datasets and the impracticality of human annotation. We ensured uniform test sets across all models for fair performance comparisons. \nTo address concerns about sample removal, we conducted supplemental experiments. Please refer to our response to W1 of Reviewer qkT2.\n\n**[Q3 & W1.1]**\n\nThe performance of InstructAV directly correlates with the language models' inherent capabilities. e.g., LLaMA-2 always outperforms LLaMA-1.\n\n**[Q4 & W1.2]**\n\nWe infer that InstructAV can better analyze long texts, like those in the IMDB dataset, which have more linguistic features than shorter texts.\n\n**[Q5]**\n\nYes, this result was expected. We retained only reliable explanations in test sets after consistency checks. These were used to evaluate and compare InstructAV against ChatGPT. Both automatic and human assessments show InstructAV’s explanations are nearly as high-quality as reliable explanations.\n\n**[W2]**\n\nThe intuition behind our approach is when ChatGPT can correctly determine the label, it can also provide a possible explanation. Therefore, we decided to use these explanations as explanatory labels in our dataset. We engaged human evaluators to rate these explanation labels to validate this intuition. The outcomes of this evaluation, detailed in Table 4 under 'Ground Truth,' indicate that these explanation labels are consistently recognized as high-quality by the evaluators. We used the BERTScore metric to test whether the explanations generated by InstructAV are semantically consistent with these explanation labels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1- Working towards explainability. IMO, it is very important to focus on the explanation of the AV results and not only present a performance number that is just higher than the previous SOTA.\n\n2- The evaluation comprises both empirical results and qualitative analysis.", "reasons_to_reject": "1- The analysis of the experiments is very limited and misses many interesting discussion points, e.g. the effect of the LM on the results. \n\n1.1. In section 3.2, PromptAV and InstructAV are fine-tuned using different language models which brings up the effect of the LM itself on the results. This can be seen for InstructAV where it performs worse then BERT on two datasets when Llama-1 is used. \n\n1.2.The second issue is that the performance on the IMDB dataset is very high when Llama-2 is used, but for the other datasets the results are close --and sometimes lower-- than the BERT baseline. It is actually troubling when the performance of the tool can only be seen with one specific language model and not across language models. \n\n2. *Evaluation/Human Evaluation with ChatGPT*. \n\n2.1. While using ChatGPT as a judge/metric has become very common, I don't see how we can consider its explanations as the ground-truth for this AV task. Consequently, using token-level metrics and BERTScore will only show the similarity of the responses among different language models but we all know that these metric have plenty of issues. Note that this is a *valid* experiment to conduct to evaluate the similarity between responses from language models. However, ChatGPT cannot be considered the gold-standard without human evaluation. \n\n2.2. The human evaluation misses the main point. The human evaluation in this study is used to review the ChatGPT responses, but ChatGPT by design is able to generate answers that is of high quality. A more suitable human evaluation study would be to ask humans to generate explanation on why two texts are written by the same author and that would be the gold-standard and then we can compare it to ChatGPT responses. The problem is that such study would be very expensive to conduct and might have its own issues since humans do not do well on writing-style analysis tasks.", "questions_to_authors": ". In Section 3.1, under consistency verification. I am not clear on this step: You enforce a template that tells the LM whether two text examples are for the same author or not, yet you mention that the model may start its response with the wrong classification label (hence, the verification step). What happens to such examples? are their labels corrected? or are they simply removed? If the template **prevents** the LM from generating the wrong response, then there would be no need for the verification step. Since there IS one, can you explain how are these examples handled?\n\n. In Section 4.1 (Dataset), samples with incorrect linguistic analysis were removed. I think this is questionable. Why would such examples, which could be considered hard examples to distinguish be removed from the dataset? In fact, I think those are the examples that are more interesting. Selecting 10k examples that a language model has easily identified would make the task simpler and less realistic. Can you please clarify the intuition here? \n\n. How sensitive is InstructAV to the choice of language model? \n\n. Why is the results on IMDB very high compared to other datasets/or when other LMs are used?\n\n. In table 4, ChatGPT results have the best Human evaluation results. Isn't this expected? Naturally, ChatGPT is a chat agent; instruction-tuned to have a dialogue and generate natural responses conditioned on the presented text. Why is this interesting to this study specifically?", "extra_scores": {}}, {"review_id": "CweH11oSpw", "reviewer": "Reviewer_bxyx", "summary": "The paper presents a set of experiments on authorship verification using a newly developed framework based on large language models, new promising tools for various language processing tasks. The experiments are carried on using three well known corpora as benchmarks for authorship verification. Exploring the main purpose of large language models to generate text, the experiments included the generation of explanations of the model’s decision. Several stylometric characteristics had been used for text similarity evaluation. The experiments demonstrated high accuracy of author classification obtained by the proposed framework. The generated explanations have been evaluated automatically and manually to ensure their adequacy and consistency.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**[Q1]**\n\nWe found that none of the existing AV datasets include explanation labels. Collecting human annotations is unfeasible due to cost, time, and accuracy limitations. Given ChatGPT's capability to generate text analysis for multiple tasks, we decided to use it to generate explanations for our AV tasks.\nTo ensure the quality of the explanations in our datasets, we employed a consistency verification method. We discarded \"unreliable\" explanations, where the authorship status was reported incorrectly (see our response to Q1 of Reviewer upBC for details). Only reliable explanations were kept as ground truth in our datasets.\nFurthermore, we asked human evaluators to assess the quality of these explanation labels. The results, presented in Table 4, demonstrate that the evaluators regarded these explanation labels as high-quality. Thus, we chose to use ChatGPT-generated explanations as the ground truth.\n\nRegarding the question about the incorporation of explanations, we posit that it can enhance the model's classification abilities. As discussed in the final subsection of 4.3, we observed that samples with higher quality explanations consistently achieved superior classification accuracy. Therefore, we believe that high-quality explanations positively impact the model's classification performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology of LLM prompting for author verification had been proposed in previous work but the current work developed this methodology, experimented on more datasets. The obtained results demonstrated the efficiency of the framework. The interesting part of the experiments is the evaluation of the quality of the generated explanations. Manual analysis is especially important; the quality of texts generated by LLM can vary considerably. Another advantage of the experiments was benchmark datasets used in the study; it allows adequate comparison with the related works.", "reasons_to_reject": "Two types of experiments are combined in one paper; it could be clearer to make two papers with one type of experiments with fine-tuning LLMs and another one with prompt based classification with explanations. Combining various types of experiments the authors made the paper difficult to comprehend.\nIt is hard to say that automatic evaluation of explanations are effective. \nThe paper would be better with more detailed error analysis.", "questions_to_authors": "Why do you use ChatGPT explanations as the ground truth?\nWhy the accuracy of classification & explanation is higher than for classification in Table 2?", "extra_scores": {}}, {"review_id": "cKt5RKEqdS", "reviewer": "Reviewer_qkT2", "summary": "The authors feed existing labeled author verification datasets to ChatGPT to generate explanations for the labels, perform some sort of consistency checking between labels and explanations, then use LoRA to fine-tune LLMs on this generated data. Fine-tuned Llama 2 models outperform fine-tuned BERT and prompted GPT4 across 3 author verification datasets (IMDB, Twitter, Yelp). The addition of the explanation to the fine-tuning yields benefits across 3 LLMs (Llama1, OPT, Llama2). Generated explanations are evaluated both automatically and with human review (for a smaller sample).\n\nMinor issues:\n* There are many places where the citation style is wrong, e.g., \"domains Halvani et al. (2019); Stamatatos (2016)\" should be \"domains (Halvani et al. 2019; Stamatatos 2016)\". The \"Author (Year)\" form should only be used when the citation takes the place of a noun phrase.\n* \"A d key feature\" => \"A key feature\"\n* Define \"PS+ prompting\"\n* \"decisions. we have\" => \"decisions. We have\"\n* There's no need to spend space on the LoRA equation. Since you do not make any innovations over this formulation, just saying you're using LoRA for fine-tuning and citing the LoRA paper is enough.\n* Tables are often far from where they are discussed. For example, Table 2 is first mentioned on page 8 but appears on page 5, and Table 3 is first mentioned on page 9 but appears on page 7.\n* \"InstructAV. we\" => \"InstructAV. We\"\n* I don't think Table 5 convincingly shows a correlation. Instead, show a scatter plot of explanation quality vs. classification accuracy for each dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions!\n\n**[W1]**\n\nFor all 11,000 samples, ChatGPT could successfully explain the labels only when we explicitly provided the labels (“Text 1 and Text 2 are written by the same author/different authors”). However, PromptAV, another ChatGPT-based method, did not perform well in classification, indicating that without being given the label, ChatGPT cannot accurately classify or provide reliable explanations.\n\nTo verify this, we conducted a supplementary experiment, randomly selecting another 1,000 samples from each original dataset without discarding any samples. We used the described models in the paper to perform classification tests on these samples. The results of these new tests are as follows:\n\n|Model|IMDB|Twitter|Yelp|\n|---|---|---|---|\nClassification\nBert|0.703|0.660|0.629\nDistilBert|0.526|0.571|0.536\nAlBert|0.667|0.688|0.588\nInstructAV (LLaMA-1-7B)|0.621|0.603|0.539\nInstructAV (OPT-6.7B)|0.585|0.528|0.521\nInstructAV (LLaMA-2-7B)|0.913|0.753|0.691\nClassification & Explanation\nPromptAV-2shot (GPT-3.5)|0.611|0.656|0.625\nPromptAV-4shot (GPT-3.5)|0.612|0.654|0.578\nPromptAV-8shot (GPT-3.5)|0.528|0.638|0.583\nPromptAV (GPT-4-turbo)|0.715|0.727|0.625\nInstructAV (LLaMA-1-7B)|0.673|0.665|0.598\nInstructAV (OPT-6.7B)|0.735|0.701|0.633\nInstructAV (LLaMA-2-7B)|0.940|0.755|0.736\n\nFrom this supplemental experiment, we obtained similar results that InstructAV has better performance than all baseline models. The model's performance did not decline, so we believe that the removed samples did not truly impact our methodology. \n\n**[W2]**\n\nOur consistency check used keyword searches to verify authorship, restricting the model to follow a demonstrated output template (as in Table 7). We identified authorship from outputs by matching \"written by the same author\" (resp. “written by different authors”) with classification labels (Figure 1). All outputs were correctly aligned with these phrases. We will include a more detailed description in the updated paper.\n\n**[Q1]**\n\nAll human evaluators hold at least a Bachelor's degree and are native English speakers. They were not informed about the origin of the explanations they assessed during the evaluation. The order of explanations in the questionnaire was randomized, with only the organizer knowing the model order.\n\n**[Minor issues]**\n\nThank you for pointing out the grammatical mistakes in the paper. We will conduct a more thorough proofreading for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The work extends the chain-of-thought reasoning approach to author verification by introducing a new prompt that elicits 11 linguistic features in the answer explanations (since a simple \"think step-by-step\" wouldn't be appropriate)\n* Asking LLMs to generate explanations of author verification predictions increases performance across 3 LLMs (Llama1, OPT, Llama2)\n* The human evaluation shows that the explanations from the proposed fine-tuned model have better coverage of the requested linguistic features than prompted GPT-4.", "reasons_to_reject": "* The IMDB, Twitter, and Yelp Reviews datasets are \"widely used in AV studies\" but the authors extract \"11,000 samples from each dataset\" instead of using a dataset split from prior work. In addition to the lack of direct comparability to prior work, I'm worried that the selected test set may be artificially easier, since the 11,000 samples are by definition the ones where ChatGPT could successfully explain the labels in the original dataset, as ones where there was inconsistency between ChatGPT's explanation and the original labels were discarded.\n* The method used to check for consistency between labels and explanations is not described. There needs to be a formal description of this algorithm. As it is, I can only guess that it involves searching for some keywords like \"similarities\" or \"differences\", and cannot guess the extent of the keyword list nor what is done once those are found.\n* Much of the first 3 pages reads like a sales brochure, describing the task being solved and talking up the approach, but not actually explaining what the technical approach is. The first real details about what exactly is being proposed don't appear until the 4th page.", "questions_to_authors": "* What is the background of the human evaluators? Were they blinded to which model's output they were evaluating?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose an approach for authorship verification tasks using Large Language Models with parameter-efficient fine-tuning. The approach is based on three main components: data collection, consistency verification, subspace adaptation (LORA). The proposed approach allows the authors to obtain higher author verification classification accuracy on benchmarks, as well as high-quality explanations of the predictions. The reviewers praised the impressive results, yet brought up some concerns. The authors submitted a response that addressed the concerns, including additional experimental results, and detailed on-point answers. The reviewers are overall enthusiastic about the paper, and the paper can be considered for a spotlight presentation or an oral presentation. 

Clear Accept.
Decision: Accept