Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification
Authors: Chia-Yu Hung
Zhiqiang Hu
Yujia Hu
Roy Ka-Wei Lee
Venue: EMNLP 2023 Findings (2023)
Keywords: Authorship Verification
Chain-of-Thought Prompting

Abstract: Authorship verification (AV) is a fundamental task in natural language processing (NLP) and computational linguistics, with applications in forensic analysis, plagiarism detection, and identification of deceptive content. Existing AV techniques, including traditional stylometric and deep learning approaches, face limitations in terms of data requirements and lack of explainability. To address these limitations, this paper proposes \textsf{PromptAV}, a novel technique that leverages Large-Language Models (LLMs) for AV by providing step-by-step stylometric explanation prompts. \textsf{PromptAV} outperforms state-of-the-art baselines, operates effectively with limited training data, and enhances interpretability through intuitive explanations, showcasing its potential as an effective and interpretable solution for the AV task.

Who Wrote it and Why? Prompting Large-Language Models for Authorship Verification
Chia-Yu Yu Hung, Zhiqiang Hu, Yujia Hu, Roy Ka-Wei Lee
Singapore University of Technology and Design {chiayu_hung, zhiqiang_hujemynne sutd. outd.edu.se.. sg (yujia_hu, roy_lee]@sutd, edu.
Abstract
n-gram frequencies, vocabulary richness, and syn- tactic structures, to capture the unique writing style of individual authors (Ding et al., 2017). Machine learning techniques, such as support vector ma- chines (SVMs) and random forests, have been em- ployed to model the relationship between these sty- lometric features and authorship (Brocardo et al., 2013). More recently, deep learning approaches have shown promising AV results by learning intri- cate patterns directly from raw text data (Bagnall, 2015; Ruder et al., 2016; Fabien et al., 2020).
Authorship verification (AV) is fundamen- tal task in natural language processing (NLP) and computational linguistics, with applica- tions in forensic analysis. plagiarism detection, and identification of deceptive content. Exist- ing AV techniques, including traditional sty- lometric and deep learning approaches, face limitations terms data requirements and lack of explainability. To address these limita- tions, this paper proposes PromptAV, a novel technique that leverages Large-Languee Mod- els (LLMs) for AV by providing step stylometric explanation prompts. PromptAV outperforms statc-of-the-art baselines, operates effectively with limited training data, and en- hances interpretability through intuitive expla- nations, showcasing its potential an effective and interpretable solution for the AV task.
Despite these advancements, existing AV tech- niques have certain limitations. Firstly, most exist- ing AV methods require a large amount of labeled training data, which can be costly to acquire, es- pecially for scenarios with limited available data. Secondly, there is a lack of explainability in the predictions made by these techniques. In many practical applications, it is crucial to understand why particular decision was reached, which is particularly relevant in legal contexts or situations where interpretability is paramount.
1 Introduction
Motivation. Authorship verification (AV) is fun- damental task in the field of natural language pro- cessing (NLP). aims to determine the authorship given text by analyzing the stylistic character- isties and patterns exhibited in the writing (Sta- matatos, 2016). The importance of AV lies in its wide range of applications, including foren- sic analysis (Iqbal et al., 2010), plagiarism detec- tion (Stein al., 201 and the detection of decep- tive or fraudulent content (Claxton, 2005). The rise of LanguaLa Models (LLMs) (Chowdhery et al.. 2022: Brown et al., 2020), has facilitated the generation of himan-like text where one would now need to factor in the possibility of machine- generated text (Uchendu et al., 2023). This de- velopment give rise to a new layer of complexity in distinguishing between machine-geneeee generated and human-written text, subsequently amplifying the significance of the AV problem.
Research Objectives. To address these limita- tions, We propose PromptAV, novel technique that utilizes LLMs for authorship attribution by employing step-by-step stylometric explanation prompts. These prompts guide the L.MSs to an- alyze and evaluate the textual features that con- tribute to authorship, enabling the model to provide explanations for its predictions. PromptAV demon- strates effectiveness in both zero-shot and few-shot settings, offering improved performance compared to existing baselines, especially in scenarios with limited training data. Additionally, detailed case studies showcase PromptAV's ability to provide intuitive explanations for its predictions, shedding light on the factors contributing to authorship attri- bution and enhancing the interpretability of the AV process.
Over the years, various techniques have been proposed to address the AV task. Traditional ap- proaches relied on stylometric features, such as
This paper introduces Promp- tAV, a prsmpt-based learning technique that har-
Contribution.

---

Chain -of -Thought prompting
PromptAV
Task:
On a scale of to 1, with 0 indicating low confidence and indicating high confidence, please provide general assessment of the likelihood that Text 1 and Text were written by the same author. Your anawer should reflect a moderate level of strictness in scoring. Let's think step by step.
Task: On scale of 1, with 0 indicaling low confidence and 1 indicating high confidence, please provide general assessment of the likellhood that Text 1 and Text 2 were written by the same author. Your answer should reflect moderate level of strictness in scoring. Here are some relevant variables to this problem.
1. punctuation style(e.g. hyphen, brackets, colon, comma, parenthesis, quotation mark)
Text 1: [T1)
Text (T2)
2. special characters style, capitalization style(e.g. Continuous capitalization, capitalizing certain words) acronyms and abbreviations(e.g g. Usage of acronyms such OMG, Abbreviations without punctuation marks such as Mr Rochester vs. Mr. Rochester, Unusual sbbreviations such as def vs. definitely) writing style
PS+ prompting
(c)
Task: On a scale of to 1, with 0 indicating low confidence and indicating high confidence, please provide general assessment of the likelihood that Text and Text were written by the same author. Your answer should reflect a moderate level of strictness in scoring. First step: Understand problem. extracting relevant variables and devise plan to solve the problem. Then, carry out the plan and solve the problem step by step. Finally, show the confidence score.
5. expressions and Idioms
6. tone and mood
sentence structure
8. any other relevant aspect First step: Understand the problem, extracting relevant variables and devise a plan to solve the problem. Then, carry out the plan and solve the probiem step by step. Finally, show the confidence score.
(b)
Text [T1]
Text [T1]
Text [T2)
Text [T2]
Figure I: Prompts used by (a) CoT, (b) PS+ prompting, (c) PromptAV for AV task. The trigger intructions of the various techniques are highlighted in the prompt.
nesses the linguistics capabilities of LLMs for AV. By providing step-by-step stylometric explanation prompts, PromptAV achieves improved perfor- mance, offers explainability in its predictions, and operates effectively in both zero-shot and few-shot settings. The experimental results demonstrate the potential of PromptAV as an effective and inter- pretable solution for the AV task.
Transformer architecture (Vaswani et al., 2017) has prompted the development of Transformer-based AV models (Ordoñez et al., 2020; Najafi and Tavan, 2022; Tyo et al., 2021).
Methodology
In an recent work, Wei al. (2023) have proposed Chain of thought (CoT) prompting, which employs a series of intermediate reasoning steps to signif- icantly improve the ability of LLMs in perform- ing complex reasoning. Wang et al. (2023) fur- ther extended the concept to propose PS+ prompt- ing, zero-shot prompting methodology that in- structs LLMs formulate a strategic plan prior to problem-solving
2 Related Work
AV is a well-established research topic. Classi- cal approaches to AV have predominantly cen- tered on stylometric features encompassing n-gram frequencies, lexical diversity, and syntactic struc- tures to discern the unique writing styles intrinsic to individual authors (Stamatatos, 2016; Lagutina et al., 2019). Classical machine learning algorithms were also used to model the associations between these stylometric attributes and authorship (Bro- cardo et al., 2013; Hu et al., 2023). Recently, deep learning models such RNNs and CNNs being employed extract more complex patterns from textual data (Bagnall, 2015; Ruder et al., 2016; Benzebouchi et al., 2018). These deep learning architectures have exhibited promising advance- ments in AV tasks. Furthermore, the advent of the
Inspired by these works, we introduce Promp- tAV, a prompting strategy that incorporates key linguistic features-identiffee in (Boenninghoff et al., 2019) as the crucial variables for LLMs to evaluate. These linguistic features serve as rich, often subtle, markers of author's distinct writ- ing style and are hence of paramount importance for the AV task. For instance, the consistent use or avoidance of certain punctuation marks, or the preference for particular special characters, can be idiosyncratic to an author. Similarly, elements

---

In the conventional setup of few-shot CoT prompt- ing, each example necessitates the manual crafting of reasoning steps, an approach that is impracti- cal for the AV task. This is primarily due to the challenges in ensuring the consistent quality of hand-crafted reasoning across diverse prompting templates. Hence, we resort to leveraging the ca- pacity of LLMs as zero-shot reasoners to generate the required reasoning steps (Kojima et al., 2023). like tone and expression, though harder to quan- tify, can offer sigmificant insights into an author's unique 'voice'. The integration of these linguistic attributes within the PromptAV framework does not simply add layers of complexity to the model, it fundamentally enhances the model's ability to capture the author-specifie nuances within textual data. This capability, in turn, improves the model's accuracy and reliability when attributing authorship unidentified texts, making PromptAV a power- ful tool in the field of AV. Figure provides an overview that compares the prompts used in CoT, PS+ prompting, and PromptAV.
Setting
Model
Accuracy
0-Shot
PromptAV
0.587
PS+ prompting
0,536
CoT
0,524
2-Shot
PromptAV
0.667
CoT 0.595
4-shot
PromptAV
0.635
CoT
0.510
Table 1: Performance of PromptAV and the baselines on k-shots settings. Highest accuracy are bolded.
iments. These examples were randomly sampled from the training set and presented to the LLM in the same order.
We also note that conventional prompting tech- niques that instruct the LLM to respond strictly with binary "yes" or "no" answers frequently result in the LLM responding "no" for the majority of instances within an AV task. To mitigate this prob- lem, PromptAV instructs the LLM to generate a confidence score ranging from 0 to I, rather than a binary response. To calibrate these generated con- fidence scores, we augment the prompt with the additional directive "Your answer should reflect moderate level of stricmess in scoring".
4 Experiments
4.1 Experimental Settings
Dataset. We evaluate PromptAV and the baselines using IMDb62 (Seroussi et al., 2014), which is commonly used dataset for AV task. Each record in dataset contains pair of texts and the ground truth labels, i.e., a "positive" pair of texts belong the same author, and "negative" for otherwise. As we are interested performing the AV task in zero- shot and few-shot settings, we focus on sampling a manageable test set of 1,000 text pairs from the original IMDb62 dataset. The resulting test set comprises 503 positive and 497 negative text pairs. Implementation. All of our experiments were conducted on the gpt-3. 5- turbo model, with the temperature parameter set to O. We benchmarked PromptAV against PS+ prompting in zero-shot settings and against CoT prompting in zero, two, and four-shot settings. Given that all three methods generate a confidence score in the range of 0 to 1, we reported the accuracy corresponding the optimal threshold that yields the highest accuracy. We call this the Optimal Threshold Methodology and we will dicuss more about in Appendix A.3.
3.1 Few Shot Prompting
To construct the few-shot PromptAV and CoT examples, we introduce an additional directive to the original prompt: "It is given that after follow- ing the instruction, the confidence score obtained is [X]. Show the step-by-step execution of the in- struction so that this score is achieved." Here, [X] is either 0.9 or 0.1, contingent upon whether the pair of texts were written by the same author. Sub- sequently, we feed this modified prompt to GPT-4 to generate intermediate reasoning steps.
4.2 Experimental Results
Table shows the performance of PromptAV and the baselines on k-shot settings. We observe that PromptAV consistently achieved higher accuracy across varying shot settings. This demonstrate PromptAV's 's efficacy in performing AV task with little or training data. Furthermore, the num- ber of shots increases from to 4, PromptAV maintains its edge over the COT. This suggests that PromptAV effectively leverages the additional in- formation provided in the few-shot scenarios, opti- mizing its decision- makinn process.
Zhao et al. (Zhao et al., 2021) has demonstrated that both the selection of training examples and their sequencing can significantly influence accu- Scy.Thereefoe, to ensure fairness, utilized the same examples for all our 2-shot and 4-shot exper-
"hapss//oppaal.oo/reeecc///g/

---

Text 1: " We 'e all sinners or about to be. piicked up this flick part of double film DVD offer along with " The Hanged Man " for just a buck, and truth be told, both movies price. The title on this one was The Gan and The Cross" ", close enough to the original to get the message across The tle character wields both, as Marjoe Gortner is revealed be gunfighter Ernie Parsons, masquerading g a preacher after he finds a real one who was bushwhacked. Donning the clothes of the dead minister. he arrives the town of Castle Walk just in time to help the local folk deal with nasty town hoss Ross ( DDvid Huddleston ... [Truncated due to length restriction)
whole lot better than either had a right be for the
Text 2: " This a mad dream Antinea. but shall put un end to the madness The title- Hercules and the Captive Women a bit of a misnoner, , there was only one woman captive, , and along with her mother, , Queen Antinea of Atlantis, were the only two women in the entire film. The International English title mentions the Conquest of Atlantis and that probably makes more sense. but in the end, it's the destruction of Atlantis that takes center stage. There's an entire mythology offered by Zantas, the last high priest of Uranus. , that explains what's going on with Antinea and her cohort of Grand Vizier clones... ..... ... ('Trnncated due to length restriction]
PromptType
Generated Explanation
Ohshot PromptAV
Based on the provided texts, here is general assessment of the likelihood that Text I and Text 2 were written by the same author:
1. Punctuation style: Both use similar punctuation styles, such commas, periods, and more parentheses and dashes compared to Text 2.
quotation marks. However, Text
Confidence: 0.7
2. Special characters style, capitalization style: Both texts have similar capitalization styles, with proper nouns and the beginning sentences capitalized. However, Text has more continuous capitalization (e.g., "The Gun and The Cross") compared to Text Confidence: 0.6
3. Acronyms and abbreviations: Both texts use acronyms and abbreviations, but there are no significant differences in their usage. Confidence: 0.8
4. Writing style: Both texts have a similar writing style, with a mix of descriptive and narrative elements. However, Text I focuses more on reviewing a movie, while Text 2 provides a summary a film's plot. Contidence: 0.7
5. Vocabulary usage: Both texts use a variety of vocabulary, but there are no significant differences in their usage. Confidence: 0.8
Overall, based on the assessment of the relevant variables, there is a moderate likelihood (confidence score: 0.7) that Text and Text 2 were written by the same author.
O-shot CoT
Based on a moderate level strictness scoring. the likelihood that Text and Text were written by the same author is low, The two texts have different subject matters and writing styles, with Text I discussing a western film and Text 2 discussing a Hercules film. Additionally, the tone and language used in each text are distinct. Therefore, it is unlikely that the same author wrote both texts.
Based on these factors, would assess the likelihood that Text I and Text 2 were written by the same author as low, with confidence level of around 0.3 on the scale of 0'O to 1.
0-shot PS+ prompting
Based on moderate level of strictness in scoring, I would assess the likelihood that Text and Text were written by the same author as low. The two texts have different themes, writing styles, and subject matters. Text 1 discusses a western film, while Text discusses Hereules film. The language and tone used in each text are also distinct. Therefore, would assign a confidence score of 0.2, indicating a low likelihood of the same authorship.
Table 2: A text pair example with the generated responses from PromptAV, CoT and PS+ Prompting
Model
Accuracy
sic linguistic characteristics, which are critical in differentiating writing styles of various authors.
PromptAV
0,587
PromptAV Obfuscated
0.580
PS+ prompting
0.536
CoT
0.524
4.3 Explainability Case Studies
Table 3: Performance of PromptAV against a series of authorship obfuscation methods. Highest accuracy are bolded
While achieving explainability in the AV task is notoriously difficult, PromptAV rises to the chal- lenge by providing detailed, interpretable solutions. This is accomplished through a meticulous anal- ysis of the linguistic features. A sample of the generated responses from PromptAV, CoT, and PS+ prompting is illustrated in Table 2. In contrast to CoT prompting or PS+ prompting, PromptAV offers a comprehensive comparison of linguistic features, such as punctuation style and vocabulary usage. For example, PromptAV detects that both
The superior performance of PromptAV over PS+ prompting and CoT in the zero-shot setting un- derscores its capacity to make effective use ofkey linguistic features for AV even in the absence of training examples. This is indicative of the fact that PromptAV is adept at understanding intrin-

---

texts have similar capitalization style and writing style, with a mix of descriptive and narrative ele- ments. On the other hand, CoT and PS+ prompting systems deliver a more superficial level of analysis. While they can recognize discrepancies in vocab- ulary selection across texts, they lack the capacity to provide exhaustive explanation or deeper analysis of these differences.
5 Conclusion
In this paper, we proposed PromptAV. a prompt- based learning technique that harnesses the lin- guistics capabilities of LLMs for AV. Our experi- mental results demonstrated PromptAV's superior performance over the state-of-the-art baselines in zero-shot and few-shot settings. Case studies were also conducted illustrate PromptAV's effective- ness and explainability for the AV task. For future works, we will conduct experiments on more AV datasets and explore other linguistic features.
4.4 Authorship obfuscation
In our pursuit to understand PromptAV's potential robustness, we took focused approach by testing against an obfuscation dataset. We selected the sug- gested Mutant-X algorithm (Mahmood et al., 2020), a recognized method the authorship obfuscation domain, and applied it to the IMDb62 dataset that was referenced in our paper. We apply PromptAV in zero-shot setting on the obfuscated dataset and benchmark against its performance on the original dataset. The experimental results are presented in Table 3. The results reveal a negligible decline performance when confronted with obfuscated text. Interestingly, the accuracy achieved, even with obfuscation, surpasses the results of zero-shot CoT and PS+ prompting non-obfuscated text. This indicates a promising level of resilience in PromptAV against authorship obfuscation methods. While these preliminary results are encouraging, we acknowledge the need for more comprehensive testing against a broader range of obfuscation meth- ods to assert PromptAV's robustness conclusively.
Limitations
This study, while substantive, acknowledges cer- tain limitations. A significant constraint is the ab- sence of ground truth for the generated explana- tion, which renders the evaluation of the efficacy of PromptAV's generated explanation somewhat challenging. This suggests a potential avenue for tapping into the expertise of forensic linguistic ex- perts to assess and curate some of the reasoning generated by PromptAV. Additionally, we have observed instances where PromptAV generates illu- sory explanations by identifying the usage of spe- cific vocabulary within the text, even when such vocabulary may not have been actually used.
References
Douglas Bagnall. 2015. Author identification us- ing multi-headed recurrent neural networks. arXiv preprint 15XYv:5506.0453
4.5 Stylometric Feature Impact
The field of stylometry is vast, with a plethora of features available for authorship analysis. The cen- tral aim of PromptAV was to showcase the flexibil- ity and adaptability of the framework when work- ing with different sets of stylometric features. We conducted series of experiments using varying sets of stylometric features with PromptAV in zero- shot setting. We report the result in Table 4. The re- sult clearly showcases that the performance varies based on the chosen feature set. The peak accu- racy achieved with 9-feature set emphasizes that PromptAV's success is contingent on the quality and appropriateness of the selected features. We genuinely believe that the field holds significant undiscovered potential, with combinations of fea- tures that could further clevate the performance of PromptAV. Our results only scratch the surface, and our intention is to spur the community into exploring these myriad possibilities.
Nacer Eddine Benzebouchi, Nabiha Azizi, Monther Ald- wairi, and Nadir Farah. 2018. Multi-classifier system for authorship verification task using word embed- dings. 2018 2nd International Conference on Nat- ural Language and Speech Processing (ICNLSP). pages 1-6. IEEE
Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Rohert M Nickel. 2019. Explainable authorship verification in social media via attention- based similarity learning. In 2019 IEEE Interna- rional Conference on Big Data (Big Data), pages 36 45, IEEE
Marcelo Luiz Brocardo, Issa Traore, Sherif Saad, and Isaac Woungang. 2013. Authorship verification for short messages using stylometry. In 2013 In- ternational Conference on Computer Information and Teleconmunication Systems (CITS), pages 1-6. IEEE.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda

---

Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Ksenia Lagutina, Nadezhda Lagutina, Elena Boychuk. Inna Vorontsova, Elena Shliakhtina, Olga Belyaeva, Ily Paramonov, and PG Demidov. 2019. A survey stylometric text features. In 2019 25th Conference of Open Innovations Association (FRUCT), pages 184-195, IEEE.
Asad Mahmood, Zubair Shafiq, and Padmini Srinivasan. 2020. A girl has name: Detecting authorship obfus- cation.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaram, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov. Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick. An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-He Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
Maryam Najafi and Ehsan Tavan. 2022. Text-to-text transformer in authorship verification via stylistic and semantical analysis. In Proceedings of the CLEF.
Juanita Ordoñez, Rafael Rivera Soto, and Barry Y Chen. 2020. Will longformers pan out for authorship verifi- cation. Working Notes of CLEF.
Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016. Character-level and multi-channel convolu- tional neural networks for large-scat scale authorship at- tribution. arXiv preprint arXiv: 1elh::660906600
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohn- ert. 2014. Authorship attribution with topic models. Computational Linguistics, 40(2):269-310.
Elitathios Stamatatos. 2016. Authorship verification: review of recent advances. Research in Computing Science, 123:9-25.
Larry D Claxton. 2005. Scientific authorship: Part window into scientific fraud? Mutation Re- searcl/Reviews in Mutation Research, 589(1)117-30.
Benno Stein, Nedim Lipka, and Peter Prettenhofer. 2011. Intrinsic plagiarism analysis. Language Resources and Evaluation, 15.63-82.
Steven HH Ding, Benjamin CM Fung, Farkhund Iqbal, and William K Cheung. 2017. Learning stylometric representations for authorship analysis. IEEE trans- actions cybernetics, 1911::07-112)
Jacob Tyo, Bhuwan Dhingra, and Zachary C Lipton. 2021. Siamese bert for authorship verification. CLEF (Working Notes), pages 2169-2177.
Adaku Uchendu, Thai Le, and Dongwon Lec. 2023.
Maël Fabien, Esau ElllatoroTello, Petr Motlicek, and Shantipriya Parida. 2020. BertAA BERT fine- tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Lan- guage Processing (ICON). pages 127-137, Indian Institute of Technology Patna, Patna, India. NLP As- sociation of India (NLPAI).
Attribution and obfuscation of neural text authorship: A data mining perspective.
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Xinyu Hu, Weihan Ou, Sudipta Acharya, Steven H Ding, Ryan D' D'Gama, and Hanbo Yu. 2023. Tdrlm: Stylo- metric learning for authorship verification by topic- debiasing. Expert Systems with Applications, page 120745.
Lei Wang. Wanyu Xu. Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Wei Lee, and Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain-of- thought reasoning by large language models.
Farkhund Iqbal, Liaquat A Khan, Benjamin CM Fung, and Mourad Debbabi. 2010. E-mail authorship ver- ification for forensic investigation. In Proceedings of the 2010 ACM Symposium on Applied computing. pages 1591-1598.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia. Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain of thothhtt prompting elic- its reasoning in large language models.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2023. Large lan- guage models are zero-shot reasoners.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models.

---

A Appendix
Confidence Score
PromptAV
PS+
CoT
A.1 Computation and Time comsumption of PromptAV
0.1
0.506
0.501
0.524
0.2
0.587
0.536
0.522
our experiments, we applied PromptAV using GPT-3.5 via the OpenAl API. Specifically, evalu- ating 1,000 examples took approximately 8 hours. The expense associated with using PromptAV via the OpenAI API is indeed calculated at rate of $0.004 per 1,000 tokens. For dataset of 1,000 samples, the estimated cost came around $10. This cost, cost, we believe, is justifiable given the unique interpretability insights PromptAV provides, espe- cially critical applications like forensic analysis where the implications of the results are profound. It is possible to batch-procest texts through sending multiple requests using multiple API keys. With adequate budget, could scale-up the AV opera- tions.
0.3
0.579
0.529
0.505
0.4
0.578
0.529
0.497
0.5
0.569
0.527
0.497
0.6
0.55
0.484
0.498
0.7
0.53
0.493
0.498
0.8
0.519
0.497
0.497
0.9
0.5
0.497
0.497
Table 5: Performance of PromptAV and the baselines in shot setting with varying confidence score threshold. Highest accuracy are bolded
model's cautious approach to attributing author- ship, ensuring a reduced number of false positives in practical scenarios. This threshold may varied, depending the complexity of the dataset, and the amount of writing sample observations the model made for each author.
A.2 Stylometric Feature Impact
The confi- dence score selection was an empirical decision based on our preliminary experiments, intending to optimize the balance between false positives and false negatives. We have conducted an experiments to show the tradeoff on performance when select- ing different threshold for the confident score. Note that experiments conducted using PromptAV with zero-shot demonstration setting.
Selection of the Confidence Score:
Number of features
Accuracy
8
0.587
9
0.602
10
0.585
11
0.564
12
0.572
Table 4: Performance of PromptAV in 0 shot setting with varying sets of stylometric features.
A44 Zero-shot vs supervised methods
Our work with PromptAV centers around its capa- bility to handle the AV task in zero-shot setting. Acquiring labeled data, especially for tasks like AV, can be quite challenging due to privacy concerns, the need for expert annotators, and other complexi- ties associated with the domain. We recognize the importance and relevance of benchmarking against state-of-the-art AV methods. Their dependence on copious amounts of labeled data inherently places them in a different operating paradigm. For in- stance, BERT able to achieve a high accuracy of 0.702 if it is trained on IMDb62 full training dataset with 74,400 samples. However, BERT will require large amount of training data to achieve the superior performance.
A.3 Optimal Threshold Methodology
The method of reporting using an optimal threshold was our initial approach to compare the results of different prompting methods. The intention was to establish baseline comparison across varying methods in controlled environment. In real-world applications, the threshold would likely require ad- justment based on specific use-cases and available data.
Practical Implications of Threshold: It is in- deed a challenge to select optimal threshold that would be universally valid. In our experiment ap- plying PromptAV on the IMDb62 dataset, we found that the optimal threshold lies between 0.2 0.3. This range suggests that the model tends generate a low confidence score, which provides significant insights into its functioning and decision makiig... The low confidence score may be indicative of the


Reviews:
[{"review_id": "z3iuiPtIqR", "reviewer": "Reviewer_4xL7", "summary": "", "questions": "According to the limitations above, there are the questions:\n\n(1) Is PromptAV able to batch-process texts with reasonable time consumption, as AV or AA (authorship attribution) models do?\n\n(2) The paper uses an optimal threshold when reporting accuracy. Is this valid in real application? Can the authors provide more insights in the selection of the confidence score?\n\n(3) Is PromptAV robust against a series of authorship obfuscation methods? If so, why?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback and questions. Your thought-provoking questions have significantly contributed to enhancing our paper's quality. We hope that we have comprehensively addressed the concerns and questions you presented. Based on the valuable discussion, we will make the necessary updates to our paper.\n\n---\n\n***W1/Q1**: The computation and time consumption of PromptAV is not analyzed. Is PromptAV able to batch-process texts with reasonable time consumption, as AV or AA (authorship attribution) models do?*\n\n**[Response to W1/Q1]**\n\nWe are glad you brought this up as this information is pivotal when discussing the practicality of our method. To provide clarity, our proposed technique, PromptAV, utilizes Large Language Models (LLMs) for authorship verification by employing step-by-step stylometric explanation prompts. The computation and time consumption is dependent on the LLMs used.\n\nIn our experiments, we applied PromptAV using GPT-3.5 via the OpenAI API. Specifically, evaluating 1,000 examples took approximately 8 hours. While this might seem substantial, it is essential to highlight that PromptAV focuses not just on accuracy but also on providing detailed step-by-step stylometric explanations, which inherently requires intricate computations and, consequently, more time. \n\nWe could also to batch-process texts. Nevertheless, OpenAI API has limits for request per minute and token per minute for each API key. To batch-process text, we can send multiple requests using multiple API keys. With adequate budget, we could scale-up the AV operations.\n\nRegarding the financial aspects, the expense associated with using PromptAV via the OpenAI API is indeed calculated at a rate of \\\\$0.004 per 1,000 tokens. For a dataset of 1,000 samples, the estimated cost came to around \\\\$10. This cost, we believe, is justifiable given the unique interpretability insights PromptAV provides, especially in critical applications like forensic analysis where the implications of the results are profound.\n\nWe can also apply PromptAV on other open source LLMs such as LLaMA-2 and OPT in our future works. Applying PromptAV on these LLMs could significantly reduced the financial cost, and we could batch-process the text without any API limitations. \n\n---\n\n***W2/Q2**: The practicality of PromptAV is not fully considered, especially in experiment part. The paper uses an optimal threshold when reporting accuracy. Is this valid in real application? Can the authors provide more insights in the selection of the confidence score?*\n\n**[Response to W2/Q1]**\n\nOptimal Threshold Methodology: As you've rightly pointed out, the method of reporting using an optimal threshold was our initial approach to compare the results of different prompting methods. The intention was to establish a baseline comparison across varying methods in a controlled environment. In real-world applications, the threshold would likely require adjustment based on specific use-cases and available data.\n\nPractical Implications of Threshold: It is indeed a challenge to select an optimal threshold that would be universally valid. In our experiment applying PromptAV on the IMDb62 dataset, we found that the optimal threshold lies between 0.2 - 0.3. This range suggests that the model tends to generate a low confidence score, which provides significant insights into its functioning and decision-making. The low confidence score may be indicative of the model's cautious approach to attributing authorship, ensuring a reduced number of false positives in practical scenarios. This threshold may varied, depending on the complexity of the dataset, and the amount of writing sample observations the model made for each author. \n\nSelection of the Confidence Score: The confidence score selection was an empirical decision based on our preliminary experiments, intending to optimize the balance between false positives and false negatives. We have conducted an experiments to show the tradeoff on performance when selecting different threshold for the confident score. Note that experiments are conducted using PromptAV with zero-shot demonstration setting. We will include this experiment results in the appendix of our camera-ready version.\n\n| Confidence Score Threshold | Acc  |\n|-|--|\n| 0.1|0.506|\n| 0.2|0.587|\n| 0.3|0.579|\n| 0.4|0.578|\n| 0.5|0.569|\n| 0.6|0.55|\n| 0.7|0.53|\n| 0.8|0.519|\n| 0.9|0.5|\n\nBased on your feedback, we acknowledge that more rigorous experiments detailing the iterative process of threshold selection and its implications on real-world tasks will bolster the practicality of our approach. We will also add the above discussion into the updated version of our paper. \n\n---\n\n***W3/Q3**: The robustness of PromptAV is not analyzed, which is crucial for AV. Is PromptAV robust against a series of authorship obfuscation methods? If so, why?*\n\n**[Response to W3/Q3]**\n\nIn our pursuit to understand PromptAV's potential robustness, we took a focused approach by testing it against an obfuscation dataset. We selected the suggested Mutant-X algorithm [1], a recognized method in the authorship obfuscation domain, and applied it to the IMDb62 dataset that was referenced in our paper. We apply PromptAV in zero-shot setting on the obfuscated dataset and benchmark against its performance on the original dataset. The experimental results are presented in the below table.\n\n| Model | Acc. |\n|-|-|\n|PromptAV| 0.587|\n|PromptAV_Obfuscated| 0.580|\n|PS+ Prompting| 0.536|\n|CoT| 0.524|\n\nThe results reveal a negligible decline in performance when confronted with obfuscated text. Interestingly, the accuracy achieved, even with obfuscation, surpasses the results of zero-shot CoT and PS+ prompting on non-obfuscated text. This indicates a promising level of resilience in PromptAV against authorship obfuscation methods.\n\nWhile these preliminary results are encouraging, we acknowledge the need for more comprehensive testing against a broader range of obfuscation methods to assert PromptAV's robustness conclusively. We genuinely appreciate this insight and will work on expanding our experiments to present a more holistic understanding of PromptAV's capabilities in this area.\n\n---\n\n***Others**: Missing References*\n\nWe will add the suggested reference and perform a more thorough literature review in the updated version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of the paper, authorship verification, is interesting and meaningful in today's Internet and social media. Using new techniques in AV is a good attempt.\n\nZero-shot PromptAV proposed by the authors is suitable for AV in real scenarios, where people may not be able to get enough training data and train a model. PromptAV makes it possible for the masses to perform AV.", "reasons_to_reject": "The topic of this article is intriguing, but I do agree that it falls short in terms of providing comprehensive experiments and further insights, making it insufficient for publication as an academic paper now. The authors should consider and address the following questions in further detail:\n\n(1) The computation and time consumption of PromptAV is not analyzed. \n\n(2) The practicality of PromptAV is not fully considered, especially in experiment part.\n\n(3) The robustness of PromptAV is not analyzed, which is crucial for AV.", "questions_to_authors": "According to the limitations above, there are the questions:\n\n(1) Is PromptAV able to batch-process texts with reasonable time consumption, as AV or AA (authorship attribution) models do?\n\n(2) The paper uses an optimal threshold when reporting accuracy. Is this valid in real application? Can the authors provide more insights in the selection of the confidence score?\n\n(3) Is PromptAV robust against a series of authorship obfuscation methods? If so, why?", "paper_topic_and_main_contributions": "This paper designs PromptAV, which adopts LLMs in solving a traditional NLP task, authorship verification, and uses chain-of-thought to prompt the verification process. With carefully designed prompts, LLMs can achieve about 60% accuracy on the IMDb62 dataset. PromptAV improves the traditional AV methods that need training data and models in supervised learning settings. It also confirms that LLMs can serve as tools for unsupervised learning in different domains.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Asad Mahmood, Faizan Ahmad, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2019. A Girl Has No Name: Automated Authorship Obfuscation using Mutant-X. Proc. Priv. Enhancing Technol. 2019, 4 (2019), 54–71.\n\nAdaku Uchendu, Thai Le, and Dongwon Lee. 2022. Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective. arXiv preprint arXiv:2210.10488 (2022).\n\nMaël Fabien, Esaú Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020. BertAA: BERT fine-tuning for Authorship Attribution. In Proceedings of the 17th International Conference on Natural Language Processing (ICON). 127–137.\n\nHope these papers can help you."}}, {"review_id": "QEDMDe3swo", "reviewer": "Reviewer_Waug", "summary": "", "questions": "Do you know if GPT-3.5 sees IMDb62 during training? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interesting questions. Your thought-provoking questions have significantly contributed to enhancing our paper's quality. We hope that we have comprehensively addressed the concerns and questions you presented. Based on the valuable discussion, we will make the necessary updates to our paper.\n\n---\n\n***W1**: It is not clear how PromptAV could address the limitation of data requirements since it is mentioned in the abstract and introduction. The paper did not provide a solution for saving data usage. If the justification is to use GPT-3.5 instead of training a model, one could argue that the paper actually requires more data as it depends on the performance of LLM.*\n\n**[Response to W1]**\n\nYou've rightly pointed out the data-intensive nature of training LLMs. However, the essence of our work with PromptAV centers around its capability to handle the Authorship Verification (AV) task in a zero-shot setting.\n\nWhile it's true that models like GPT-3.5 have been pretrained on vast datasets, this is not an additional data requirement imposed by our approach. In fact, the primary advantage we highlight with PromptAV is its elimination of the need for a labeled AV-specific dataset for further training. As you may appreciate, acquiring labeled data, especially for tasks like AV, can be quite challenging due to privacy concerns, the need for expert annotators, and other complexities associated with the domain. Our solution alleviates this need by leveraging the power of pretrained LLMs and effectively guiding them through prompts to achieve competitive performance without further AV-specific training.\n\nIn sum, while we acknowledge the extensive data needs of LLMs like GPT-3.5 in their initial training phases, PromptAV's contribution is to leverage these pretrained models for AV without necessitating additional labeled data, making it a practically feasible solution for many real-world scenarios.\n\nFor future work, we will also explore evaluating PromptAV’s performance tradeoffs using LLMs of various sizes.\n\n---\n\n***Q1**: Do you know if GPT-3.5 sees IMDb62 during training?*\n\n**[Response to Q1]**\n\nWe are not privy to the precise datasets OpenAI utilized in pretraining GPT-3.5. OpenAI hasn't expressly delineated the inclusion of IMDb62 in its pretraining corpus. Similarly, while we are aware that the authorship verification task isn't among the instruction tuning tasks (FLAN) offered by Google, we cannot categorically state whether OpenAI incorporated the authorship verification task in GPT-3.5's fine-tuning process.\n\nThat said, there are strong empirical reasons supporting our belief that GPT-3.5 hasn’t encountered IMDb62. As you correctly noted, if GPT-3.5 was familiar with IMDb62, one would expect significantly better accuracy from methods such as zero-shot chain of thought prompting or PS+ prompting. Instead, these methods produce results roughly analogous to a random guess, implying GPT-3.5's unfamiliarity with the IMDb62 dataset.\n\nWhile this observation is indicative, we are striving to ensure our claims remain consistent with the data we have on hand. We believe it's vital to differentiate between what we empirically observe and what we can confirm categorically. This distinction will be made more explicit in the limitation section of our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introducing linguistic features as prompts for AV seems to be novel. \n2. PromptAV yields better performance than baselines.", "reasons_to_reject": "1. It is not clear how PromptAV could address the limitation of data requirements since it is mentioned in the abstract and introduction. The paper did not provide a solution for saving data usage. If the justification is to use GPT-3.5 instead of training a model, one could argue that the paper actually requires more data as it depends on the performance of LLM.\n", "questions_to_authors": "Do you know if GPT-3.5 sees IMDb62 during training? ", "paper_topic_and_main_contributions": "The paper studies authorship verification (AV) and focuses on two problems: 1. data requirements and 2. lack of explainability. The paper proposes PromptAV to leverage LLM (GPT-3.5) and provide step-by-step stylometric explanation prompts. Specifically, PromptAV instructs LLM with linguistic attributes in order to capture author-specific nuances. The paper shows that PromptAV outperforms baselines on 0-short, 2-short, and 4-short settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8yS2LTaMoV", "reviewer": "Reviewer_ug9M", "summary": "", "questions": "1. Was the threshold for the confidence score fixed for all three methods?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your interesting questions. Your thought-provoking questions have significantly contributed to enhancing our paper's quality. We hope that we have comprehensively addressed the concerns and questions you presented. Based on the valuable discussion, we will make the necessary updates to our paper.\n\n---\n\n***W1**: First, the reliance on the pre-determined set of stylometric features. There are a lot of stylometric features in use in the community and common consensus on utility only holds for a few of them. So, this limitation may be detrimental.*\n\n**[Response to W1]**\n\nWe fully acknowledge that the field of stylometry is vast, with a plethora of features available for authorship analysis. It's indeed true that a consensus on the utility of many of these features remains elusive. The central aim of PromptAV was not to assert that a particular set of features are universally superior, but rather to showcase the flexibility and adaptability of the framework when working with different sets of stylometric features.\n\nPromptAV's design allows it to be feature-agnostic, meaning it has the potential to accommodate any feature set that researchers or practitioners might find relevant. This flexibility ensures that as the domain of stylometry evolves, and as new insights about feature utility emerge, PromptAV can continue to adapt and remain relevant.\n\nTo underscore this adaptability, we conducted a series of experiments using varying sets of stylometric features with PromptAV in zero-shot setting. The data clearly showcases that the performance varies based on the chosen feature set. The peak accuracy achieved with a 9-feature set emphasizes that PromptAV's success is contingent on the quality and appropriateness of the selected features.\n\n| Number of features|Accuracy|\n|-|-|\n|8| 0.587 |\n|9 | 0.602 |\n|10| 0.585 |\n|11| 0.564 |\n|12| 0.572 |\n\nWe genuinely believe that the field holds significant undiscovered potential, with combinations of features that could further elevate the performance of PromptAV. Our results only scratch the surface, and our intention is to spur the community into exploring these myriad possibilities. We will actively encourage the community to explore various feature sets, and we hope that collectively, we can push the boundaries of what is achievable in authorship verification using PromptAV.\n\n---\n\n***W2**: Second, the method was only compared with existing prompting methods (chain-of-through and PS+prompting). Comparing this approach to the state-of-the-art methods in AV is necessary to truly understand the utility and potential of the proposed approach.*\n\n**[Response to W2]**\n\nYour point about the comparison with only existing prompting methods is valid. Our primary aim was to juxtapose PromptAV with methods that operate under similar conditions and constraints. The methods we chose for comparison, namely the chain-of-thought and PS+ prompting, align closely with the premise and operating principles of PromptAV.\n\nWe certainly recognize the importance and relevance of benchmarking against state-of-the-art AV methods. However, the vast majority of these methods are supervised, leveraging large volumes of labeled training data to achieve their results. Their dependence on copious amounts of labeled data inherently places them in a different operating paradigm. For instance, BERT is able to achieve a high accuracy of 0.702 if it is trained on IMDb62 full training dataset with 74,400 samples. However, BERT will require large amount of training data to achieve the superior performance.  \n\nIn our study, we deliberately focused on a zero-shot AV setting, which, as you might agree, presents a uniquely challenging landscape. PromptAV, as well as the other prompting baselines we evaluated, operates without the benefit of labeled training data, relying instead on the generalization capabilities of language models. The essence of our research was to probe the extent to which such methods could navigate the complexities of the AV task in the absence of explicit training on it. Furthermore, it is essential to highlight that PromptAV focuses not just on accuracy but also on providing detailed step-by-step stylometric explanations.\n\nHowever, we acknowledge the value of your feedback and recognize that benchmarking PromptAV's performance against state-of-the-art methods, even with the stated caveats, could offer additional layers of insights into its efficacy and potential areas of enhancement. We will include a discussion on supervised and zero-shot AV task in our updated version, and report the performance of some of the state-of-the-art supervised AV methods.\n\n ---\n\n***Q1**: Was the threshold for the confidence score fixed for all three methods?*\n\n**[Response to Q1]**\n\nTo clarify, the threshold for the confidence score was not held constant across all methods. Instead, based on empirical observations, we determined specific optimal thresholds for each method to achieve the best performance. For instance, while we set the threshold for PromptAV at 0.2, the thresholds for PS+ prompting and CoT were fixed at 0.2 and 0.1 respectively. Note that this is for the zero-shot settings.\n\nWe have also responded to a similar question by Reviewer 4xL7. Specifically, we utilized the optimal thresholding approach primarily to facilitate a straightforward comparative assessment across the different prompting techniques. By doing so, we could establish a foundation for understanding each method's inherent strengths and weaknesses in a controlled setting. However, we acknowledge that in real-world applications, determining the ideal threshold could necessitate further tailoring to meet the intricacies of specific datasets and LLMs.\n\nThe derivation of the confidence score was empirically driven, grounded in preliminary experimentation. Our aim was to strike a judicious balance between false positives and false negatives. We have conducted an experiments to show the tradeoff on performance (in terms of accuracy) when selecting different threshold for the confident score for the various models. Note that experiments are conducted in zero-shot setting. We will include this experiment results in the appendix of our camera-ready version.\n\n| #Confidence Score Threshold | PrompAV | PS+ | CoT |\n|-|-|-|-|\n| 0.1|0.506|0.501|0.524|\n| 0.2|0.587|0.536|0.522|\n| 0.3|0.579|0.529|0.505|\n| 0.4|0.578|0.529|0.497|\n| 0.5|0.569|0.527|0.497|\n| 0.6|0.55|0.484|0.498|\n| 0.7|0.53|0.493|0.498|\n| 0.8|0.519|0.497|0.497|\n| 0.9|0.5|0.497|0.497|\n\n---  \n\n***Others**: Grammar Issues*\n\nThank you for pointing out the grammar mistakes in the paper. We have corrected them, and we will perform a more thorough proofreading in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Prompting language model for assessing stylistic similarity between text samples is a fresh perspective on AV. Leveraging the vast capabilities of large language models without corpus-specific training (as demonstrated through zero-shot and few-shot experiments) demonstrates significant potential. Further, the human comprehensible explanation in natural language may be very useful in legal and forensic contexts especially to users not well-versed in AI methods. ", "reasons_to_reject": "There are two weaknesses in the paper. First, the reliance on the pre-determined set of stylometric features. There are a lot of stylometric features in use in the community and common consensus on utility only holds for a few of them. So, this limitation may be detrimental. Second, the method was only compared with existing prompting methods (chain-of-through and PS+prompting). Comparing this approach to the state-of-the-art methods in AV is necessary to truly understand the utility and potential of the proposed approach.   ", "questions_to_authors": "1. Was the threshold for the confidence score fixed for all three methods?", "paper_topic_and_main_contributions": "The paper introduces an approach for prompting large language models (GPT 3.5 in this case) to evaluate stylistic similarity between two text samples for authorship verification (AV). The prompt used in the paper guides the language models towards using a pre-determined set of known stylometric features for evaluation culminating in the generation of a confidence score. This confidence score is thresholded to decide if the samples belong to the same author. The output generated includes human-comprehensible reasoning for interpretability. The prompting introduced in this paper, called PromptAV, is compared to existing prompt-based methods using zero-shot and few-shot evaluation protocols on data from IMDB62 (a common AV corpus). Effective accuracy ranges from 59% for zero-shot to a maximum of 67% for few-shot learning. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 2 line 106 steps to significantly improves the ability\nPage 3 line 136 PS+promptingm\nPage 3 line 203 textbfPromptAV\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces PromptAV, a novel approach that uses Large Language Models (LLMs) for the task of authorship verification, incorporating chain-of-thought to prompt the verification process. Reviewers recognized the significance and interest in the paper's focus on authorship verification. While acknowledging the paper's contributions, reviewers also raised some valid concerns, which the authors addressed during the rebuttal stage and committed to resolving in the final version.
Decision: Accept-Findings