HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis
Authors: Nafis Irtiza Tripto
Adaku Uchendu
Thai Le
Mattia Setzu
Fosca Giannotti
Dongwon Lee
Venue: EMNLP 2023 Findings (2023)
Keywords: Authorship analysis
Spoken text
Large Language Model
AI text detection
TL;DR: We present a benchmark comprising 17 human spoken text datasets with 3 LLM-generated spoken texts and perform authorship analysis on the benchmark.

Abstract: $\textit{Authorship Analysis}$, also known as stylometry, has been an essential aspect of Natural Language Processing (NLP) for a long time. Likewise, the recent advancement of Large Language Models (LLMs) has made authorship analysis increasingly crucial for distinguishing between human-written and AI-generated texts. However, these authorship analysis tasks have primarily been focused on $\textit{written texts}$, not considering $\textit{spoken texts}$. Thus, we introduce the largest benchmark for spoken texts - ${\sf HANSEN}$($\underline{H}$uman $\underline{AN}$d ai $\underline{S}$poken t$\underline{E}$xt be$\underline{N}$chmark). ${\sf HANSEN}$ encompasses meticulous curation of existing speech datasets accompanied by transcripts, alongside the creation of novel AI-generated spoken text datasets. Together, it comprises 17 human datasets, and AI-generated spoken texts created using 3 prominent LLMs: ChatGPT, PaLM2, and Vicuna13B. To evaluate and demonstrate the utility of ${\sf HANSEN}$, we perform Authorship Attribution (AA) \& Author Verification (AV) on human-spoken datasets and conducted Human vs. AI text detection using state-of-the-art (SOTA) models. 
While SOTA methods, such as, character n-gram or Transformer-based model, exhibit similar AA \& AV performance in human-spoken datasets compared to written ones, there is much room for improvement in AI-generated spoken text detection. The ${\sf HANSEN}$ benchmark is available at: https://huggingface.co/datasets/HANSEN-REPO/HANSEN

HANSEN:
Human and AI Spoken Text Benchmark for Authorship Analysis
Nafis Irtiza Tripto', Adaku Uchendu Thai Le
Mattia Setzu", Fosca Giannotti', Dongwon Lee'
'Tee Pennsylvania State University, USA [ni t5154, dongwon edu
MIT Lincoln Laboratory, USA
The University of Mississippi, USA, "ISTI-CNR, Pisa, Italy
adaku.. wchendu@ll.miee mit. edu, "thaile@olemiss. edu *mattia. setzuuunipi it, fosca. giannotti@sns.
Abstract
Create post match interview with tennis player Roger Federer who ... <rest of the prompt>. The reporter will ask exactly the following questions:
PROMPT:
Authorship Analysis, also known stylom- etry, has been an essential aspect of Natu- ral Language Processing (NLP) for long time. Likewise, the recent advancement of Large Language Models (LLMs) has made au- thorship analysis increasingly crucial for dis- tinguishing between human-wrilten and AI- generated texts. However, these authorship analysis tasks have primarily been focused on written texts, not considering spoken texts. Thus, we introduce the largest benchmark for spoken texts HANSEN (Human ANd ai Spoken tExt beNchmark). HANSEN encom- passes meticulous curation of existing speech datasets accompanied by transcripts, along- side the creation of novel Al-generated spo- ken text datasets. Together, it comprises 17 human datasets, and Al-generated spoken texts created using 3 prominent LLMs: Chat- GPT, M2. and V3C To evaluate and demonstrate the utility of HANSEN, we per- form Authorship Attribution (AA) Author Verification (AV) human-spoken datasets and conducted Human vs. AI spoken text de- tection using sffte-of-the-ar (SOTA) models. While SOTA methods, such as, character gram or Transformer based model, exhibit sim- ilar AA & AV performance in human-spoken datasets compared to written ones, there is much room for improvement in Al-generated spoken text detection. The HANSEN bench- mark is available at: httpps://uugingg: co/ to/ddatsstts/HANEENN REEENSEENN-REO/AASEEM
Q...............................
Q: What do you think Williams sisters' rivalry with one another has meant to sport?
HUMAN: don't 't know. I'd say the players themselves have been great for the sport. don't know if the matches themselves is what people remember that much.
ChatGPT: The Williams sisters have been amazing for tennis, and their rivalry has been a huge part of that. They'v 'e both achieved so much in their careers, and they ve pushed each other to be better.
PaLM2: The Williams sisters' rivalry with one another has been one of the most compelling and enduring in tennis history. The rivalry has mcant a great deal to the sport of tennis.
Vicuna 13B: The Williams sisters' rivalry has been in- credible for the sport. It's been amazing to see the two of them compete against each other and push each other to be their best.
Snippet of the HANSEN Al-text samples, Given the actual question (part of the prompt), the HU- MAN (ROGER FEDERER) answer vs three LLMs: ChatGPT, PaLM2, and Vicuna13B.
Figure 1:
2007). Both have been extensively researched as text classification problems due to their substan- tial impacts on various applications, such as author profiling, author forensic analysis, resolving copy- right disputes, and similar issues (Neal al., 2017). Currently, most of the research in NLP text classifi- cation focuses on written text due vast amounts of text data available for training and evaluation (Kowsari et al., 2019). However, "text" is also inherent in spoken language as textual represen- tation of what individuals say, commonly known as spoken text (Biber, 1991). Although spoken lan- guage has always existed before written language (i.c., considering human history of language as twelve-inch ruler, written language has only ex- isted for the "last quarter of an inch" (Wrench et al., 2008)), has not received much attention from the NLP community. Simultancously. recent adyancements in speech-to-texx technology have
1 Introduction
Authorship analysis is a longstanding research area in NLP that has garnered significant attention over the years. Two tasks at the core of authorship analysis Authorship Attribution (AA) and Author- ship Verification (AV). AA the process of iden- tifying the author of a text document (Kjell al., 1994). Similarly, AV determines whether two doc- uments are written by the same author (Halteren,

---

Written example
Spoken example
Remarks
Today, the Commission considers adopting a final rule to enhance the disclosures related to share bayhacks support this final mile because it will increase[...] [..]
Good moming. 1 am pleased to join the Investor
Advisory Committee. [-.]
Written statement
look forward hearing about your potential
and spoken speech from US Security
Firss, the rule will require issuers disclose periodically the prior period's daily buyback activity. This will include such information as the date of the purchase, the amount of shares repurchased, and the average purchase price for the date. [-.]
recommendation today reganding customer account
statements. We look forward to your input ahout private markets.[... We also look forward to your discussion on open end funds.
Exchange (SEC) Commission
Chairperson Gary Gensler about corresooning issues.
Second, issuers required by the rule to provide disclosure shout their buyback programs. Such disclosure will include details about the objectives or rationales for the buyback as well as the process|.... I Different people nuturally have differing life experiences and differing viewpoints- which inevitably resulis in varying agendar
Now,
turn your panel the oversight of investment advisers. I am glad you are discussing advisers, because [...
do want try steak, yeal. duukk uss....] I-I don't think it's bad. I think it's good thing. Erm, because I-1 did by be veggie for, think it was six months I did it, but I gained lot weight. think probubly because didn't i really know the other options could have ate instead. I think was eating more carbs and stuff like that. So I was gaining weight so stoppod[...]
Written essay and informal interview transeript from sample participant in Aston Idiolects Corpus.
when implemening the law accordling to their own personal discretion. In this respect- - Birks is correct that on over-rellonee on conseience alone to provide equitubie solutions inevitably leads to a messy framework of case low that would result in judges navigoting uncertotely by feel than by the sold path of precedent...
Written and spoken text samples from same individual. Written texts contain linking words, passive voice (Akinnaso, 1982), complicated words, and complex sentence structures. However, spoken texts contain more first & second person usages, informal words (Brown et al., 1983), grammatically incorrect phrases (Biber et al., 2000), repetitions, and other differences. Sentences in spoken texts are also shorter (Farahani et al., 2020).
Figure 2:
expanded the availability of spoken text corpora, enabling researchers explore new avenues for NLP research focused on spoken language. Identification of speakers from speech has been successfully addressed through various audio fea- tures, with without the availability of associ- ated text information (Kabir et al., 2021). Whether speaker identification can be achieved solely based on how individuals speak, as presented in spoken text, remains mostly unexplored in the existing lit- erature. However, the rise of audio podcasts and short videos in the digital era, driven by widespread social media usage, has increased the risk of pla- giarism as individuals seek to imitate the style and content of popular streamers. Therefore, spoken text authorship analysis can serve as valuable tool in addressing this situation. It can be challenging for several reasons. First, text is not the only mes- sage we portray in speaking since body language, tone, and delivery also determine what we want to share (Berkun, 2009). Second, spoken text is more casual, informal, and repetitive than written text (Farahani al., 2020) Third. the word choice and sentence structure also differ for spoken text (Biber, 1991). For example, Figure 2 portrays the difference between written and spoken text from the same individuals. Therefore, discovering the author's s style from the spoken text can be an excit- ing study leveraging the current advanced AA and AV techniques.
podcasts, and YouTube videos is expected to be- come increasingly prevalent. Figure 1 exemplifies how various LLMs can adeptly generate replics to an interview question. While substantial research efforts are dedicated to discerning between human and Al-generated text, also known as Turing Test (TT) (Uchendu et al., 2023), evaluating these meth- ods predominantly occurs in the context of written texts, such as the Xsum (Narayan et al., 2018) SQuAD (Rajpurkar et al., 2016) datasets. Thus, spoken text authorship analysis and detecting Al- generated scripts will be crucial for identifying pla- giarism & disputing copyright issues in the future.
To tackle these problems, we present HANSEN (Human ANd ai Spoken (Ext beNchmark), a bench- mark of human-spoken & Al-generated spoken text datasets, and perform three authorship anal- ysis tasks to better understand the limitations of existing solutions on "spoken" texts. In summary, our contributions are as follows.
We compile & curate existing speech corpora and create new datasets, combining 17 human- -amannspoken datasets for the HANSEN benchmark.
We generate spoken texts from three popular, re- cently developed LLMs: ChatGPT, PaLM2 (Anil et al., 2023), and Vicuna-133 (Chiang et al., 2023) (a fine-tuned version of LLaMA (Touvron et al., 2023) user-shared conversations, com- bining ~21 samples in total, and evaluate their generated spoken text quality.
On top of classical plagiarism detection prob-
lems, there is a looming threat of synthetically gen-
We assess the efficacy of traditional authorship analysis methods (AA, AV, & TT) on these HANSEN benchmark datasets.
crated content from LLMs. Therefore, using LLMs, such as ChatGPT, for generating scripts for speech,

---

2 Related Work
Type: Speech
Conversation
Authorship analysis: Over the last few decades, both AA and AV have been intensively studied with a wide range of features and classifiers. N- gram representations have been the most common feature vector for text documents for long time (Kjell et al., 1994). Various stylometry features, such as lexical, syntactic, semantic, and structural are also popular in authorship analysis (Stamatatos, 2009; Neal et al., 2017). While machine learn- ing (ML) & deep learning (DL) algorithms with n-gram different stylometry feature sets have been popular in stylometry for long time (Neal al., 2017), Transformers have become the state- of-the-art (SOTA) model for AA and AV, as well as many other text classification tasks (Tyo et al., 2022). Transformers, such as BERT (Bidirectional Encoder Representations from Transformers) (De- vlin et al., 2018) and GPT (Generative Pre-trained Transformer) (Radford et al., 2019), are generally pre-trained on large amounts of text data before fine-tuning for specific tasks (Wang et al., 2020), such as AA & AV. Also, different word embed- dings, such as GLOVE (Pennington al., 2014) 4) and Fast (Bojanowski et al., 2017), can cap- ture the semantic meaning between words and are employed with DL algorithms for AA as well.
Human
Interview
Datasets
Datuset
Prompt
HASEN
Formality
Benchmark
subset
Toplcs
Tasks
Datasets
ChatGPT
PaLM2
Vicunal3B
Figure 3: HANSEN benchmark framework
utilize various speech features, and DL approaches to recognize the speaker's s emotions, such as anger or sadness, and subsequently define the speaking style based on these emotional cues.
Building the Benchmark: HANSEN
Existing conversational toolkits and datasets, such as ConvoKit (Chang et al., 2020), prioritize analyz- ing social phenomena in conversations, thus limit- ing their applicability in authorship analysis. Simi- larly, current LLM conversational benchmarks, in- cluding HC3 (Guo et al., 2023), ChatAlpaca (Bian et al., 2023), and XP3 (Muennighoff et al., 2023), primarily focus on evaluating LLMs' question- answering and instruction -oolrooiion-folooiing abilities rather than their capability to generate authentic spoken language in conversations speeches. Therefore, we introduce the HANSEN datasets address this gap, incorporating human and Al Al-generated spoken text from different scenarios as portrayed in Figure 3, thereby providing valuable resource for author- ship analysis tasks. The datasets of HANSEN are available through Python Hugging Face library.
LLM text detection: LLMs' widespread adop- tion and mainstream popularity have propelled re- search efforts toward detecting Al-generated text. Supervised detectors work by fine-tuning language models specific datasets of both human & Al- generated texts, such the GPT2 detector (So- laiman et al., 2019), Grover detector (Zellers et al., 2019). ChatGTT detector (Guo al., 2023), and others. However, they only perform well within the specific domain LLMs for which they are fine- tuned (Uchendu et al., 2023; Pu et al., 2023). Statis- tical or Zero-shot detectors can detect AI text with- out seeing previous examples, making them more flexible. Therefore, it has led to the development of several detectors, including GLTR (Gchrmann al., 2019), DetectGPT (Mitchell al., 2023), GPT-zero (Tian, 2023), and watermarking based techniques (Kirchenbauer et al., 2023).
3.1 Human spoken text datasets
The HANSEN benchmark comprises 17 human datasets, where utilize several existing speech corpora and create datasets. Our contribution involves curating and preparing these datasets for authorship analysis by leveraging metadata infor- mation, aligning spoken text samples with their re- spective speakers, and performing necessary post- processing. Table shows the summary of the human datasets.
Style from speech: While some studies have at- tempted infer the author's style from speech, they primarily focus on the speech emotion recognition task and rely on audio data. These works (Shi- nozaki et al., 2009; Wongpatikaseree et al., 2022)
Existing datasets selection: When constructing the HANSEN benchmark datasets, several crite- ria guided our selection of speech corpora. Firstly, we consider datasets with readily available tran- scripts (e.g., TED, BNC, BASE) or the ability to

---

Type Topic
Sample
Dataset
Description
Characterisdics
#Speakers #Sumples #Tokens
Definition
TED
Speech
Transcripts of TED talks collected from the
Mostl unscripted
Entire
3349
4122
7.19M
talk
Mixed
official website.
& informal
speech
It is a collection biglist-languigg Spotity
Speech
Mostly unseripted
Entire
Spotify
podcust episodes transcripts compiled
17490
105357
684,99M
Mixed
(Clifton 2020).
& informal
speech
The British Academic Spoken Binglish
All utterance
Specch/
Mostl unseripted
(BASF) corpus (Thompson and Nesi, 2001)
in a session
Conversation
informal
1.66M
comprises lectures and seminars recerded
lecture/
Academic
discussion
in a university.
seminar]
We utilize spokea portion British
Conversation
Unscripted
All utterance
National Corpus (BNC) (Consortium 2007).
BNC
Daily life
informal
in a conversation
2461
90153
53.6M
It is collection of late twentieth century
topies
British English language.
conversations
Conversation
A contemporary version of the previous
All utterance
Unseripted informal
BNC14
Daily life
corpus (Love et al., 2017). Both datasets
in conversation
459
1868
4.65M
topics
are guthered from various real life contexts.
The MSU Switchboard Dialog Corpus
(Stolcke al., 2000) includes
Conversation pre-delined
All utterance in a phone
MSU
phone calls between two purticipants. The callers ask receivers questions about child care, recycling. the news media, and other provdedd topics.
Unseripted & informal
Switchboard
440
2310
1.91M
lopics
PAN23
Speech/
It is the training set PAN-23 AV task and originally introduced by (Petyko al., 2022).
conversation Daily life
Unseripted & informal
utterance session
36
17672
14.4M
Idiolect
The spoken part contains interviews and
corpus)
topics
speech transcriptions from students.
Originally introduced (Liye al., 2016).
All answers in interview
Interview Sports
Unseripted informal
Tennis
it contains Tennis single post match interviews in major toumaments 2007-2015.
358
6467
6,58M
It contains interviews with CEO companies
financial persons
individual very Few interview available for each CEO
associated with stock markets. We created it from the available public transcripts of these interviews from three different
since
CEO interview
Interview Financial
Unseripted & formal
6248
15072
7.01M
sources: CEO-today magazine, Wall Street
Journal, and Seeking Alpha website.
Tr comprises interview transcripis featuring various celebrities soureed from YouTube. Leveraging the original corpus established by (Nagrani al.).. we extract the You Tube transcripts to form this version.
Interview Celebrity
Unscripted & informal
Voxceleb
in
3753
57702
16:05M
interview
British
Question/
Initially introduced by (Zhang et al., 2017),
Scripted/ Unseripted
All utterance in a session
Parliament
Answer
this dataset contains questions & answers
1065
23849
18.66M
(BP)
Politics
British House Commons.
formal
We utilize English portion of the original Vaxpopuli (Wang al., 2021)
Scripted/
Speech Politics
All utterance in session
Voxpopuli
Unscripted formal
924
24549
73.1M
dataset, which contains the raw data from
Europcan Parliament recordings.
We created this datnset by compiling the transcripts from the popular talk show.
Face the
Mostl unseripted formal
Talk-show Mixed
Each utterance
Nation (FTN)
Nation, by episode led host who discusses
Each
1613
84942
5.24M
contemporary topics with multiple guests. Initially introduced (Mao it contains podcasts from the US Life
Ali utterance from an Act in episode
US Life
2020)
Talk-show Mixed
Unscripted & informal
Prdcast (USP)
165
3881
1.5IM
radio program from 1995 to 2020.
compile dutaset from the Security Exchange Commission (SEC) press release. It contains the transcripts of speech (spoken text) and statements (written text) from various personnel.
Speech Financial
Scripted formal
Entire speech
SEC
59
1101
1.46M
Argumeniative
This datuset, iniraduced by (Zhing al., 2016)
Cin contain
Each utterance
Debate
conversation
contains transcripts debates held as part
470
1.82M
Mixed
Intelligence Squared Debates.
characteristies
This mussive corpus consists of collection
Argumentative
of cases from the U.S. Supreme Court, along
Mostl unscripted & formal
Supreme Court
utterance session
conversation
with transcripts of oral arguments. The transcripts were collected from the Oyer. website (Urofsky, 2001)
8978
66758
79.66M
Legal
Table Summary of the HANSEN Human datasets.

---

extract transcripts (e.g.. Voxceleb (Nagrani et al.) via YouTube automated transcriptions). Secondly, speaker information for each sample was required, leading to the exclusion of corpora such as Coca (Davies, 2015). Finally, we also emphasize datasets where speakers did not recite identical content or scripts. excluding datasets like Ljspeech (Ito and Johnson, 2017) and Movie-Dialogs Corpus (Danescu-Niculescu-Mii and Lee, 201 1) for their lack of spontaneity & originality in representing natural spoken language.
Dataset characteristies: The benchmark is rep- resentative of a diverse range of naturally occur- ring dialogue forms, encompassing monologue (speech/podeess//ecuur))) dialogue (conversation/mn terview/argument), and multi-party dialogue (talk show). It also includes scripted and unscripted speech across spectrum of formality levels. The transcripts of these datasets are often manually gen- erated and annotated as part of the official corpora (e.g., BASE, BNC, BNC14, MSU), obtained di- rectly from official sources (e.g.. TED, USP, Ten- nis) or automatically generated through speech to text techniques (e.g.. Spotify & Voxpopuli). Given the topic-dependent nature of authorship analy- sis (Sari et al., 2018), the benchmark comprises datasets covering specific or diverse topics.
New dataset creation: We have introduced three new datasets within our benchmark'. The SEC dataset comprises transcripts from the Security Ex- change Commission (SEC) website, encompass- ing both spoken (speech) and written (statements) content from various personnel in the commission. The Face the Nation (FTN) dataset compiles tran- scripts from the Face the Nation talk show by CBS News, where episodes feature discussions on con- temporary topics with multiple guests. The CEO interview dataset contains interviews with financial personnel, CEOs of companies, and stock market associates assembled from public transcripts pro- vided by CEO-today magazine, Wall Street Journal, and Seeking Alpha. For authorship analysis, we retained only the answers from the interviewees, excluding the host's questions.
3.2 AI (LLMMaggeeeateed spoken text dataset
With the growing societal impact of LLMs, we will soon observe their extensive in generating scripts for speech and guidelines for interviews or conversations. Thus, it motivates us to include AI Al-generated spoken text HANSEN. Since chat- based LLMs can follow instructions and generate more conversation-like text than traditional LLMs (Ouyang et al., 2022), we have utilized three recent prominent chat-based L.Mss: ChatGPT (gpt3.5- turbo), PaLM2 cchat-bison@00)) @001), and Vicuna- 13B in our study. We also ensure the spoken nature of the generated texts and evaluate their quality.
Dataset curating: We have followed a system- atic process for all datasets our benchmark prepa- ration. involves eliminating tags, URLs, meta- data from the text, and aligning each utterance with its respective speaker for generating labeled data for authorship analysis. To maintain consistent text lengths and contextual coherence within utterances, we adjust the span of each sample, detailed in Table 1. In cases where sample was excessively long, such as an lengthy TED talk, we split into smaller segments to ensure uniform lengths. Fur- thermore, we partition each dataset into training, validation, and testing subsets while preserving a consistent author ratio across these partitions.
Spoken text generation: LLMs predomi- nantly trained on written text from diverse sources, including BookCorpus (Zhu et al., 2015). Open WebText (Aaron Gokaslan*, 2019), and Wikipedia (Merity et al., 2017), containing a small portion of spoken texts, while the exact proportion is un- known. Therefore, effective prompt engineering is crucial for generating spoken text.
Different metadata associated with each speaker and text samples can be utilized to construct ef- ficient prompts. For instance, by leveraging ele- ments like talk descriptions, speaker details, and talk summaries, LLMs can produce more coherent TED talk samples rather than providing the topic only. To create spoken texts from LLMs, we uti- lize subsets from five human datasets (TED, SEC, Spotify, CEO, & Tennis), chosen for their rich metadata and the involvement of notable public figures as speakers. Additional insights regarding prompt construction, dataset selection, and related attributes can be found in the Appendix.
LLM
TED
Spotify
SEC
CEO
Tennis
Total
ChatGPT
3491
2778
301
2021
8652
PaLM2
2461
2040
284
59
1991
6835
Vicuna13B
2868
2524
290
60
2008
7750
Table 2: -.Le-rrerrttedd samples in each category
link of the websites & other details in Appendix

---

Dutasets
SEC
PAN
ChatGPT
PuLM2
VicunoL3B
Features
written
spoken
written
spoken
written
spoken
written
spoken
written
spoken
Avg length
5.50=0.31
5.30s0.29
4.31
4.90s0.53
4.78
Avg word sen
25.4544.44
22.16=382
22.11
17.87=2.75
20 17+2.73
16.853300
17.14:5.33
14,74
21 1188333.88
18.0253.55
Voe: richness
0.4920.06
0.5100.55
0:40:0.09
0.32:0.04
0:45.0.05
04550.08
0.5:80144
0.38:0.07
0.5150
0.430007
Long word fraction
0.29:0.04
0.26+0.03
0.22:0.07
0.10=0.02
0.26+0.07
0.2240:07
0.22=0.06
0.20=0.08
0.26+0007
0.21+0.07
1st person count
6.98+5.90
8.0645.54
9.82=11.84
28.70=14.66
3.3965.77
14.95e11.09
5.39=7.47
19.04=12.24
2.7765.74
15.34
1.83s2.83
4.226.16
5.3726.43
1.75s4.19
4.0728.79
1.0022667
6.34g5.73
Readability score
36.89211.37
45.73:9.74
50.11=15.54
80,3946.27
51.94114448
62 26.15.39
63. 63.99219.22
68.53=16.6]
53.13:15:82
62.67415:47
1.19:0.17
1.13:0.19
1.1340.66
1.12+0.15
1.25=0.51
1.00=0.20
1.3440.34
1.124021
Misspelling %
12,50=2.89
13.37=2.44
13.96+3.61
16.64+2899
10.04+2.03
11 11,3+22377
10.49
11,75#6.29
10.6743.08
10.86+2.77
Table 3: Feature difference between written vs spoken text.
The value in corresponding cell indicate the meantvariance of that feature. Red colored cell indicate statistical significance was not found for that feature.
BERT score (TED)
BERT (Spotify)
BERT score (SEC)
BERT score (CEO)
BERT (lennis)
CharEPT
PaLM 2
Viuum 138
cãs
GRUEN Scone (TE.)
GRUEN Score (Spottty)
GRUEN Score (SBC)
GRUEN Score (CEO)
GRUEN Score (Tannis)
ChetGPt
P
Vituna 138
DM score (TED)
DW score (Spotify)
DIV score ISEC)
DIV scone (CEO)
DW score (lnnis)
ChatciP
PaLM 2
Vicuna 1:B Human
Figure 4: Distribution of BERT, GRUEN, Text diversity (DIV) score for three LLMs (the higher, the better).
Dataset
TED (Speech)
Spotify (Speech)
SEC (Speech)
CEO (Interview)
Tennis (Interview)
# speakers
10
10(L)
50
10())
10
10(L)
30
10(L)
10
10(L)
100
Char n-gram
0.95
0.89
0.93
0.98
0.91
0.82
0.9
0.54
0.86
0.58
0.99
0.83
0.84
Stylometry
0.71
0.67
0.64
0.87
0.83
0.61
0.68
0.69
0.46
0.68
0.35
0.91
0.89
0.66
FastText
0.73
0.59
0.6
0.93
0.84
0.47
0.68
0.67
0.42
0.59
0.61
0.28
0.87
0.87
0.46
Bert-AA
0.69
0.53
0.48
0.9
0.87
0.54
0.55
0.64
0.22
0.72
0.74
0.55
0.76
0.84
0.22
BERT-R
0.89
0.72
0.75
0.98
0.94
0.78
0.83
0.81
0.63
0.76
0.83
0.62
0.91
0.94
0.59
Table 4: Macro-Fl score for different AA-methods for variable number of speakers. 10(L) indicates the setup where each LLM has been considered as speaker (7 human speakers with 3 LLMs).
Evaluating spoken nature of Al-generated texts: To ensure the spoken nature of Al-generated texts, corresponding written texts in the same context are essential. We select subsets for each category and instruct LLMs to generate written articles based on their respective spoken texts. In HANSEN, only two human datasets (SEC and PAN) provide writ- ten and spoken samples from the same individuals. Stylometry properties of these samples from hu- man and L.LM-generated datasets compared, and a student I-test (Livingston, 2004) was used determine if there were significant differences in these stylometric features between written and
spoken texts. The results (Table 3) indicate that LLM-generated spoken and written text exhibits similar trends to human datasets, including varia- tions in word lengths, first/second person counts, vocabulary richness, and other linguistic features, thus validating that LLMs have successfully gener- ated spoken text with a spoken-like quality in most scenarios.
ChatGPT and Vicunal 13B demonstrate similar differences compared to humans, whereas PaLM shows minor differences in some cases, indicat- ing the need for more training with human-spoken texts. It is essential to consider the distinct nature

---

of the SEC and PAN datasets; SEC speeches and statements are rigorously scripted and reviewed due to their potential impact on financial markets and the economy (Burgoon et al., 2016), while the PAN dataset comprises mainly informal spoken and written texts. Additional detailed findings are available in the Appendix.
Dataset
TED
SEC
Tennis
of speakers
20
20(L)
20
20(L)
20
20(L)
Char n-gram
0.92
0.37
0.7
0.33
0.95
0.42
Stylometry BERT-ft
0.68
0.39
0.55
0.29
0.63
0.43
0.84
0.51
0.73
0.45
0.84
0.67
Table 5: Ablation study comparing AA AA-20()) as top human speakers and corresponding samples from 3 L.LMs (considering as 20 distinct classes) against sce- nario with only the top 20 human speakers. Results show substantial performance drop, indicating LLMs' ability to impersonate human speakers.
Quality of Al-generated spoken text: Addition- ally, we use different automatic evaluation metrics, such as BERT score (Zhang* et al., 2020), GRUEN (Zhu and Bhat, 2020), MAUVE (Pillutla et al., 2021), text diversity score, and readability score (Kincaid et al., 1975) to evaluate quality of the gen- crated text (Figure 4). In interview datasets, the consistently higher
speaker and performs AA ten speakers (7 hu- mans 3 LLMs). Also, texts from the same LLM are generated using specific prompt (prompt- ing them to symbolize the actual human persona). Thus, LLM is supposed to exhibit a different per- sona in all its instances. Therefore, we have per- formed an ablation study using samples from the top 5 human speakers (with most samples) and their corresponding samples generated by ChatGPT, Vi- cuna 13B, and PaLM2, resulting in total of 20 classes for comparison. We employed conventional text classification evaluation metrics, including Ac- curacy, macro Fl score, Precision, Recall, and Area under the curve (AUC) score. Table 4, 5 and 6 present the macro Fl score and demonstrate that character (char) n-gram performs best in most sce- narios, with BERT-ft being close contender. Our observations align with the AA findings of Tyo et al. (2022) on written text datasets. However, the performance of BERT-AA is subpar when directly applied to spoken text datasets, but finetuning them improves performance substantially.
BERT scores for all LLMs than 0.8 (out of 1.0) and mostly similar, the original questions guide the generated texts. How- ever, for open-ended speech datasets like TED and Spotify, PaLM2 shows lower BERT scores com- pared to other models. The GRUEN scores across all datasets are uniformly distributed, indicating variation in linguistic quality among individual- generated samples. PaLM2 exhibits lower GRUEN scores than the other models. Surprisingly, we observe lower text diversity for human texts, con- tradicting existing Al vs. human analyses (Guo et al., 2023) but supporting the repetitive nature of human-spoken texts (Farahani et al., 2020).
4 Authorship Analysis on HANSEN
We conduct three authorship analysis tasks using HANSEN and evaluate how existing SOTA meth- ods perform in the context of "spoken" texts. De- tails on AA, AV, & TT methods are provided in Tables 13, 14, 15 in the Appendix section. We run the experiments five times and report the average.
4.2 Author Verification (AV)
Author verification (AV) is binary classification problem that, given pair of spoken texts (Ti, T2), detects whether they were generated by the same speakers or different speakers. We have used PAN AV baselines: N-gram similarity, Predictability via Partial Matching (PPM), and current PAN SOTA methods: Adhominem (Boenninghoff et al., 2019), Stylometry feature differences (Weerasinghe et al., 2021), and finetuned BERT. We have used tradi- tional PAN evaluation metries for AV (Bevendorff et al., 2022), including F1, AUC, c@ F F_0.5u, and Brier scores. Table reports the Fl score for several datasets. Unlike AA, Adhominem showed the best performance in AV, with finetuned BERT being particularly notable.
4.1 Author Attribution (AA)
Author Attribution (AA) is closed-set multi class classification problem that, given a spoken text T, identifies the speaker from a list of candidate speak- ers. We have primarily performed AA on human datasets as well as considering each LLM as an in- dividual speaker. We have utilized N- N-grams (char- acter & word). Stylometry features (WritePrint (Ab- basi and Chen, 2008) LIWC (Pennebaker et al., 2001)), FastText word embeddings with LSTM, BERT-AA (Fabien al., 2020), Finetuned BERT (BERT-ft) as AA methods.
Additionally, we have evaluated the realistic see- nario where each LLM performs as an individual

---

Type/Topic
Conversation (daily- -ife topiclother) Interview
Speech (Political)
Arguments Court
Talk shows
Dataset
BASE
BNC
BNC14
MSU
PAN
Voveeleb
BP
Voxpopuli
Dehate
USP
FTN
AVG
Char n-gram
0.98
0.89
0.92
0.33
1
0.86
0.98
0.94
0.91
0.83
0.52
0.58
0.84
Stylometry
0.84
0.71
0.79
0.24
1
0.68
0.84
0.76
0.79
0.74
0.42
0.42
0.70
FastText
0.8
0.64
0.76
0.33
1
0.77
0.84
0.79
0.73
0.58
0.28
0.51
0.69
Bert-AA
0.94
0.7
0.66
0.26
1
0.87
0.69
0.87
0.82
0.72
0.3
0.64
0.71
BERT-A
0.99
0.88
0.2
0.28
1
0.87
0.92
0.93
0.82
0.85
0.4
0.65
0.82
Bold and underhied values represent each datasel's 's highesi and Sennndd--ighhs--eeriiie
method (based on macro FT).
Table 6: AA result for N=10 speakers in different datasets, Results with higher value of N are present in Appendix.
Type/Topic
Speech Spotify
Conversation
Interview
Political Voxpopuli
Legal
Talk shows
Dataset
TED
SEC
BASE
MSU
PAN
CEO
Tennis
Court
USP
FTN
AVG
Char n-gram
0.86
0.67
0.67
0.78
0.66
0.67
0.68
0.68
0.64
0.61
0.65
0.66
0.68
PPM
0.85
0.65
0.63
0.76
0.55
0.72
0.64
0.62
0.55
0.66
Feature Diff.
0.87
0.69
0.72
0.47
0.99
0.72
0.76
0.73
0.67
0.51
0.7
Adhominem
0.84
0.88
0.81
0.96
0.62
1.0
0.93
0.91
0,91
0.92
0.72
0.87
BERT-ft
0.91
0.83
0.64
0.93
0.46
0.67
0.81
0.73
0.79
0.81
0.67
0.77
Table 7: AV results (Fl score) for several datasets. Results for other metrics and datasets are presented in Appendix.
LLM
TED
Spotify
SEC
CEO
Tennis
Methods
SEC(w)
SEC(s)
PAN(w)
PAN(S)
M: 0.59
M: 0.53
M: 0.51
M: 0.69
M: 0.51
Char
Fl: 0.58
Fl: 0.57
Fl: 0.45
Fl: 0.41
ChatGPT
R: 0.82
R: 0.7
R: 0.74
R: 0.76
R: 0.82
N-gram
A=(-0.31)
A (-0.32)
A: (-0.55)
A: (-0.59)
P
M: 0.82
0.88
M: 0.65
R: 0.87
M: 0.71
R: 0.81
M: 0.82
R: 0.96
M: 0.59
R: 0.97
Stylometry
A: (-0.30)
F1: F1:
A: (-0.27)
0.41
A: (-0.82)
Fl: 0,18
A: (-0.79)
Fl: 0.21
Vicuna
M: 0.74
M: 0.58
M: 0.68
M: 0.75
M: 0.58
Finetuned
F1: 0.5
FI: 0.53
Fl: 0.32
FI: 0.35
13B
R: 0.83
R: 0.8
R: 0.8
R: 0.85
R: 0.93
BERT
A: (-0.33)
A: (-0.30)
A: (40.68
A: (-0.65)
Table 8: The Mauve score (M) and avg recall value (T?)
Table 9: Results combining both spoken & written samples from same individuals. SEC/PAN (w) specifies that the training set was written only and test set was spoken texts by same speakers. SEC/PAN (s) specifies the vice-versa. Each cell value is the macro Fl score and the difference PAN/SEC dataset in the AA-10 class problem.
(considering all detectors) for each LLM. A higher M
means the distribution is more similar to humans. A higher R means that the LLM is easily detectable. The violet color indicates better performance for LLM (higher Mauve score and lower recall value), while the red indicates worse performance (and vice versa).
score (A) with the original
vary in spoken text. Furthermore, the low recall of Roberta-Large specificallyy for ChatGPT, may be attributed to the distinct nature of spoken texts gen- erated by ChatGPT compared to the written texts from previous GPT models.
4.3 Turing Test (TT) for Spoken Text
We frame the human vs AI spoken text detection problem as Turing Test (TT), a binary classifica- tion problem that, given a spoken text T, identifies whether it is from a human or AI (LLM). We have utilized several supervised and zero-shot detectors: OpenAl detector, Roherta-Large, DetectGPT, and GPT Zero in our study. Table 10 highlights the results of different TT methods in various dataset- s/LLMs. Notably, no single method emerges as the apparent "best" option, as performance varies sub- stantially across different datasets. Overall, the OpenAl text detector excels in speech datasets (TED, Spotify, SEC), while DetecGPT performs better in interview datasets (CEO, Tennis). How- ever, all methods exhibit limitations across various settings, with either low precision or low recall, For instance, GPT Zero demonstrates low preci- sion scores in interview datasets and TED talks, suggesting perplexity and burstiness measures may
Discussion
Character n-gram dominates in AA but strug- gles in AV: While character n-gram performs best in AA for most datasets, it underperforms in the AV task. More intricate DL models, such as Ad- hominem, excel in AV, consistent with findings in written text datasets (Tyo et al., 2022). Larger DL models tend to outperform smaller models when the datasets have more words per class, leading to better AV performance since it only has two classes (Tyo et al., 2022). Character n-grams outperform word n-grams by notable margin (5%-10% in general), emphasizing the potential enhancement of AA performance through the inclusion of infor- mal words (e.g.. "eh," "err," "uhh").

---

Dataset
TED (S, mixed)
Spotify (S, mixed)
SEC financial)
CEO (1. financial)
Tennis (I. sports)
Methods
palm222 vicuna
gpt3.5
palm2
vicuna
gpt3.5
2alm2p vieuna
gpt3.5 palm2 vicuna
gpt3.5
palm2
vicuna
0.87
0.87
0.92
0.91
0.91
0.95
0.89
0.74
0.9
0.82
0.89
0.88
0.4k
0.48
0.5
OpenAl
0.91
0.94
0.92
0.69
0.96
0.81
0.86
0.87
0.93
0.6
0.95
0.82
0.97
1.0
0.98
defector
0.89
0.9
0.92
0.78
0.94
0.88
0.88
0.8
0.91
0.7
0.92
0.84
0.64
0.65
0.66
1.0
1.0
1.0
0.72
0.9
0.88
0.99
0,99
0.99
0.89
0.93
0.92
0.97
0.99
0.99
Roberta
0.52
0.79
0.69
0.31
0.76
0.61
0.5
0.77
0.63
0.62
0.92
0.77
0.43
0.9
0.73
Large
0.68
0.88
0.82
0.43
0.82
0.72
0.66
0.86
0.77
0.73
0.92
0.84
0.6
0.94
0.84
0.8
0.52
0.74
0.68
0.88
0.88
0.59
0.86
0.63
0.74
0.79
0.81
0.97
0.85
0.95
Detect
0.85
0.65
0.91
1.0
0.97
0.97
0.67
0.7
0.65
0.83
0.81
0.89
0.89
1.0
1.0
GPT
0.82
0.58
0.82
0.81
0.92
0.92
0.63
0.77
0.64
0.78
0.8
0.85
0.93
0.92
0.97
0.6
0.49
0.62
0.91
0.89
0.93
0.81
0.78
0.81
0.57
0.58
0.58
0.62
0.58
0.61
GPT
0.99
0.93
0.98
0.78
0.77
0.8
0.93
0.84
0.96
1.0
0.97
0.95
1.0
0.96
1.0
Zero
0.75
0.64
0.76
0.84
0.83
0.86
0.87
0.81
0.88
0.73
0.73
0.72
0.77
0.73
0.76
Table 10: TT results on different datasets (Speech (S) or Interview (D) for three LLMS: CharGPT (gpt3.5), PaI. 2, and Vicuna-13B with precision, recall, and score sequentially each cell. The bold scores indicate the highest performing method (based on F1 score). The underlined scores indicate low precision scores (predicting most texts as Al-generated). The bold & underlined scores show a low recall score (can not detect most Al-generated texts).
AA and AV performance is dataset specific: Our study highlights significant performance vari- ations in AA& & AV across datasets, influenced by factors such as dataset type, domain, and modal- ity. Daily-life conversation datasets (BASE, BNC, PAN) generally yield high F1 scores, except for MSU, which comprises simulated conversations with predefined topics, potentially limiting speak- ers' natural speaking styles. Conversely, talk-show- type datasets (USP, FTN) exhibit poor AA& & AV performance due to a skewed distribution of sam- ples from show hosts and the influence of Com- munication Accommodation Theory (Giles, 1973), suggesting speech adaptation to host styles. Individuals written & spoken samples are vastly different: Our results affirm the distinctions be- tween individuals' written and spoken texts, consis- tent with prior corpus-buse based research (Biber et al., 2000; Farahani et al., 2020). We also conduct an ablation study on SEC and PAN datasets, training on written texts only and testing on spoken texts (and vice versa). The results in Table 9 show a substantial decrease in the macro fl score for both character n-gram and BERT-ft. Also, stylometry features exhibit poor performance, underscoring the stylistic differences between the two text forms and thus emphasizing the significance of separate authorship analysis for spoken text.
in humans compared to LLMs (Guo al., 2023), observe an opposite trend that humans tend to exhibit more repetitions in their speech. These find- ings suggest a further investigation into the specific characteristics of Al spoken language.
How close are LLM-generated spoken texts to humans? The AA-10(L) and AA-200 experi- ments reveal decrease in the fl score compared to AA with all human speakers, indicating that LLM- generated spoken texts exhibit greater similarity to other speakers in the experiments. It highlights the potential for further research in training LLMs to replicate individual spoken styles accurately.
Which LLM is the winner? Identifying the best LLM for spoken text generation remains an open question, influenced by multiple factors. PaLM2 demonstrates a lower GRUEN DIV score and is more easily detectable than other LLMs in all datasets. On the contrary, ChatGPT exhibits lower Mauve scores, indicating its generation outside of human distribution, but its higher average recall value exhibits that it is difficult to detect.
6 Conclusion
We present HANSEN, a benchmark of human and Al-generated spoken text datasets for authorship analysis. Our preliminary analysis suggests that existing SOTA AA & AV methods behave simi- larly for spoken texts in most scenarios. However, the stylistics difference in spoken & written for the same individuals and Al-generated spoken texts show different characteristics than existing notions, emphasizing the need for a more nuanced under- standing of the spoken text.
AI Al-generated spoken and written texts have dif- ferent characteristies: Table 8 reveals negative correlation between the Mauve score and the de- tection rate of LLMs. challenging the assumption that higher Mauve scores indicate harder-to-detect texts (Uchendu et al., 2023). Similarly, contrary to existing studies that highlight greater text diversity

---

Limitations
mark, other security applications can be discovered to mitigate other risks which LLMs pose. Finally, we believe that the benefits of this benchmark, out- weighs the risks.
While the HANSEN benchmark encompasses mul- tiple human and Al-generated spoken text datasets, it is essential to note that they are currently lim- ited to English. Variations in spoken text struc- tures and norms among humans differ significantly across languages (Crystal, 2007). Consequently, the performance of authorship analysis techniques may vary when applied to other languages (Halvani et al., 2016). Additionally, numerous spoken text datasets exist in various settings that are not yet in- cluded in the HANSEN benchmark. Furthermore, due to the computational demands of generating texts from LLMs, our study focuses on three spe- cific LLMs while acknowledging the availability of other LLM models that could be explored in future research.
A Acknowledgements
This work was in part supported by U.S. Na- tional Science Foundation (NSF) awards #1 820609, #1934782, #21 14824, and #2131144, and by EU HE project SoBig receivinng funding from the E6rrрpa Union's 's Horizon 2020 research and innovation programme under grant agreements No. 654024, 871042, 101079043 and National Recov- ery and Resilience Plan - Prot. IR0000013 Avviso n. 3264 del 28/12/2021, and by project PNRR M4C2 Investimento 1.3, Partenariato Esteso PE00000013 "FAIR Future Artificial Intelli- gence Research" Spoke "Human-centered AT", funded by the European Commission under the NextGeneration EU programme.
While our study focused on LLMS and their de- tection, is important to acknowledge the ongoing arms race between the development of LLMs and LLM detectors (Mitchell et al., 2023). Therefore, our TT study may not encompass all LLM detec- tors, and it is possible that newly developed de- tectors outperform existing ones these datasets. We deliberately refrained from finetuming any de- tectors on the spoken datasets as we believe that the ovaluation of detectors should be conducted in open-ended scenarios. Although human evalua- tion of Al-generated texts still presents challenges (Clark et al., 2021), we consider it crucial area for future work. Given the subjective nature of speech (Berkun, 2009), incorporating human evaluations can provide valuable insights into the quality of generated texts and further enhance our understand- ing of their performance in real-world applications.
References
Ellie Pavlick Stefanie Tellex Aaron Gokaslan
Vanya Cohen 2019. Openwebtext corpus. http: //Skylion007 gi wtw.tttt.....oooeeooeooeoe hoo/ppenwebbextoorpus..
Ahmed Abbasi and Hsinchun Chen. 2008. Writeprints: A stylometric approach to identity-level identification and similarity detection in eyberspace. ACM Trans- actions on Information Systems (TOIS), 26(2):1-29.
Ahmed Abbasi, Abdul Rehman Javed, Farkhund Iqbal, Zunera Jalil, Thippa Reddy Gadekallu, and Natalia Kryvinska. 2022. Authorship identification using ensemble learning. Scientific reports. 12(1):9537.
Niyi Akinnaso. 1982. On the differences between spoken and written language. Language and speech, 25(2):97-125.
Ethics Statement
While the ultimate goal of this work is to build the first large-scale spoken texts benchmark, includ- ing --M-geeerated texts, we understand that this dataset could be maliciously used, We evaluate this dataset with several SOTA AA, AV, and TT models and observe that there is room for improvement. However, we also observe that these findings could be used by malicious actors to improve the quality of LLM-generated spoken texts for harmful speech, in order to evade detection. Due such reasons, we release this benchmark on huggingface's data repo to encourage researchers to build stronger and more robust detectors to mitigate such potential misuse. We also claim that by releasing this bench-
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint 2305
Scott Berkum, 2009, Confessions of public speaker. O' O'Reilly Media, Inc.".
Janek Bevendorff, Berta Chulvi, Elisabetta Fersini, An- nina Heini, Mike Kestemont, Krzysztof Kredens, Maximilian Mayerl, Reyner Ortega-Buent Piotr Pęzik, Martin Potthast, al. 2022. Overview of pan 2022: Authorship verification, profiling irony and stereotype spreaders, style change detection, and trigger detection. In European Conference on Infor- mation Retrieval, pages 331-338. Springer.

---

Ning Bian, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, and Ben He. 2023, Chatalpaca: A multi- turn dialogue corpus based on alpaca instructions. https:// /ttps://gi/i thub. wtw.co/cccaaao http//aacc//haatpaaeee
Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu. Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosic Jones. 2020. 100,000 podcasts: A spoken En- glish document corpus. In Proceedings of the 28th International Conference on Computational Linguis- rics, pages 5903-5917, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Douglas Biber. 1991. Variation across speech and writ- ing. Cambridge University Press.
Douglas Biber, Stig Johansson, Geoffrey Leech, Su- Conrad, and Edward Finegan. 2000. Longman grammar of spoken and written English. Longman London.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- lingull representation learning at scale. pages 8440-842 8451.
Benedikt Boenninghoff, Steffen Hessler, Dorothea Kolossa, and Robert M Nickel. 2019. Explainable authorship verification in social media via attention- based similarity learning. In 2019 IEFE Interna- tional Conference on Big Data (Big Data), pages 36-45. IEEE
BNC Consortium et al. 2007. British national corpus. Oxford Text Archive Core Collection.
David Crystal. 2007. How language works: How bubies babble, words change meaning, and languages live or die. Penguin.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the associa- tion for computational linguistics, 5:135-146.
Cristian Danescu- Niwulescuuuuiiiii and Lillian Lee. 2011. 1. Chameleons in imagined conversations: new ap- proach to understanding coordination of linguistic style in dialogs.
Gillian Brown, Gillian D Brown, George Yule, Gillian Brown, and Brown Gillian. 1983. Discourse analysis. Cambridge university press.
P Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011.
Judee Burgoon, William J Mayew, Justin Scott Gi- boney, Aaron C Elkins, Kevin Moffitt, Bradley Dorn, Michael Byrd, and Lee Spitzley. 2016. Which spo- ken language markers identify deception in high- stakes settings? evidence from earnings conference calls. Journal of Language and Social Psychology. (5(2):123-157
Mark Davies. 2015, Corpus of Contemporary American English (COCA).
Jacob Devlin, Ming-W Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04800
Jonathan P Chang, Caleb Chiam, Liye Fu, An- drew Wang, Justine Zhang, and Cristian Danescu- Niculescu-Mizie 2020. Convokit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 57-60.
Maël Fabien, Esaú Villatoro-Tello, Petr Motlickk, and Shantipriya Parida. 2020. Bertaa: Bert fine-tuning for authorship attribution. In Proceedings of the 17th International Conference on Natural Language Pro- cessing (ICON), pages 127-137
Mehrdad Vasheghani Farahani et al. 2020. Metadis- course in academic written and spoken english: A comparative corpus-based inquiry. Research in Lan- guage (RiL), 18(3):319 341.
Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuaa. Imsys. org (accessed 14 April 2023).
Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics: System Demonstrations, pages 111-116.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong. Suchin Gururangan, and Noah A Smith. 2021. All that's' human'i, is not gold: Evaluating hu- man evaluation of generated text. In Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7282-7296.
Howard Giles. 1973. Accent mobility: A model and some data. Anthropological linguistics, pages 87- 105.
Biyang Guo, Xin Zhang, Ziyuan Wang, Mingi Jiang. Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597
Alexandra A Cleland and Martin Pickering. 2006. Do writing and speaking employ the same syntactic rep- resentations? Journal of Memory and Language, 54(2):185-195

---

Hans Van Halteren. 2007. Author verification by lin- guistie profiling: An exploration of the parameter space. ACM Transactions on Speech and Language Processing (TSLP), 4(1):1-17.
Edward H Livingston. 2004, Who was student and why do we care so much about his t-test? 1. Journal of Surgical Research, 1 118( (3(1):58665....
Fu Liye, C. Danescu, and I Lillian Lee. 2016. Tie-breaker: Using language models to quantify gender bias in sports journalism. Computation and Language.
Oren Halvami, Christian Winter, and Anika Pflug, 2016. Authorship verification for different languages, gen- res and topies. Digital Investigation, (6633--43.
Robbie Love, Claire Dembry, Andrew Hardie, Vaclav Brezina, and Tony McEnery. 2017. The spoken bnc2014; Designing and building spoken corpus of everyday conversations. International Journal of Corpus Linguistics, 22(3)3119344.
Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. MGTBench: Benchmark- ing
Text Detection. CoRR
abs/2303. 14822.
Keith Ito and Linda Johnson. 2017. The speech dataset. https://keithito.cc LJ-Speech-Dataset..
Huanru Henry Mao, Shuyang Li, Julian J. McAuley, and Garrison W. Coturll. 2020. Speech recogni- tion and multi-speaker diarization long conversa- tions. In Interspeech 2020, 2/ Annual Conference of the International Speech Communication Associa- tion, Shanghai, China, 25-29 October 2020.
Muhammad Mohsin Kabir, Muhammad F Mridha, Jung- pil Shin, Israt Jahan, and Abu Quwsar Ohi 2021. A survey of speaker recognition: Fundamental theories. recognition methods and opportunities. IEEF Access, 9779236-79263.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. International Conference on Learning Repre- sentations.
Mike Kestemont, Enrique Manjavacas, Iii Markov, Janek Bevendorff. Matti Wiegmann, Efstathios Sta- matatos, Martin Potthast, and Benno Stein. 2020. Overview of the cross-domain authorship verification task at pan 2020. In CLEF (Working Notes).
Fric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. De- tectgpt: Zero-shot machine-generate text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning. volume PMIR 202, pages 24950-24962.
Peter Kincaid, Robert P Fishburne Jr. Richard Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability in- dex, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Re- search Branch.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao. M Saiful Bari, Sheng Shen, Zheng Xin Yong. Hai- ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al- banie, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza- tion through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15991-1611 Toronto, Canada, Association for Computational Linguistics.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. 2023. A watermark for large language models. In Proceed- ings of the 40th International Conference on Machine Learning, volume PMLR 202, pages 17061-17084.
Hendrik Kirchner, L Ahmad, S Aaronson, and J Leike. 2023. New ai classifier for indicating ai-written text. OpenAI.
Arsha Nagrani. Joon Son Chung, and Andrew Zisser- man. Voxceleh: a large-scale speaker identification dataset. Telephony, 3:33-039.
Bradley Kjell, W Addison Woods, and Ophir Frieder. 1994. Discrimination of authorship using visual- ization. Information processing & management, 30(1):141-150.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797-1807.
Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Hei- darysafa, Sanjana Mendu, Laura Barnes, and Donald Brown. 2019. Text classification algorithms: A sur- vey. Information, 1504):150.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to teee-to-seeee so-squennce pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Tempestt Neal, Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon Woodard. 2017. Surveying stylometry techniques and applica- tions. ACM Computing Surveys (CSuR), 1-36)
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al.

---

2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Andreas Stolcke, Klaus Ries, Noah Coccaro, Eliza- beth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
James W Pennebaker, Martha E Francis, and Roger J Booth. 2001. Linguistic inquiry and word count: Liwe 2001. Mahway: Lawrence Erlbaum Associates, 71(2001):
William Teahan and David Harper. 2003, Using compression based language models for text cate- gorization. Language modeling for information re- trieval, pages 141-165.
Jeffrey Pennington, Richard Socher, and Christopher D Manning, 2014. Glove: Global vectors for word rep-
resentation. In Proceedings of the 2014 conference on empiricul methody in natural language processing (EMNLP), pages 1543.
Paul Thompson and Hilary Nesi. 2001. The british academic spoken english (base) corpus project. Lan- guage Teaching Research, 5(3):263-264.
Marton Petyko, Llciaa Busso, Tim Grant, and Sarah Atkins. 2022. The aston forensic linguistic databank (fold). Language and Law/Linguagem e Direito.
Edward Tian. 2023. Gptzero. Online; accessed 23-Mar- 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun. Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap be- tween neural text and human text using divergence frontiers. Advances Neural Information Process- ing Systems, 34:4816-4828
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1399
Jiameng Pu, Zain Sarwar, Sifat Muhammad Abdullah, Abdullah Rehman, Yoonjin Kim, Parantapa Bhat- tacharya, Mobin Javed, and Bimal Viswanath. 2023. Deepfake text detection: Limitations and opportu- nities. In 2023 IEEE Symposium on Security and Privacy (SP). pages 1613-1630. IEEE.
Jacob Tyo, Bhuwan Dhingra, and Zachary C Lipton. 2022. On the state of the art authorship attribution and authorship verification. arXiv e-prints, pages arXiv-2209
Adaku Uchendu, Thai Lc, and Dongwon Lee. 2023. At- tribution and obfuscation of neural text authorship: A data mining perspective. ACM SIGKDD Explo- rations Newsletter, 25(1):1-18.
Alec Radford. Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188):
Melvin I Urofsky. 2001. The oyez project: Us supreme court multimedia database. The Journal of American History, 88(2):753.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPop- uli: A large-scale multilingual speech corpus for rep- resentation learning, semi-supervised learning and interpretation. Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993-1003, Online. Association for Computational Linguistics.
Yunita Sari, Mark Stevenson, and Andreas Vlachos. 2018. Topic or style? exploring the most useful features for authorship attribution. In Proceedings of the 27th International Conference on Computational Linguistics, pages 343-353.
Takahiro Shinozaki, Mari Ostendorf, and Les Atlas. 2009. Characteristics of speaking style and impli- cations for speech recognition. The Journal of the Acoustical Society of America, 126(3):1500-1510.
Chengyi Wang. Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020. Bridging the gap between pre- training and fine-tuning for end end-to-end speech trans- lation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9161-9168.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu. Alec Rad- ford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the so- cial impacts of language models. arXiv preprint m:Xiv:1908.0020)
Janith Weerasinghe, Rhia Singh, and Rachel Greenstadt. 2021. Feature vector difference based authorship ver- ification for open-world settings. In CLEF (Working Notes), pages 2201-2207.
Efstathios Stamatatos. 2009. A survey of modern au- thorship attribution methods. Journal of the Ameri- can Society for information Science and Technology, 60(3):538-556.
Konlakorn Wongpatikaseree, Sattaya Singkul, Narit Hnoohom, and Sumeth Yuenyong. 2022. Real-time

---

end-to-end speech emotion recognition with cross- domain adaptation. Big Data and Cognitive Comput- ing, 6(3):79.
Jason S Wrench, James C McCroskey, and Virginia P Richmond. 2008. Human communication in every- day life: Explanations and applications. Pearson Education.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32.
Justine Zhang, Ravi Kumar, Sujith Ravi, and Cris- tian Canescu-iiculesu--iiill 2016. Conversational flow in oxford-style debates. In Proceedings of NATCL-HLT, pages 136-141.
Justine Zhang, Arthur Spirling, and Cristian Danescu- Niculescu-Mizie 2017. Asking too much? the rhetor- ical role of questions in political discourse. In 2017 Conference on Empirical Methods Natural Lan- guage Processing, EMNLP 2017, pages 1558-1572.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. International Conference on Learning Representations.
Wanzheng Zhu and Suma Bhat. 2020. Gruen for eval- uating linguistic quality of generated text. In Find- ings the the Association for Computational Linguistics: EMNLP 2020, pages 94-108.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Con- ference on Computer Vision (ICCV).

---

A HANSEN benchmark datasets & access
from datasets import Load_datase
import pandas as pd
task on smals dataset AA_TED= 1oad_dataseee
A.1 Sources of new datasets
We have created three new datasets in our bench- mark. The SEC dataset compiled from the press release of the Security Exchange Commis- sion (SEC)² website. It contains the transcripts of speech (spoken text) and statements (written text) from various personnel in the commission. We have created the Face the Nation (FTN) dataset by compiling the transcripts from the popular talk show program Face the Nation by CBS News. Each episode is led by a host who discusses contem- porary topics with multiple guests. We need to ex- tract the speaker and align the corresponding utter- ance for a speaker from the raw transcripts. Finally, the CEO interview dataset contains interviews with different financial personnel, CEOs of companies, and other stock market associates. We have com- piled this dataset by collecting the available public transcripts of these interviews from three differ- ent sources: CEO-today magazine*. Wall Street Journal and Seeking Alpha We have removed the questions from the interview hosts for author- ship analysis purposes and only kept the answers provided by the hosts.
name=
"AA_TED_small" split= pplit='trainn
df df_AA AA pd.DataFrame.. frome.frromcoee from_dict (AA_TED)
AV task on SEC dataset
AV_SEC 1oad_ dataset(
"HANSEN -REPO/EPPAA REPO/HANSEN
name= AV_SEC" split= split='test') test')
df_AV = pd. DataFrame from_dice (AV_SEC)
# TT task on human vs ChatGPT on TED
TT TT_ChatGPT__ER TED = (
"HANSEN RRPO HAASSEN name trrin')
'TT_ChatGPT_TG '
df_TT = pd. DataFrame from_
IT_ChatGPT_TED)
Figure 5: Python code for loading HANSEN datasets from Hugging Face API
"good-usage" form. We have also checked the existence of Personal Identifiable Information (PII) using off- fff-the-shelf automated tools?, for all three modules. While it found the full name, affiliation, and program/website contact information, did not find any sensitive PII, such as identification numbers or personal contact information.
A.2 Benchmark release & access
B Selection of subsets for LLM prompting
The datasets HANSEN can be accessed through the Python Hugging Face library, as shown in Fig- ure 5. In order to comply with the distribution rights of HANSEN datasets and tackle the existing challenges of Al-generated misinformation prop- agation, HANSEN datasets consist of three mod- ules: (1) Open-source data/existing datasets that are free re-distribute, (2) Open-source data that are public but may not be re-distributed, as users have to download/scrape themselves, and (3) AI- generated data that we have generated. Module (1) is accessible by default, (2) is partially included such that we only provide download-related info (e.g., URLs, line numbers, and Readme files), and scripts for hwwwloadiing/craapg/ppppaannnnni but do not contain the data, (3) is available by completing
Due to the computational costs of generating text from LLMs, we chose to work with a subset of Hansen human datasets. First, we select the datasets with enough metadata and well-known speakers (such TED speakers. SEC commission- ers, or Tennis players). TED, Spotify, and SEC datasets are mostly monologues (speech category). and CEO & Tennis are interview datasets.
For TED datasets, we removed the talks with music or instrumental focused. Spotify and Tennis contain numerous samples in the original version. Therefore, we considered the top speakers with the most samples and used them for LLM generation. Also, for Spotify, the speaker is not individual, such a tutorial channel or multi-party collaborations. Similarly, for the CEO dataset, we considered the subsets from ceo-today magazines since the questions are specific and guest-focused rather than the overall
removed the samples where
Phttps: http://wwwwwseee sec. http///csssseecehesseeeeeeeeeeee statements htttps: / //www. cbsnews. www/faceeoeceeeeeetceeoo the- inteenatiin// 'https: htttps://wwwr www.odaymaggzine.ecc.o wtt.ee/ccccooaccc///eeeee
opinion/ wttp:tthe--coneeiieeee.
"https://www.ww http:///www.wws..comm/orrttteea http:///////eeetrraaaannnnn/e/
topics/transcripts
"https://seekingallpp htt.:ringagaaaaaacuuhhhhooo con/author/
ceo-interviews

---

financial situation like the subset in Wall Street Journal.
E Experimental details for authorship analysis
C Prompts used for generating spoken-text
The details about our AA, AV, and TT methods are discussed in Tables 13, 14, and 15. Since the devel- opment of new methods is not the primary purpose of our paper, we used them with the default con- figurations and srprrpparaeeter settings in most cases. For the text generations with LLM, we used the top_p 0.7 to ensure more creativity well maintaining a coherent text and max_tokens to the highest limit for that LLM.
In our experiments, we explored different prompt techniques to enhance the coherence and seman- tic similarity of the generated content by LLMs. We observed that providing sufficient context with the prompt yielded better results. For the inter- view dataset categories, we included the original questions asked by the host. In the case of TED talks, we utilized the original talk description and the opening lines to allow the LLMs to learn the subtle style of the talks. We used the BART sum- marizer tool (Lewis et al., 2020) for the Spotify podcast and SEC speeches to obtain summaries, which were then used as prompts. Additionally, we explicitly instructed the LLMS to generate plain text avoid any unwanted formatting. Table shows the prompts used for different categories. the case of speeches, we instructed the L.LMs to generate speech content using the same word counts as the original speech or up to the maximum allowed number of tokens (lypically around 1024 tokens or approximately 800 words for LLMs).
We applied a pre-prcessiing step to ensure com- patibility with various methods to achieve a uni- form text length range for all authorship tasks. This involved removing samples from datasets that fell below below-specified thresholds (100 for AA and AV, 200 for TT) as specific methods, such as Finetuned BERT, Adhominem, or OpenAl Detector, require specific text lengths. We sometimes combined mul- tiple samples from the same conversation or inter- view to reach the desired length, as observed in datasets like MSU or CEO. Additionally, we em- ployed sentence splitting for large samples such as TED or Spotify to create new samples with text lengths that remained within the maximum allowed token limits (approximately 1000 tokens) for dif- ferent methods.
D
More about evaluating spoken nature of the Al-generated texts
Table 16 shows the AA results for both small and large versions of the datasets. We consider large version N different for these datasets ensure substantial samples per class for classification. We observe that performance drops more substantially for Transformer based methods when the number of speakers N increases. Therefore, it validates that DL methods underperform if per-class word counts decrease (Tyo et al., 2022). Also, character n-gram performs considerably better than word n- gram in all scenarios. Similarly, Table 17 shows the AV results with different metrics for all datasets. While observe similar trend for classifiers in both AA and AV tasks compared to written text, the overall performance of these methods is less than different written datasets, as observed in previous studies (Stamatatos, 2009; Neal et al., 2017; Abbasi et al., 2022; Tyo et al., 2022). This leaves room for further investigation regarding a more nuanced analysis of spoken texts.
In section 3, we provide a brief analysis to show a comparison of LLM-generated spoken vs. writ- ten (Table 3) in parallel with the human spoken vs. written and how the differences align with the humans. To further validate, we have per- formed small-scale study on a subset (100 sam- ples from each) of the LLMM-generated spoken (L.) and corresponding written (Lw) for the TED dataset and consider another human spoken (H₂) and written (Hw) in different domains (H from Xsum (Narayan al., 2018) datasets: news ar- ticles, H, is from BASE dataset in HANSEN: academic conversations) to show the domain in- dependence. We measure the stylistic difference using the cosine distance of features in Table 10. We observe dist(H, I < dist Hw: T.a) and dist(H,, L,) < dist(H, Lw)) which can provide an overall idea that our LLM-ggnerated spoken datasets contain more similarity with the spoken language rather than the written ones. Figure portrays the distribution difference between some features in the written& & spoken samples.
We have run the experiments for each setup five times for the AA, AV, and TT tasks and reported the average in all tables. In most cases, we get the exact results for the methods for AA & AV, except

---

Category Prompt
Generate a TED talk as <SPEAKER SAPE>,, who is <SPEAKER DIO>>. The talk should be around
<WORD COUNT RANGES, The talk description as follows:
TED
<TALK_ DESCRIPPIITTOONN
The original talk starts as follows:
SHEWSSAARTING SENTENCES FROM_ORIGINAL ORIGINAL TALK>
Generate Spotify podcast <SPEAKER NAMESaround <WORD COUNT RANGE>words.
Spotify
Generated text should be plain text only. The original podcast summary is as follows:
<SUMMARY_ S____F_TTHEE SFF_THE__RRR ORIGINALL,
Generate a speech Security Exchange Commission persom: <SPEAKER LEEER__AMEaabuut
<ORIGINAL_ TITLL_ OF THE SPEECH>. The speech should be around <WORD_ COUNT RANGE>
SEC
words. The summary of the original speech is as follows:
<SUMMARY OF THE ORIGINAL SPEECH>
Generate interview with the guest <GUEST_NNE NAME>. The following is the interview description: <INTERVIEW DESCRIPTIONS. The host will ask the following questions:
CEO
<ORIGINAL QUESTIONS COTTIIONSSSSKDDDDE AAS_SSKDDBBB _B HOSTT>
Create a post match interview with tennis player <PLAYER who has <WONILOST>the match against <OPPONENT LITEETTNNAAII NAMESat
<TOURNAMENT NAMME _NAMES.
Tennis
The reporter will ask exactly the following questions:
<ORIGINAL_ QUESTIONS ASKED TO THE _PLAYER>
Table 11: Prompts used generating spoken texts different categories
ChattPP
RALM
Vicuna
Mann spokin
wittten 42:ben
sritlen spoken
wilten
sporio
spoten
4.0
4.5
3.0
3.5
0.0
0.5
4.0
5.0
5.5
0.0
0.5
4.0
4.5
5.0
5.5
6.0
6.5
4.0
5.0
5.5
6.0
0.5
4.0
4.5
5.0
5.5
5.0
5.5
SEC
PAN
ChatGPT
PALM
Vicuna
wittton spocen
written spoken
writen spoien
spoken
0.7
0.3
2.4
0.6
0.2
0.3
0.4
0.5
0.6
0.7
Voc Nichencca
Voc Richness
10c. Richness
Vod Richress
Figure 6: Avg. Word length and Vocabulary richness distribution of the different datasets for written and spoken text
Method
Change of text property
the overall result comparison. Although we ob- serve minor fluctuations in the text property across different runs within limited subset of our test- ing (e.g., z-score from DetectGPT or Perplexity from GPT-Zero), it is essential to note that the out- put, indicating the likelihood of Al/human origin, remains consistent across all our experimental sce- narios. Table 12 summarizes the findings for the TT methods.
OpenAl detector No score or text property provided
Score for each label changes
Roberta-Large
average 0.007+0.0053
Z-score changes on average 0.003+10.077
DetectGPT
Perplexity per line changes on average 1.05=0.087
GPT-Zero
Burstiness changes on average 3+4.27
Table 12: Effect of multiple runs on the TT tasks. Change labels was not observed for any methods.
A discussion about written and spoken texts of humans: The overall dimension and differences between written and spoken language from sta- tistical perspectives
those that include random initialization, such as character n-gram for AV with grid search. How- ever, the standard deviation of these results lies in the 0.0001-0.003 range, which will not impact
well-studied problem in various corpus linguistics (Biber, 1991; Biber et al., 2000; Brown et al., 1983). While there is a notice-

---

AA method
Description
N-gram
We utilize the chur, word level n-grams (represented IF-IF scores) with Logistic Regression (LR) classitier We utilize the combination of both Linguistic Inquiry Word Count (LIWC) (Pennebaker et al., 2001) and WritePrint (Abbasi and Chen, 2008) as stylometry features with LR classifier. LIWC analyzes the text based on over 60 categories representing various social, cognitive, and alfective processes. WritePrint extracts lexical and syntactic features from the text, including char, word, letter, bigram, trigram, vocabulary richness, pos-tags, punctuation, and function words.
Stylometry
We represent texts using FastText (Bojanowski et al., 201 7) word emheddings since they can utilize the subword level information and train an LSTM model for the classification
BERT-AA
Initially proposed by Fabien et al. (2020), it combines pre-trained BERT with dense layer for classification. As line-tuned LM have benn SOTA text classification tasks, we fine-tune BERT (bert-base-cased) on each dataset training set and evaluate the test set.
Finetuned BERT
Table 13: Description of the AA methods for spoken text.
AV method
Description
This method works as the PAN AV task baseline, where char n-grams represent cach text. Cosine similarity is computed between text puirs, and threshold values i and are optimized to adjust verification scores. Initially developed by (Teahan and Harper, 2003). it another bascline for PAN AV tasks. Given pair of texts, it estimates the entsooy of the second text using the first text's ' PPM model and vice versa. A I.R model Matching (PPM) is then used generate verification score based the mean and absolute difference of the two cross-cntropies.
N-gram
Prediction
by Partial
The method proposed by (Weerasinghe al., 2021) achieved the second-highes performance utilized stylometric features, such as character and POS n-grams, special characters, function words, vocabulary richness, and unique spellings to represent text as feature vectors and used the LR trained on the absolute difference between text pairs.
PAN-2022
Feature
differences
Initially introduced by (Boenninghoff' et al., 2019). this method is the highest- perffes-prrfonming in recent PAN AV tasks (Kestemont et al., 2020). As the foundation for a Siamese network, this appmach uses a hierarchical Bil STM setup with Fasttext word embeddings and custom word embeddings learned using a leveell CNN, Finetuned BERT Similar to AA, we fine-tuned the BERT for each datusets training set with the concatenation of text pains as sample.
ADHOMINEM
Table 14: Description of the AV methods for spoken text
able distinction between spoken and written text from a syntactic standpoint (Cleland and Pickering, 2006), whether an individual's 's subtle style is mir- rored in both formats has yet to be answered com- putationally. Although the ablation study in our paper (on SEC and PAN datasets, Table 9) initially suggests visible similarity, the result should not be considered conclusive. More parallel datasets for human individuals are needed to ensure conclu- sive evidence on whether individuals' unique style is represented in both forms. However, creating such parallel datasets manually will add unwanted bias. Therefore, it leaves room for future explo- ration a separate study. Also, training LLMs to make them capable of replicating individuals' written and spoken styles may be possibility that can shed more light on the problem.
(>80% on average); we do not observe such perfor- mance in spoken texts. Additionally, the overall re- sult is less than the written counterpart. While fine- tuning language models for TT on these datasets would likely result in improved performance (He et al., 2023), relying solely on fine-tuning is not the ultimate solution. Therefore, future detectors should also consider the spoken text characteristics to make the detection more robust.
A discussion about TT task on Al-generated spoken text: Although we observe the best- performing method in each category scores around 90% fl in most scenarios, lower precision or recall value seems significant problem for all detectors (Table 10). Also, recent evaluation studies on other benchmarks (He et al., 2023; Mitchell et al., 2023) (which are primarily written) show that the perfor- mance of DetectGPT or GPT Zero is very high

---

TT method
Description
OpenAl developed this fine-tuned GPT model, which estimates the possibility of a text being Al-generated, OpenAl detector specificlly from the GPT family. We have uccessed this detector from the official website (Kirchner el ul., 2023)
and considered "Possibly/Likely Al-generated" tags Al-generated spoken text.
Roberta
was initially developed the GPT-2 output detector model, obtained by fine tuninu RoBERTa large
-Large
model with the oulputs of the 1.S--prrameter GPT-2 model (Conneau el al., 2020).
DetecGPT (Mitchell et al., 2023) a zero-shot Al-generated text classifier by generating perturhed samples from the original text and calculating their prohability under the model parameters.
DetectGPT
GPT-Zero (Tian, 2023) utilizes perplexity: measure the complexity variations sentences to determine whether the text is Al-generated.
and Burstiness: compare the
GPT-Zero
Table 15: Summary of Al-generated spoken text detection methods
= 10 speakers
Dataset
TEDSEC
BANE
Debate
MSU
BP
Court Voxceleb VoxpopuliSpotily USP FTN CEOPAN
Samples #
393 1208
4492
2066
1739
2915
2257
382
4715
29.4K
1557
4341
42.2K
249721.9K 1338 2446
Char n-gram
0,95 0,9
0.98
0.89
0.92
0.99
0.83
0.33
0.98
0.91
0.86
0,94
0.98
0,52
0.58
0.85
Word n-gram
0.93 0.76
0.94
0.73
0.87
0.98
0.69
0.19
0.98
0.88
0.8
0.93
0.95
0.48
0.49
0.65
Stylometry
0.71 0.68
0.84
0.71
0.79
0,91
0.74
0.24
0.84
0.79
0.68
0.76
0.87
0.42
0.42
0.6
FastText
0.73
0.64
0.76
0.58
0.33
0.84
0.73
0.77
0.28
0.56
0.59
Bert-AA
0.69 0.55
0.94
0.7
0.66
0.76
0.72
0.16
0.69
0.82
0.87
0.87
0.9
0.3
0.46
0.72
Finetuned BERT
0.89 0.83
0.99
0.88
0.9
0.91
0.85
0.26
0.92
0.89
0.87
0.93
0.98
0.4
0.64
0.76
N
30 for SEC, FTN, USP;
N
50 for TED, PAN; N
700 for others
Samples #
1219 1589
4492
10K
6320
6848
4577
1978
15.4K
70.7K
12K
17.7K
145K
282033.7K
3520
8544
Char n-gram
0.94
0.93
0.73
0.66
0.84
0.7
0.09
0.9
0.8
0.74
0.68
0.81
0.25
0.38
0.58
Word n-gram
0.82 0.43 0.82
0.48
0,46
0.72
0.51
0.06
0.86
0.68
0.59
0.67
0.14
0.34
0.35
Stylometry
0.64 0.46 0.64
0.54
0.51
0.66
0.45
0.04
0.56
0.42
0.43
0.39
0.61
0.24
0.27
0.35
FastTeex
0.6 0.42 0.61
0.43
0.37
0.46
0.34
0.05
0.5
0.28
0.38
0.33
0.47
0.18
0.26
0.28
Bert-AA
0.48 0.22 0.8
0.52
0.21
0.22
0.48
0.03
0.3
0.43
0.59
0.39
0.54
0.13
0.3
0.55
Finetuned BERT
0.75 0,63 0.84
0.78
0.56
0.56
0.64
0.04
0.63
0.77
0.66
0.77
0.77
0.16
0.32
0.62
Table 16: Results for the AA task small and large number of speaker (N). The values indicate the Macro-FI score for the proposed method. The bold and underlined values indicate the highest and second highest scores for specific datasets.
Datuset
TED SEC BASEBNC BNCI4Tennis Debatee Parliament Court Voxceleb VoxpopuliSpotify USP FTN CEOPAN
-Samples #
26.4 3244 16.4K 22.8 18.2K 15.3K
19.4K
6.4K
20.9K
29.4K
28.9K
21.1K
29.7K
10.1K 13.4K 7.9K 17.8K
score
N-gram
0.57
0.6
PPM
0.58
0.65
0.65
0.57
Feature difference
0.94
0.8
0.88
0.8
0.86
0.73
0.44
0.8]
0.81
0.6
0.77
0.69
0.52
0.8
0.99
Adhominem
0.8
0.79
0.94
0.92
0.73
0.92
0.55
0.84
0.88
0.88
0.76
0.87
0.89
0.68
0.89
1.0
Finetuned BERT
0.96
0.69
0..88
0.89
0.8
0.9
0.97
0.52
0.83
0.87
0.87
0.8
0.9
0.72
0.94
0.56
F1
N-gram
0.86
0.71
0.66
0.61
0.61
0.64
0.67
0.68
0.67
PPM
0.85
0.63
0.76
0.68
0.68
0.72
0.71
0.57
0.66
0.64
0.64
0.61
0.65
0.62
0.55
0.67
0.55
Feature difference
0.87
0.6
0.72
0.81
0.74
0.76
0.67
0.47
0.68
0.73
0.73
0.55
0.69
0.67
0.51
0.72
0.99
Adhominem
0.84
0.81
0.96
0.93
0.77
0.91
0.93
0.62
0.9
0.91
0.91
0.8
0.88
0.92
0.72
0.93
1.0
Finetuned BERT
0,91
0.64
0.93
0.79
0.69
0.81
0.91
0.46
0.74
0.79
0.79
0.73
0.83
0.81
0.67
0.88
0.67
Overall score
Negram
0.84
0.65
0.79
0.69
0.7
0.69
0.69
0.6
0.66
0.62
0.62
0.61
0.67
0.56
0.6
0.7
0.61
PPM
0.87
0.67
0.8
0.71
0.71
0.74
0.74
0.6
0.7
0.66
0.66
0.64
0.68
0.65
0.59
0.7
0.58
Feature difference
0.89
0.6
0.74
0.83
0.75
0.79
0.69
0.48
0.71
0.76
0.76
0.58
0.73
0.67
0.55
0.74
0.99
Adhominem
0.79
0.78
0.82
0.9
0.72
0.89
0.9
0.58
0.84
0.87
0,87
0.75
0.85
0.87
0.69
0.89
0.99
Finetuned BERT
0.92
0.67
0.94
0.82
0.73
0.83
0.92
0.54
0.37
0.81
0.81
0.75
0.85
0.83
0.69
0.89
0.62
Table 17: AUC, F1, and overall score (avg of all scores) for different methods in AV task.


Reviews:
[{"review_id": "Pp45LISj2G", "reviewer": "Reviewer_bJGW", "summary": "", "questions": "Question A: I wonder about the attribution of the authorship of spoken texts. For some formats, like TV shows, the speech comes from a combination of the pre-written script (not necessarily written by the speaker) and the speaker's ad-lib. With this property, would a speaker's speaking style be consistent across instances? Could false positives be possible (same speaker, different scriptwriter)?\n\nQuestion B: Are the AA and AV experiments performed on the whole HANSEN benchmark datasets (i.e., the one including both human-spoken and A.I.) generated? If yes, do you label, for example, all ChatGPT-generated instances as a single class from a single entity? If yes, how do you ensure that generated texts from the same model retain the same persona or consistent speaking style?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reviews and well-thought-out questions. We briefly discuss our response in the following points.\n\n> It is unclear in the main paper how the quality assessment of LLM-generated data and curation of existing datasets is performed.\n\nWe agree with the reviewer that mentioning the curation and preprocessing steps of the existing datasets is essential since we are creating a benchmark. We briefly mention the procedures for creating new datasets in Appendix A. The source and brief details of the existing datasets are mentioned in Table 1. Due to the lack of space, we could not include this information and other preprocessing steps in the main paper. Given the extra page, we will discuss them in the revised main paper. \n\n> For example, one could question how to ensure these datasets exhibit different fundamental stylistic properties from written text ones like the PAN ones with lots of dialogue in fiction.\n\nRegarding the quality of the LLM-generated datasets, Figure 4 in Section 3.3 of the main paper highlights some quality scores that are widely used in evaluating the LLM-generated datasets [1,2,3]. We have also evaluated these datasets' Mauve score and readability score, which confirms the distribution similarity with the original human sources and the overall ease-to-read nature of spoken texts (similar found in Table 10 in the Appendix with human datasets). This will be included in the revised version/Appendix of the paper. Regarding the example of fiction, it can be written and spoken by nature [4]. The dialogue and conversations between characters are spoken language, while the description and other parts can be written. Therefore, comparing stylistic similarity between fiction and LLM-generated datasets in our paper might not be applicable. \n\n> One might also be skeptical of whether the LLM-generated dataset, despite the prompt, can be considered a spoken text and not just another written one.\n\nWe agree that ensuring the spoken nature of LLM-generated text is crucial for the benchmark's validity. Therefore, in the Appendix section of the paper, we have shown a comparison of LLM-generated spoken vs. written (Table 10) in parallel with the human spoken vs. written, and the differences align with the humans. To further validate, we have performed a small-scale study on a subset of the LLM-generated spoken ($L_s$) and corresponding written ($L_w$) for the TED dataset and consider another human spoken ($H_s$) and written ($H_w$) in different domains (to show the domain independence) and measure the stylistic difference (using the features in Table 10). We observe $dist(H_w, L_w)<dist(H_w, L_s)$ and $dist(H_s, L_s)<dist(H_s, L_w)$, which can provide an overall idea that our LLM-generated datasets contain more similarity with the spoken language rather than the written ones. \n\n> It is unclear whether the experiment runs are performed once or multiple times. One may question the reliability of observations and analyses, with the possibility of some models performing better than others with the effect of randomness.\n\nSince we are creating a benchmark, train, test, and validation split for each dataset are fixed for further research. The papers that specifically mention the number of experiments run primarily consider cross-validation. We have run the experiments for each setup five times for the AA and AV tasks and reported the average in our paper. In most cases, we get the exact results for the methods, except those that include random initializations, such as char-n-gram AV with grid search. However, the standard deviation of these results lies in the 0.0001-0.005 range, which will not impact the overall result comparison. We will, however, report the number of experiments run in the revised version.\n\nAlthough we observe minor fluctuations in the text property across different runs within a limited subset of our testing (e.g., z-score from DetectGPT or Perplexity from GPT-zero), it is important to note that the output, indicating the likelihood of AI/human origin, remains consistent across all our experimental scenarios. The table below presents a summary of our rerun on the TED subset for the TT methods (five run per sample). \n\n| Method           | Change of text property (average over all samples)    | Comment                                                                           |\n|------------------|-------------------------------------------------------|-----------------------------------------------------------------------------------|\n|$\\textbf{OpenAI detector}$   | N/A                                                   | As OpenAI pulled its AI Classifier, it is not possible to rerun the experiments.  |\n| $\\textbf{Roberta Large}$    | Score for each label changes on average 0.007±0.0053  | Class assignment does not change                                                  |\n| $\\textbf{DetectGPT }$       | Z-score changes on average 0.003±0.0071               | Class assignment does not change                                                  |\n| $\\textbf{GPT Zero}$         | Perplexity per line changes on average 1.05±0.087, Burstiness changes on average 3±4.27    | Class assignment does not change  \n\n> I wonder about the attribution of the authorship of spoken texts. For some formats, like TV shows, the speech comes from a combination of the pre-written script … \n\nWe already had the same concern while building the benchmark, so we did not include movie/TV series subtitles as spoken text datasets in HANSEN since the speech will not be from the actual actors but from the scriptwriters. We also mention this issue for the SEC (scripted) and MSU (simulated) datasets as characteristics (Table 1) and how it can affect the result (Discussion section: Line 391-394). The first paragraph of Section A in the Appendix pointed out the criteria we have considered to make representative spoken text datasets. However, we thank the reviewer for pointing out that it could be an exciting study for future works to distinguish ad-lib from scripted part of the speech for TV/movies/other media. \n\n> Are the AA and AV experiments performed on the whole HANSEN benchmark datasets ... \n\nThe AA experiment, involving human and AI-generated texts, contains top-7 human speakers and 3 LLMs (each LLM has been considered a speaker) simulating the 10-class AA problem [AA-10(L)] for each dataset. It was designed this way because we tried to simulate the scenario where LLM can perform as an individual speaker. However, we agree with the reviewer's point. Since texts from the same LLM are generated using a specific prompt (prompting them to symbolize the actual human persona), LLM is supposed to exhibit a different persona in all its instances. Therefore, we have also conducted an ablation study for the TED dataset with samples of the top 5 human speakers along with corresponding samples generated from the ChatGPT (considering five classes), Vicuna13B, and PaLM 2. So, it will be like a 20-class AA task (TED-20-w-LLM). The result comparison with the actual 20 class AA task (all human speakers, TED-20) is given in the following table (macro-f1 score). It demonstrates the substantial drop in performance, highlighting that LLMs can impersonate the actual human speaker. \n\n| Method       | TED-20  | TED-20-w-LLM  |\n|------------------|-------------|-------------------|\n| $\\textbf{Char n gram}$  | 0.92        | 0.37              |\n| $\\textbf{Stylometry }$ |  0.68       | 0.39              |\n| $\\textbf{FastText}$ |  0.67       | 0.26              |\n| $\\textbf{BertAA}$ |  0.56       | 0.46              |\n|$\\textbf{Bert-ft}$  |  0.84       | 0.51              |\n\n> …  might be helpful to introduce character n-gram … \n\nWe will update it in the revised version.\n\n> I don't think the full description of the dataset is necessary to include in the main text …\n\nWe believe that at least a dataset description and characteristics in a short format are necessary to get a hold of the primary results presented in the main paper rather than going through the appendix all the time. Otherwise, it would be difficult to understand some aspects of the discussion and results, such as poor AA & AV performance for talk-show typed datasets FTN and USP (lines 395-400 in the discussion section). Also, sample and token size is essential to report to understand the size of the datasets and corresponding results for different methods. Understanding the curation and generation procedure without providing the dataset characteristics will be difficult for the readers, in our opinion. \n\n> In Table 8, it would be great to use non-color coding for accessibility reasons. You could use diacritics, underlining, or italics.\n\nWe thank the reviewer for pointing out the issue. We will incorporate it in the revised version to ensure accessible reading. \n\n\n$\\textbf{References}:$\n\n1. Uchendu, A., Le, T., & Lee, D. (2023). Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective. ACM SIGKDD Explorations Newsletter, 25(1), 1-18.\n\n2. Pu, J., Sarwar, Z., Abdullah, S. M., Rehman, A., Kim, Y., Bhattacharya, P., ... & Viswanath, B. (2023, May). Deepfake text detection: Limitations and opportunities. In 2023 IEEE Symposium on Security and Privacy (SP) (pp. 1613-1630). IEEE.\n\n3. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1-38.\n\n4. Tannen, D. (1980, October). Spoken/written language and the oral/literate continuum. In Annual Meeting of the Berkeley Linguistics Society (Vol. 6, pp. 207-218).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The presented benchmark datasets introduce novel tasks of spoken text authorship analysis (AA, AV, and TT) on a large collection of diverse datasets in terms of speech contexts, formality levels, and topics.\n2. This paper clearly demonstrates via experiments how baseline and SOTA methods of written AA, AV, and TT task perform on spoken text tasks.\n3. The analysis section reveals that the spoken texts benchmark provides interesting research gaps for further research, such as that models trained on written texts do not perform well on spoken text test data, an observation worthy of further investigation.\n", "reasons_to_reject": "1.  It is unclear in the main paper how the quality assessment of LLM-generated data and curation of existing datasets is performed. While the details are included in the appendix and briefly introduced in the benchmark creation section (like in Figure 4), it is essential (in my opinion) to discuss the process clearly. For example, one could question how to ensure these datasets exhibit different fundamental stylistic properties from written text ones like the PAN ones with lots of dialogue in fiction. One might also be skeptical of whether the LLM-generated dataset, despite the prompt, can be considered a spoken text and not just another written one. Together, all these might affect the applicability and novelty of the presented benchmark and the credibility of the experimental results and analyses presented.\n\n2. It is unclear whether the experiment runs are performed once or multiple times. One may question the reliability of observations and analyses, with the possibility of some models performing better than others with the effect of randomness.\n", "questions_to_authors": "Question A: I wonder about the attribution of the authorship of spoken texts. For some formats, like TV shows, the speech comes from a combination of the pre-written script (not necessarily written by the speaker) and the speaker's ad-lib. With this property, would a speaker's speaking style be consistent across instances? Could false positives be possible (same speaker, different scriptwriter)?\n\nQuestion B: Are the AA and AV experiments performed on the whole HANSEN benchmark datasets (i.e., the one including both human-spoken and A.I.) generated? If yes, do you label, for example, all ChatGPT-generated instances as a single class from a single entity? If yes, how do you ensure that generated texts from the same model retain the same persona or consistent speaking style?\n", "paper_topic_and_main_contributions": "This paper addresses the current research gap in authorship attribution, authorship verification, and AI-generated text detection, being primarily written texts, despite potential applications for spoken text analyses. The authors present the following contributions.\n1. They curate existing speech corpora and new human-spoken datasets.\n2. They generated spoken text datasets from three popular LLMs and demonstrated the data quality assessment.\n3. They evaluate traditional authorship analysis methods on the resulting HANSEN benchmark datasets.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. This is just my opinion, but it might be helpful to introduce character n-gram (char n-gram) models before using char n-gram onwards. I also doubt the C should be capitalized.\n\n2. I don't think the full description of the dataset is necessary to include in the main text, especially when there are so many datasets that it took a whole valuable page and that one could make room for the data curation and generation process (in the weakness section).\n\n3. In Table 8, it would be great to use non-color coding for accessibility reasons. You could use diacritics, underlining, or italics.\n"}}, {"review_id": "xbNNm081Ly", "reviewer": "Reviewer_Ky6M", "summary": "", "questions": "- Question A: What can you do in the short term to mitigate the risks that you mentioned in the Ethics Statement section.\n- Question B: Do you have the rights to re-distribute all of these datasets? If so, under which license? Is there any Personal Identifiable Information (PII) in this dataset? If so, did you ask for consent from these people to redistribute their data?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting these critical aspects of the paper. We try to incorporate these issues in the following points.\n\n> The Ethics Statement section leaves a lot to be desired, the authors claim that their data can be used by ”malicious actors” in order to ”improve the quality LLM-generated spoken texts for harmful speech”, yet they use this exact same claim to justify the release of the data. I.e. the authors do not provide any convincing arguments about what they can do to mitigate the harm that their own benchmark might cause.\n\nThanks for pointing out this very important issue. What we really meant to state in the Ethics statement was the following. Like much research, our research can have both good and bad usage. While adversaries can use our dataset to harness the quality of AI-generated spoken texts for malicious usage, researchers can equally use our dataset to advance the science in generative AI for spoken texts and improve defensive solutions for good usage. More importantly, we believe the benefits of the latter outweigh those of the former, thus justifying our release of the dataset.\n\nAs to the second point on mitigation, which is a very valid critique, we accordingly changed our data release protocol as follows:\n1. Hansen will consist of three modules: (1) open-source data/existing datasets that we are free to re-distribute, (2) open-source data that we may not freely re-distribute but users have to download/scrape themselves, and (3) AI-generated data that we have generated.\n\n2. Hansen will be re-packaged such that (1) is included by default, (2) is partially included such that we only provide download-related info (e.g., URLs, line numbers, and Readme files), and scripts that we built for the download/scraping/preparing, but do not contain the data, (3) is not included by default and will be provided to users who provide PI’s contact information and sign a “good-usage” form.\n\nWe acknowledge that (3) cannot completely prevent Hansen from being used for malicious usage, but we hope to scare away those with bad intentions at least. That said, please understand that this issue of the “double-edged sword” usage of research works is not unique to Hansen but a general one for any research with a security flavor. We removed the existing anonymous release of Hansen in the Hugging Face and re-released it under the above protocol.\n\n> There is little to no discussion about the distribution rights that the authors hold to some of the curated datasets. For instance, there is no license information specially for the TED, CEO and FTN datasets. Yet the authors have already published this dataset in the Hugging Face Hub.\n\nThanks for this point which we neglected to consider. Upon examining our release, we found that: [https://www.ted.com/about/our-organization/our-policies-terms/ted-talks-usage-policy] TED talks adopted the Creative Commons(CC) license, and others (such as FTN, CEO) assure the use for non-commercial purposes but prohibit redistribution. As such, we will repackage our release into three modules, explained above, and re-release a new version where module (2) contains only the download info of the data that prohibits redistribution of the actual copies.\n\n>  What can you do in the short term to mitigate the risks that you mentioned in the Ethics Statement section.\n\nTo reduce the malicious usage of our dataset, instead of directly sharing our dataset with the general people in the Hugging Face, we will give access to those who (1) sign the “good-usage” form and (2) share the contact information of the PI. While this cannot completely prevent our dataset from malicious usage, we hope this will mitigate the short-term risks. We have removed the existing anonymous release of Hansen in the Hugging Face and will re-release it under the above protocol.\n\n> Do you have the rights to re-distribute all of these datasets? If so, under which license? \n\nWe neglect not to consider the redistribution aspect of our newly created datasets (FTN, CEO). While they are publicly available and allow personal and educational usage, they prohibit redistribution of the original data. We will fix the problem using the three module approaches we proposed above. \n\n> Is there any Personal Identifiable Information (PII) in this dataset? If so, did you ask for consent from these people to redistribute their data?\n\nAs mentioned above, we will repackage our data into three modules, where only modules (1) and (3) will have actual contents. In contrast, module (2), especially those we collected from web-scraping, such as FTN and CEO, has only download links (thus no data to include PIIs). We have checked the existence of PII using off-the-shelf automated tools, including (https://pypi.org/project/piianalyzer/0.1.0/) and (https://pypi.org/project/piicatcher/) for the module (1) and (3). While it found the full name, affiliation, and program/website contact information, it did not find any sensitive PII, such as identification numbers or personal contact information. Additionally, module (1) is openly available and accessible through other repositories; hence, it does not fall under our jurisdiction. Finally, module (3) is from AI-generated spoken text. Thus, data with PII appearance are from non-existent people or artificially generated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors curate and preprocess existing spoken datasets for this specific task, making them more accessible and usable.\n- The authors generate new samples of automatically generated samples of spoken text.\n- The authors provide extensive evaluation of existing AA methods on the proposed benchmark.\n- The Benchmark and the evaluations as a whole constitute a valuable resource for the community\n- The paper is clearly written.\n- The authors provide a very valuable literature review in the introduction and related work sections.", "reasons_to_reject": "- The Ethics Statement section leaves a lot to be desired, the authors claim that their data can be used by *”malicious actors”* in order to *”improve the quality LLM-generated spoken texts for harmful speech”*, yet they use this exact same claim to justify the release of the data. I.e. the authors do not provide any convincing arguments about what they can do to mitigate the harm that their own benchmark might cause.\n- There is little to no discussion about the distribution rights that the authors hold to some of the curated datasets. For instance, there is no license information specially for the TED, CEO and FTN datasets. Yet the authors have already published this dataset in the Hugging Face Hub.", "questions_to_authors": "- Question A: What can you do in the short term to mitigate the risks that you mentioned in the Ethics Statement section.\n- Question B: Do you have the rights to re-distribute all of these datasets? If so, under which license? Is there any Personal Identifiable Information (PII) in this dataset? If so, did you ask for consent from these people to redistribute their data?", "paper_topic_and_main_contributions": "The authors introduce a new Benchmark for authorship analysis in human and automatically generated *Spoken Text*. The main contributions are as follow:\n- The authors compile, pre-process and filter 17 existing human spoken datasets.\n- The authors generate more than 20K samples of spoken text with 3 different LLMs: ChatGPT, PaLM2 and Vicuna.\n- The authors extensively evaluate traditional authorship analysis methods on the benchmark resulting from the 2 previously mentioned contributions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GklmwCFNKb", "reviewer": "Reviewer_sdko", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper is well organized, with detailed sections describing the proposed approach.\n\nWe are delighted that the reviewer likes the organization of the paper. \n\n>One thing that needs to be clarified is whether using AI-generated datasets to construct a benchmark is necessary (e.g.: the need to ensure the AI-text data is similar to the human text,...)\n\nWe are unsure about the clarification the reviewer pointed out in this sentence. However, We aim to address this aspect comprehensively, approaching it from two distinct perspectives.\n\n> …  whether using AI-generated datasets to construct a benchmark is necessary\n\n We highlighted the significance of the research regarding AI-generated spoken text contents and their evaluation in the Introduction section of the paper, such as identifying synthetically generated scripts, detecting plagiarism, and solving potential copyright disputes (Lines 91-107). Therefore, creating and improving a benchmark with AI-generated spoken texts is necessary to initiate this research, especially in the age of AI and LLMs. \n\n> (e.g.: the need to ensure the AI-text data is similar to the human text,...)\n\nAlso, ensuring the quality of AI text and how similar it is to the human text is crucial to assuring the quality of the benchmark. Recall that our work is preemptively studying an issue where AI can generate \"spoken\" text, much in the same way that ChatGPT generates written texts with human quality (that causes many technical and societal problems at the moment). Therefore, to study the characteristics of AI-generated spoken texts, we need to first ensure they are similar to human-generated spoken texts. In fact, this year's EMNLP theme track is the Large Language Model and the future of the NLP. Therefore, we belive that this paper can provide substantial merit to such future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well organized, with detailed sections describing the proposed approach. Furthermore, the HANSEN benchmark proposed aims to contribute an effective way toward spoken text in terms of authorship analysis.", "reasons_to_reject": "One thing that needs to be clarified is whether using AI-generated datasets to construct a benchmark is necessary (e.g.: the need to ensure the AI-text data is similar to the human text,...)", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents the largest benchmark, HANSEN, which consists of a combination of human and AI-generated datasets for authorship analysis on spoken texts. Moreover, experiments with SOTA models are conducted to perform Authorship Attribution and Author Verification tasks on both human-spoken datasets and Human vs. AI spoken text detection tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the current research gap in authorship attribution, authorship verification, and AI-generated text detection, being primarily written texts, despite potential applications for spoken text analyses. The authors present the following contributions.

- They curate existing speech corpora and new human-spoken datasets.
- They generated spoken text datasets from three popular LLMs and demonstrated the data quality assessment.
- They evaluate traditional authorship analysis methods on the resulting HANSEN benchmark datasets.

Reasons To Accept:
- The presented benchmark datasets introduce novel tasks of spoken text authorship analysison a large collection of diverse datasets in terms of speech contexts, formality levels, and topics.
- This paper clearly demonstrates via experiments how baseline and SOTA methods perform on spoken text tasks.
- The analysis section reveals that the spoken texts benchmark provides interesting research gaps for further research, such as that models trained on written texts do not perform well on spoken text test data, an observation worthy of further investigation.
- The paper is clearly written.
- The authors provide a very valuable literature review in the introduction and related work sections.

Reasons To Reject: 
- It is unclear in the main paper how the quality assessment of LLM-generated data and curation of existing datasets is performed.
Decision: Accept-Findings