## 3 Research Methodology
### Authorship Verification Model  
The [PERSON] et al. authorship verification approach has been used before in PAN authorship verification-shared tasks [22; 48], a PAN writing style change detection task [23], to link Reddit accounts with similar writing styles and to identify user accounts across Twitter and Reddit that are maintained by the same person [24]. This approach was shown to work well on both large (3.5 K tokens) and small (1.2 K tokens) and to be largely topic-agnostic (the model is able to identify similarities in writing style and not the similarities in content).  
This model was selected primarily due to its versatility (applicable in other non-fanfiction domains, such as social media), its simplicity, and the interpretability of the results.  
The AV approach works as follows: Given two documents \(D_{A}\) and \(D_{B}\), stylometry features \(X_{A}\) and \(X_{B}\) are extracted from each document. Then, the difference between the features are computed as \(X_{diff}=|X_{A}-X_{B}|\). The different features are then fed into a logistic regression model to predict if the two documents were written by the same author. The feature vectors \(X_{A}\) and \(X_{B}\) are scaled so that they have a zero-mean and a unit-variance, and the difference vector \(X_{diff}\) is scaled again before it is passed to the classifier.  
The set of features used in this AV model are fairly common features used in previous stylometry work [49; 50]. They include TF-IDF values of character tri-grams, POS tag tri-grams, punctuation marks, function words, and the ratio between all POS tag pairs. Features that encode the syntactic structure of sentences include TF-IDF values of POS tag chunk tri-grams. To capture stylistic information about word order while also preventing topic-related biases, this model creates a \"masked\" version of the text where all words except function words are replaced with their part-of-speech tags. Then, the TF-IDF values the tri-grams from this modified text. The model also includes features that encode the fraction of commonly misspelled English words, common typos, common errors with determiners, British spelling of words, popular online abbreviations, vocabulary richness, and word length. The goal of adding the features capturing misspelled words was to add signals about consistent typos that were made by the authors. Similarly, the British English spellings were added to account for different spelling conventions and identify British authors and American authors. However, these features were not among the top features in the previous applications of this model. The most influential features were the character tri-gram features, part of speech tag features, and the \"masked\" function word (stopword) features [22; 24].  
For this work, the existing AV model trained on the PAN 2020/2021 large fanfiction dataset (downloaded from [[https://zenodo.org/records/5106099](https://zenodo.org/records/5106099)]([https://zenodo.org/records/5106099](https://zenodo.org/records/5106099)), accessed on 19 January 2025) was used. This model will be referred to as the AV model. This AV model was trained on a dataset compiled by [PERSON] [25]. The dataset contains English documents collected from fanfiction.net. Each record in the dataset consists of two documents which may or may not be written by the same person. It also specifies the \"random\" that each fanfiction belongs to. Examples of fandoms include Harry Potter, Star Wars, and Supernatural (TV show). The ground truth specifies the author identifiers and the label indicating if the two documents were written by the same person. The dataset is a balanced dataset with 275,486 document pairs in which 54% of the records are same-author document pairs and the rest are different-author document pairs. It is important to note that (to the best of our knowledge) that these fanfictions are written by humans and are not AI-generated.