## 3 Research Methodology
### Applying Authorship Verification  
Given two human-written fanfiction pairs, and their corresponding two AI-generated versions (that were generated as described in the previous section), the authorship verification model could be applied across several document pair configurations as seen in Figure 3 and the average model scores were measured for each configuration:  
* **Human-Human (Same author)** Comparison between the human-written fanfiction pairs from the PAN dataset. The average score for this comparison should be high, indicating that the writing style between the two documents is similar. The average can be computed by \[\frac{1}{N}\sum_{i=1}^{N}av(\text{Human\_Doc}_{A}^{i},\text{Human\_Doc}_{B}^ {i})\] where \(av(A,B)\) represents the AV model score for documents \(A\) and \(B\), and \(N\) is the number of document pairs.
* **Human-AI** The comparison between the human-written fanfiction and the corresponding LLM-generated version. We take the average score between both document pair sets: \[\frac{1}{2N}\left(\sum_{i=1}^{N}av(\text{Human\_Doc}_{A}^{i},\text{AI\_Doc}_{A }^{i})+\sum_{i=1}^{N}av(\text{Human\_Doc}_{B}^{i},\text{AI\_Doc}_{B}^{i})\right)\]
* **AI-AI (Same Author)** Two LLM-generated texts from two fanfictions that were authored by the same person. The average score is given by \[\frac{1}{N}\sum_{i=1}^{N}av(\text{AI\_Doc}_{A}^{i},\text{AI\_Doc}_{B}^{i})\]
* **AI-AI (Different Authors)** Two AI-generated texts from prompts belonging to different authors. We measured this value by randomly pairing up LLM-generated fanfictions and averaging the model scores. Here, \(j\) represents the author index of another random author in the dataset. \[\frac{1}{N}\sum_{i=1}^{N}av(\text{AI\_Doc}_{A}^{i},\text{AI\_Doc}_{A}^{j})\]  
\begin{table}
\begin{tabular}{l l l c c c} \hline \hline
**LLM** & **Model** & **Generation Date** & **Token Limit** & **NUM\_PARAS** & **Avg. Tokens** \\ \hline GPT2 & gpt2-medium & 17 February 2024 & 1024 & 20 & 4811 \\ GPT3 & gpt-3.5-turbo-instruct & 11 February 2024 & 4096 & 10 & 4791 \\ ChatGPT & gpt-3.5-turbo & 15 February 2024 & 4096 & 20 & 4825 \\ LLaMA 2 & meta-llama/Llama-2-7b-chat-hf & 11 March 2024 & 4096 & 10 & 4950 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Summary of LLM settings.  
In this work, we opted to use the average model score as the primary metric. This follows our previous work [24], in which we used the model scores as a _stylometric similarity_ measure to compare writing style similarity between sections of the same document, across different documents from the same author, and across content from different social media platforms belonging to the same author. Unlike in AI-detector-style problem settings that use accuracy and F1-score as the primary evaluation metrics, in which clear positive (indicating AI-generated text) and negative (indicating human-written text) ground truth labels exist, since our approach uses a pair of documents to compare the language style, it is hard to clearly define positive and negative ground truth labels. This is because our problem formulation is not strictly a detector, but rather, a language style comparison. Therefore, we sought to model the writing style similarity as a similarity score rather than a binary label.