Assessing the Applicability of Authorship Verification Methods

Abstract: Authorship verification (AV) is a research subject in the field of digital text forensics that concerns itself with the question, whether two documents have been written by the same person. During the past two decades, an increasing number of proposed AV approaches can be observed. However, a closer look at the respective studies reveals that the underlying characteristics of these methods are rarely addressed, which raises doubts regarding their applicability in real forensic settings. The objective of this paper is to fill this gap by proposing clear criteria and properties that aim to improve the characterization of existing and future AV approaches. Based on these properties, we conduct three experiments using 12 existing AV approaches, including the current state of the art. The examined methods were trained, optimized and evaluated on three self-compiled corpora, where each corpus focuses on a different aspect of applicability. Our results indicate that part of the methods are able to cope with very challenging verification cases such as 250 characters long informal chat conversations (72.7% accuracy) or cases in which two scientific documents were written at different times with an average difference of 15.6 years (>75% accuracy). However, we also identified that all involved methods are prone to cross-topic verification cases.

Digital text forensics aims at examining the originality and credibility of information in electronic documents and, in this regard, at extracting and analyzing information about the authors of the respective texts BIBREF0 . Among the most important tasks of this field are authorship attribution (AA) and authorship verification (AV), where the former deals with the problem of identifying the most likely author of a document INLINEFORM0 with unknown authorship, given a set of texts of candidate authors. AV, on the other hand, focuses on the question whether INLINEFORM1 was in fact written by a known author INLINEFORM2 , where only a set of reference texts INLINEFORM3 of this author is given. Both disciplines are strongly related to each other, as any AA problem can be broken down into a series of AV problems BIBREF1 . Breaking down an AA problem into multiple AV problems is especially important in such scenarios, where the presence of the true author of INLINEFORM4 in the candidate set cannot be guaranteed.
In the past two decades, researchers from different fields including linguistics, psychology, computer science and mathematics proposed numerous techniques and concepts that aim to solve the AV task. Probably due to the interdisciplinary nature of this research field, AV approaches were becoming more and more diverse, as can be seen in the respective literature. In 2013, for example, Veenman and Li BIBREF2 presented an AV method based on compression, which has its roots in the field of information theory. In 2015, Bagnall BIBREF3 introduced the first deep learning approach that makes use of language modeling, an important key concept in statistical natural language processing. In 2017, Castañeda and Calvo BIBREF4 proposed an AV method that applies a semantic space model through Latent Dirichlet Allocation, a generative statistical model used in information retrieval and computational linguistics.
Despite the increasing number of AV approaches, a closer look at the respective studies reveals that only minor attention is paid to their underlying characteristics such as reliability and robustness. These, however, must be taken into account before AV methods can be applied in real forensic settings. The objective of this paper is to fill this gap and to propose important properties and criteria that are not only intended to characterize AV methods, but also allow their assessment in a more systematic manner. By this, we hope to contribute to the further development of this young research field. Based on the proposed properties, we investigate the applicability of 12 existing AV approaches on three self-compiled corpora, where each corpus involves a specific challenge.
The rest of this paper is structured as follows. Section SECREF2 discusses the related work that served as an inspiration for our analysis. Section SECREF3 comprises the proposed criteria and properties to characterize AV methods. Section SECREF4 describes the methodology, consisting of the used corpora, examined AV methods, selected performance measures and experiments. Finally, Section SECREF5 concludes the work and outlines future work.
Over the years, researchers in the field of authorship analysis identified a number of challenges and limitations regarding existing studies and approaches. Azarbonyad et al. BIBREF8 , for example, focused on the questions if the writing styles of authors of short texts change over time and how this affects AA. To answer these questions, the authors proposed an AA approach based on time-aware language models that incorporate the temporal changes of the writing style of authors. In one of our experiments, we focus on a similar question, namely, whether it is possible to recognize the writing style of authors, despite of large time spans between their documents. However, there are several differences between our experiment and the study of Azarbonyad et al. First, the authors consider an AA task, where one anonymous document INLINEFORM0 has to be attributed to one of INLINEFORM1 possible candidate authors, while we focus on an AV task, where INLINEFORM2 is compared against one document INLINEFORM3 of a known author. Second, the authors focus on texts with informal language (emails and tweets) in their study, while in our experiment we consider documents written in a formal language (scientific works). Third, Azarbonyad et al. analyzed texts with a time span of four years, while in our experiment the average time span is 15.6 years. Fourth, in contrast to the approach of the authors, none of the 12 examined AV approaches in our experiment considers a special handling of temporal stylistic changes.
In recent years, the new research field author obfuscation (AO) evolved, which concerns itself with the task to fool AA or AV methods in a way that the true author cannot be correctly recognized anymore. To achieve this, AO approaches which, according to Gröndahl and Asokan BIBREF9 can be divided into manual, computer-assisted and automatic types, perform a variety of modifications on the texts. These include simple synonym replacements, rule-based substitutions or word order permutations. In 2016, Potthast et al. BIBREF10 presented the first large-scale evaluation of three AO approaches that aim to attack 44 AV methods, which were submitted to the PAN-AV competitions during 2013-2015 BIBREF11 , BIBREF5 , BIBREF12 . One of their findings was that even basic AO approaches have a significant impact on many AV methods. More precisely, the best-performing AO approach was able to flip on average INLINEFORM0 % of an authorship verifier’s decisions towards choosing N (“different author”), while in fact Y (“same author”) was correct BIBREF10 . In contrast to Potthast et al., we do not focus on AO to measure the robustness of AV methods. Instead, we investigate in one experiment the question how trained AV models behave, if the lengths of the questioned documents are getting shorter and shorter. To our best knowledge, this question has not been addressed in previous authorship verification studies.
Before we can assess the applicability of AV methods, it is important to understand their fundamental characteristics. Due to the increasing number of proposed AV approaches in the last two decades, the need arose to develop a systematization including the conception, implementation and evaluation of authorship verification methods. In regard to this, only a few attempts have been made so far. In 2004, for example, Koppel and Schler BIBREF13 described for the first time the connection between AV and unary classification, also known as one-class classification. In 2008, Stein et al. BIBREF14 compiled an overview of important algorithmic building blocks for AV where, among other things, they also formulated three AV problems as decision problems. In 2009, Stamatatos BIBREF15 coined the phrases profile- and instance-based approaches that initially were used in the field of AA, but later found their way also into AV. In 2013 and 2014, Stamatatos et al. BIBREF11 , BIBREF16 introduced the terms intrinsic- and extrinsic models that aim to further distinguish between AV methods. However, a closer look at previous attempts to characterize authorship verification approaches reveals a number of misunderstandings, for instance, when it comes to draw the borders between their underlying classification models. In the following subsections, we clarify these misunderstandings, where we redefine previous definitions and propose new properties that enable a better comparison between AV methods.
Reliability is a fundamental property any AV method must fulfill in order to be applicable in real-world forensic settings. However, since there is no consistent concept nor a uniform definition of the term “reliability” in the context of authorship verification according to the screened literature, we decided to reuse a definition from applied statistics, and adapt it carefully to AV.
In his standard reference book, Bollen BIBREF17 gives a clear description for this term: “Reliability is the consistency of measurement” and provides a simple example to illustrate its meaning: At time INLINEFORM0 we ask a large number of persons the same question Q and record their responses. Afterwards, we remove their memory of the dialogue. At time INLINEFORM1 we ask them again the same question Q and record their responses again. “The reliability is the consistency of the responses across individuals for the two time periods. To the extent that all individuals are consistent, the measure is reliable” BIBREF17 . This example deals with the consistency of the measured objects as a factor for the reliability of measurements. In the case of authorship verification, the analyzed objects are static data, and hence these cannot be a source of inconsistency. However, the measurement system itself can behave inconsistently and hence unreliable. This aspect can be described as intra-rater reliability.
Reliability in authorship verification is satisfied, if an AV method always generates the same prediction INLINEFORM0 for the same input INLINEFORM1 , or in other words, if the method behaves deterministically. Several AV approaches, including BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 , BIBREF16 fall into this category. In contrast, if an AV method behaves non-deterministically such that two different predictions for INLINEFORM2 are possible, the method can be rated as unreliable. Many AV approaches, including BIBREF4 , BIBREF13 , BIBREF26 , BIBREF1 , BIBREF27 , BIBREF3 , BIBREF28 , BIBREF29 , BIBREF30 belong to this category, since they involve randomness (e. g., weight initialization, feature subsampling, chunk generation or impostor selection), which might distort the evaluation, as every run on a test corpus very likely leads to different results. Under lab conditions, results of non-deterministic AV methods can (and should) be counteracted by averaging multiple runs. However, it remains highly questionable if such methods are generally applicable in realistic forensic cases, where the prediction INLINEFORM3 regarding a verification case INLINEFORM4 might sometimes result in Y and sometimes in N.
Another important property of an AV method is optimizability. We define an AV method as optimizable, if it is designed in such a way that it offers adjustable hyperparameters that can be tuned against a training/validation corpus, given an optimization method such as grid or random search. Hyperparameters might be, for instance, the selected distance/similarity function, the number of layers and neurons in a neural network or the choice of a kernel method. The majority of existing AV approaches in the literature (for example, BIBREF13 , BIBREF23 , BIBREF24 , BIBREF22 , BIBREF31 , BIBREF4 , BIBREF32 , BIBREF16 ) belong to this category. On the other hand, if a published AV approach involves hyperparameters that have been entirely fixed such that there is no further possibility to improve its performance from outside (without deviating from the definitions in the publication of the method), the method is considered to be non-optimizable. Non-optimizable AV methods are preferable in forensic settings as, here, the existence of a training/validation corpus is not always self-evident. Among the proposed AV approaches in the respective literature, we identified only a small fraction BIBREF21 , BIBREF2 , BIBREF30 that fall into this category.
From a machine learning point of view, authorship verification represents a unary classification problem BIBREF22 , BIBREF13 , BIBREF16 , BIBREF33 , BIBREF14 . Yet, in the literature, it can be observed that sometimes AV is treated as a unary BIBREF25 , BIBREF23 , BIBREF26 , BIBREF16 and sometimes as a binary classification task BIBREF30 , BIBREF32 , BIBREF22 , BIBREF2 . We define the way an AV approach is modeled by the phrase model category. However, before explaining this in more detail, we wish to recall what unary/one-class classification exactly represents. For this, we list the following verbatim quotes, which characterize one-class classification, as can be seen, almost identically (emphasis by us):
“In one-class classification it is assumed that only information of one of the classes, the target class, is available. This means that just example objects of the target class can be used and that no information about the other class of outlier objects is present.” BIBREF34 
“One-class classification (OCC) [...] consists in making a description of a target class of objects and in detecting whether a new object resembles this class or not. [...] The OCC model is developed using target class samples only.” BIBREF35 
“In one-class classification framework, an object is classified as belonging or not belonging to a target class, while only sample examples of objects from the target class are available during the training phase.” BIBREF25 
Note that in the context of authorship verification, target class refers to the known author INLINEFORM0 such that for a document INLINEFORM1 of an unknown author INLINEFORM2 the task is to verify whether INLINEFORM3 holds. One of the most important requirements of any existing AV method is a decision criterion, which aims to accept or reject a questioned authorship. A decision criterion can be expressed through a simple scalar threshold INLINEFORM4 or a more complex model INLINEFORM5 such as a hyperplane in a high-dimensional feature space. As a consequence of the above statements, the determination of INLINEFORM6 or INLINEFORM7 has to be performed solely on the basis of INLINEFORM8 , otherwise the AV method cannot be considered to be unary. However, our conducted literature research regarding existing AV approaches revealed that there are uncertainties how to precisely draw the borders between unary and binary AV methods (for instance, BIBREF36 , BIBREF16 , BIBREF33 ). Nonetheless, few attempts have been made to distinguish both categories from another perspective. Potha and Stamatatos BIBREF33 , for example, categorize AV methods as either intrinsic or extrinsic (emphasis by us):
“Intrinsic verification models view it [i. e., the verification task] as a one-class classification task and are based exclusively on analysing the similarity between [ INLINEFORM0 ] and [ INLINEFORM1 ]. [...] Such methods [...] do not require any external resources.” BIBREF33 
“On the other hand, extrinsic verification models attempt to transform the verification task to a pair classification task by considering external documents to be used as samples of the negative class.” BIBREF33 
While we agree with statement (2), the former statement (1) is unsatisfactory, as intrinsic verification models are not necessarily unary. For example, the AV approach GLAD proposed by Hürlimann et al. BIBREF22 directly contradicts statement (1). Here, the authors “decided to cast the problem as a binary classification task where class values are Y [ INLINEFORM0 ] and N [ INLINEFORM1 ]. [...] We do not introduce any negative examples by means of external documents, thus adhering to an intrinsic approach.” BIBREF22 .
A misconception similar to statement (1) can be observed in the paper of Jankowska et al. BIBREF24 , who introduced the so-called CNG approach claimed to be a one-class classification method. CNG is intrinsic in that way that it considers only INLINEFORM0 when deciding a problem INLINEFORM1 . However, the decision criterion, which is a threshold INLINEFORM2 , is determined on a set of verification problems, labeled either as Y or N. This incorporates “external resources” for defining the decision criterion, and it constitutes an implementation of binary classification between Y and N in analogy to the statement of Hürlimann et al. BIBREF22 mentioned above. Thus, CNG is in conflict with the unary definition mentioned above. In a subsequent paper BIBREF25 , however, Jankowska et al. refined their approach and introduced a modification, where INLINEFORM3 was determined solely on the basis of INLINEFORM4 . Thus, the modified approach can be considered as a true unary AV method, according to the quoted definitions for unary classification.
In 2004, Koppel and Schler BIBREF13 presented the Unmasking approach which, according to the authors, represents a unary AV method. However, if we take a closer look at the learning process of Unmasking, we can see that it is based on a binary SVM classifier that consumes feature vectors (derived from “degradation curves”) labeled as Y (“same author”) or N (“different author”). Unmasking, therefore, cannot be considered to be unary as the decision is not solely based on the documents within INLINEFORM0 , in analogy to the CNG approach of Jankowska et al. BIBREF24 discussed above. It should be highlighted again that the aforementioned three approaches are binary-intrinsic since their decision criteria INLINEFORM1 or INLINEFORM2 was determined on a set of problems labeled in a binary manner (Y and N) while after training, the verification is performed in an intrinsic manner, meaning that INLINEFORM3 and INLINEFORM4 are compared against INLINEFORM5 or INLINEFORM6 but not against documents within other verification problems (cf. Figure FIGREF15 ). A crucial aspect, which might have lead to misperceptions regarding the model category of these approaches in the past, is the fact that two different class domains are involved. On the one hand, there is the class domain of authors, where the task is to distinguish INLINEFORM7 and INLINEFORM8 . On the other hand, there is the elevated or lifted domain of verification problem classes, which are Y and N. The training phase of binary-intrinsic approaches is used for learning to distinguish these two classes, and the verification task can be understood as putting the verification problem as a whole into class Y or class N, whereby the class domain of authors fades from the spotlight (cf. Figure FIGREF15 ).
Besides unary and binary-intrinsic methods, there is a third category of approaches, namely binary-extrinsic AV approaches (for example, BIBREF3 , BIBREF30 , BIBREF29 , BIBREF37 , BIBREF32 , BIBREF1 , BIBREF2 ). These methods use external documents during a potentially existing training phase and – more importantly – during testing. In these approaches, the decision between INLINEFORM0 and INLINEFORM1 is put into the focus, where the external documents aim to construct the counter class INLINEFORM2 .
Based on the above observations, we conclude that the key requirement for judging the model category of an AV method depends solely on the aspect how its decision criterion INLINEFORM0 or INLINEFORM1 is determined (cf. Figure FIGREF15 ):
An AV method is unary if and only if its decision criterion INLINEFORM0 or INLINEFORM1 is determined solely on the basis of the target class INLINEFORM2 during testing. As a consequence, an AV method cannot be considered to be unary if documents not belonging to INLINEFORM3 are used to define INLINEFORM4 or INLINEFORM5 .
An AV method is binary-intrinsic if its decision criterion INLINEFORM0 or INLINEFORM1 is determined on a training corpus comprising verification problems labeled either as Y or N (in other words documents of several authors). However, once the training is completed, a binary-intrinsic method has no access to external documents anymore such that the decision regarding the authorship of INLINEFORM2 is made on the basis of the reference data of INLINEFORM3 as well as INLINEFORM4 or INLINEFORM5 .
An AV method is binary-extrinsic if its decision criterion INLINEFORM0 or INLINEFORM1 is determined during testing on the basis of external documents that represent the outlier class INLINEFORM2 .
Note that optimizable AV methods such as BIBREF18 , BIBREF25 are not excluded to be unary. Provided that INLINEFORM0 or INLINEFORM1 is not subject of the optimization procedure, the model category remains unary. The reason for this is obvious; Hyperparameters might influence the resulting performance of unary AV methods. The decision criterion itself, however, remains unchanged.
Each model category has its own implications regarding prerequisites, evaluability, and applicability.
One advantage of unary AV methods is that they do not require a specific document collection strategy to construct the counter class INLINEFORM0 , which reduces their complexity. On the downside, the choice of the underlying machine learning model of a unary AV approach is restricted to one-class classification algorithms or unsupervised learning techniques, given a suitable decision criterion.
However, a far more important implication of unary AV approaches concerns their performance assessment. Since unary classification (not necessarily AV) approaches depend on a fixed decision criterion INLINEFORM0 or INLINEFORM1 , performance measures such as the area under the ROC curve (AUC) are meaningless. Recall that ROC analysis is used for evaluating classifiers, where the decision threshold is not finally fixed. ROC analysis requires that the classifier generates scores, which are comparable across classification problem instances. The ROC curve and the area under this curve is then computed by considering all possible discrimination thresholds for these scores. While unary AV approaches might produce such scores, introducing a variable INLINEFORM2 would change the semantics of these approaches. Since unary AV approaches have a fixed decision criterion, they provide only a single point in the ROC space. To assess the performance of a unary AV method, it is, therefore, mandatory to consider the confusion matrix that leads to this point in the ROC space.
Another implication is that unary AV methods are necessarily instance-based and, thus, require a set INLINEFORM0 of multiple documents of the known author INLINEFORM1 . If only one reference document is available ( INLINEFORM2 ), this document must be artificially turned into multiple samples from the author. In general, unary classification methods need multiple samples from the target class since it is not possible to determine a relative closeness to that class based on only one sample.
On the plus side, binary-intrinsic or extrinsic AV methods benefit from the fact that we can choose among a variety of binary and INLINEFORM0 -ary classification models. However, if we consider designing a binary-intrinsic AV method, it should not be overlooked that the involved classifier will learn nothing about individual authors, but only similarities or differences that hold in general for Y and N verification problems BIBREF32 .
If, on the other hand, the choice falls on a binary-extrinsic method, a strategy has to be considered for collecting representative documents for the outlier class INLINEFORM0 . Several existing methods such as BIBREF32 , BIBREF1 , BIBREF2 rely on search engines for retrieving appropriate documents, but these search engines might refuse their service if a specified quota is exhausted. Additionally, the retrieved documents render these methods inherently non-deterministic. Moreover, such methods cause relatively high runtimes BIBREF11 , BIBREF5 . Using search engines also requires an active Internet connection, which might not be available or allowed in specific scenarios. But even if we can access the Internet to retrieve documents, there is no guarantee that the true author is not among them. With these points in mind, the applicability of binary-extrinsic methods in real-world cases, i. e., in real forensic settings, remains highly questionable.
In the following, we introduce our three self-compiled corpora, where each corpus represents a different challenge. Next, we describe which authorship verification approaches we considered for the experiments and classify each AV method according to the properties introduced in Section SECREF3 . Afterwards, we explain which performance measures were selected with respect to the conclusion made in Section UID17 . Finally, we describe our experiments, present the results and highlight a number of observations.
A serious challenge in the field of AV is the lack of publicly available (and suitable) corpora, which are required to train and evaluate AV methods. Among the few publicly available corpora are those that were released by the organizers of the well-known PAN-AV competitions BIBREF11 , BIBREF5 , BIBREF12 . In regard to our experiments, however, we cannot use these corpora, due to the absence of relevant meta-data such as the precise time spans where the documents have been written as well as the topic category of the texts. Therefore, we decided to compile our own corpora based on English documents, which we crawled from different publicly accessible sources. In what follows, we describe our three constructed corpora, which are listed together with their statistics in Table TABREF23 . Note that all corpora are balanced such that verification cases with matching (Y) and non-matching (N) authorships are evenly distributed.
As a first corpus, we compiled INLINEFORM0 that represents a collection of 80 excerpts from scientific works including papers, dissertations, book chapters and technical reports, which we have chosen from the well-known Digital Bibliography & Library Project (DBLP) platform. Overall, the documents were written by 40 researchers, where for each author INLINEFORM1 , there are exactly two documents. Given the 80 documents, we constructed for each author INLINEFORM2 two verification problems INLINEFORM3 (a Y-case) and INLINEFORM4 (an N-case). For INLINEFORM5 we set INLINEFORM6 's first document as INLINEFORM7 and the second document as INLINEFORM8 . For INLINEFORM9 we reuse INLINEFORM10 from INLINEFORM11 as the known document and selected a text from another (random) author as the unknown document. The result of this procedure is a set of 80 verification problems, which we split into a training and test set based on a 40/60% ratio. Where possible, we tried to restrict the content of each text to the abstract and conclusion of the original work. However, since in many cases these sections were too short, we also considered other parts of the original works such as introduction or discussion sections. To ensure that the extracted text portions are appropriate for the AV task, each original work was preprocessed manually. More precisely, we removed tables, formulas, citations, quotes and sentences that include non-language content such as mathematical constructs or specific names of researchers, systems or algorithms. The average time span between both documents of an author is 15.6 years. The minimum and maximum time span are 6 and 40 years, respectively. Besides the temporal aspect of INLINEFORM12 , another challenge of this corpus is the formal (scientific) language, where the usage of stylistic devices is more restricted, in contrast to other genres such as novels or poems.
As a second corpus, we compiled INLINEFORM0 , which represents a collection of 1,645 chat conversations of 550 sex offenders crawled from the Perverted-Justice portal. The chat conversations stem from a variety of sources including emails and instant messengers (e. g., MSN, AOL or Yahoo), where for each conversation, we ensured that only chat lines from the offender were extracted. We applied the same problem construction procedure as for the corpus INLINEFORM1 , which resulted in 1,100 verification problems that again were split into a training and test set given a 40/60% ratio. In contrast to the corpus INLINEFORM2 , we only performed slight preprocessing. Essentially, we removed user names, time-stamps, URLs, multiple blanks as well as annotations that were not part of the original conversations from all chat lines. Moreover, we did not normalize words (for example, shorten words such as “nooooo” to “no”) as we believe that these represent important style markers. Furthermore, we did not remove newlines between the chat lines, as the positions of specific words might play an important role regarding the individual's writing style.
As a third corpus, we compiled INLINEFORM0 , which is a collection of 200 aggregated postings crawled from the Reddit platform. Overall, the postings were written by 100 Reddit users and stem from a variety of subreddits. In order to construct the Y-cases, we selected exactly two postings from disjoint subreddits for each user such that both the known and unknown document INLINEFORM1 and INLINEFORM2 differ in their topic. Regarding the N-cases, we applied the opposite strategy such that INLINEFORM3 and INLINEFORM4 belong to the same topic. The rationale behind this is to figure out to which extent AV methods can be fooled in cases, where the topic matches but not the authorship and vice versa. Since for this specific corpus we have to control the topics of the documents, we did not perform the same procedure applied for INLINEFORM5 and INLINEFORM6 to construct the training and test sets. Instead, we used for the resulting 100 verification problems a 40/60% hold-out split, where both training and test set are entirely disjoint.
As a basis for our experiments, we reimplemented 12 existing AV approaches, which have shown their potentials in the previous PAN-AV competitions BIBREF11 , BIBREF12 as well as in a number of AV studies. The methods are listed in Table TABREF33 together with their classifications regarding the AV characteristics, which we proposed in Section SECREF3 .
All (optimizable) AV methods were tuned regarding their hyperparameters, according to the original procedure mentioned in the respective paper. However, in the case of the binary-extrinsic methods (GenIM, ImpGI and NNCD) we had to use an alternative impostors generation strategy in our reimplementations, due to technical problems. In the respective papers, the authors used search engine queries to generate the impostor documents, which are needed to model the counter class INLINEFORM0 . Regarding our reimplementations, we used the documents from the static corpora (similarly to the idea of Kocher and Savoy BIBREF30 ) to generate the impostors in the following manner: Let INLINEFORM1 denote a corpus with INLINEFORM2 verification problems. For each INLINEFORM3 we choose all unknown documents INLINEFORM4 in INLINEFORM5 with INLINEFORM6 and append them the impostor set INLINEFORM7 . Here, it should be highlighted that both GenIM and ImpGI consider the number of impostors as a hyperparameter such that the resulting impostor set is a subset of INLINEFORM8 . In contrast to this, NNCD considers all INLINEFORM9 as possible impostors. This fact plays an important role in the later experiments, where we compare the AV approaches to each other. Although our strategy is not flexible like using a search engine, it has one advantage that, here, it is assumed that the true author of an unknown document is not among the impostors, since in our corpora the user/author names are known beforehand.
According to our extensive literature research, numerous measures (e. g., Accuracy, F INLINEFORM0 , c@1, AUC, AUC@1, INLINEFORM1 or EER) have been used so far to assess the performance of AV methods. In regard to our experiments, we decided to use c@1 and AUC for several reasons. First, Accuracy, F INLINEFORM2 and INLINEFORM3 are not applicable in cases where AV methods leave verification problems unanswered, which concerns some of our examined AV approaches. Second, using AUC alone is meaningless for non-optimizable AV methods, as explained in Section UID17 . Third, both have been used in the PAN-AV competitions BIBREF5 , BIBREF12 . Note that we also list the confusion matrix outcomes.
Overall, we focus on three experiments, which are based on the corpora introduced in Section SECREF21 :
The Effect of Stylistic Variation Across Large Time Spans
The Effect of Topical Influence
The Effect of Limited Text Length
In the following each experiment is described in detail.
In this experiment, we seek to answer the question if the writing style of an author INLINEFORM0 can be recognized, given a large time span between two documents of INLINEFORM1 . The motivation behind this experiment is based on the statement of Olsson BIBREF38 that language acquisition is a continuous process, which is not only acquired, but also can be lost. Therefore, an important question that arises here is, if the writing style of a person remains “stable” across a large time span, given the fact that language in each individual's life is never “fixed” BIBREF38 . Regarding this experiment, we used the INLINEFORM2 corpus.
The results of the 12 examined AV methods are listed in Table TABREF41 , where it can be seen that the majority of the examined AV methods yield useful recognition results with a maximum value of 0.792 in terms of c@1. With the exception of the binary-intrinsic approach COAV, the remaining top performing methods belong to the binary-extrinsic category. This category of AV methods has also been superior in the PAN-AV competitions BIBREF11 , BIBREF5 , BIBREF12 , where they outperformed binary-intrinsic and unary approaches three times in a row (2013–2015).
The top performing approaches Caravel, COAV and NNCD deserve closer attention. All three are based on character-level language models that capture low-level features similar to character INLINEFORM0 -grams, which have been shown in numerous AA and AV studies (for instance, BIBREF39 , BIBREF26 ) to be highly effective and robust. In BIBREF19 , BIBREF28 , it has been shown that Caravel and COAV were also the two top-performing approaches, where in BIBREF19 they were evaluated on the PAN-2015 AV corpus BIBREF12 , while in BIBREF28 they were applied on texts obtained from Project Gutenberg. Although both approaches perform similarly, they differ in the way how the decision criterion INLINEFORM1 is determined. While COAV requires a training corpus to learn INLINEFORM2 , Caravel assumes that the given test corpus (which provides the impostors) is balanced. Given this assumption, Caravel first computes similarity scores for all verification problems in the corpus and then sets INLINEFORM3 to the median of all similarities (cf. Figure FIGREF49 ). Thus, from a machine learning perspective, there is some undue training on the test set. Moreover, the applicability of Caravel in realistic scenarios is questionable, as a forensic case is not part of a corpus where the Y/N-distribution is known beforehand.
Another interesting observation can be made regarding COAV, NNCD and OCCAV. Although all three differ regarding their model category, they use the same underlying compression algorithm (PPMd) that is responsible for generating the language model. While the former two approaches perform similarly well, OCCAV achieves a poor c@1 score ( INLINEFORM0 ). An obvious explanation for this is a wrongly calibrated threshold INLINEFORM1 , as can be seen from the confusion matrix, where almost all answers are N-predictions. Regarding the NNCD approach, one should consider that INLINEFORM2 is compared against INLINEFORM3 as well as INLINEFORM4 impostors within a corpus comprised of INLINEFORM5 verification problems. Therefore, a Y-result is correct with relatively high certainty (i. e., the method has high precision compared to other approaches with a similar c@1 score), as NNCD decided that author INLINEFORM6 fits best to INLINEFORM7 among INLINEFORM8 candidates. In contrast to Caravel, NNCD only retrieves the impostors from the given corpus, but it does not exploit background knowledge about the distribution of problems in the corpus.
Overall, the results indicate that it is possible to recognize writing styles across large time spans. To gain more insights regarding the question which features led to the correct predictions, we inspected the AVeer method. Although the method achieved only average results, it benefits from the fact that it can be interpreted easily, as it relies on a simple distance function, a fixed threshold INLINEFORM0 and predefined feature categories such as function words. Regarding the correctly recognized Y-cases, we noticed that conjunctive adverbs such as “hence”, “therefore” or “moreover” contributed mostly to AVeer's correct predictions. However, a more in-depth analysis is required in future work to figure out whether the decisions of the remaining methods are also primarily affected by these features.
In this experiment, we investigate the question if the writing style of authors can be recognized under the influence of topical bias. In real-world scenarios, the topic of the documents within a verification problem INLINEFORM0 is not always known beforehand, which can lead to a serious challenge regarding the recognition of the writing style. Imagine, for example, that INLINEFORM1 consists of a known and unknown document INLINEFORM2 and INLINEFORM3 that are written by the same author ( INLINEFORM4 ) while at the same time differ regarding their topic. In such a case, an AV method that it focusing “too much” on the topic (for example on specific nouns or phrases) will likely predict a different authorship ( INLINEFORM5 ). On the other hand, when INLINEFORM6 and INLINEFORM7 match regarding their topic, while being written by different authors, a topically biased AV method might erroneously predict INLINEFORM8 .
In the following we show to which extent these assumptions hold. As a data basis for this experiment, we used the INLINEFORM0 corpus introduced in Section UID30 . The results regarding the 12 AV methods are given in Table TABREF44 , where it can be seen that our assumptions hold. All examined AV methods (with no exception) are fooled by the topical bias in the corpus. Here, the highest achieved results in terms of c@1 and AUC are very close to random guessing. A closer look at the confusion matrix outcomes reveals that some methods, for example ImpGI and OCCAV, perform almost entirely inverse to each other, where the former predicts nothing but Y and the latter nothing but N (except 1 Y). Moreover, we can assume that the lower c@1 is, the stronger is the focus of the respective AV method on the topic of the documents. Overall, the results of this experiment suggest that none of the examined AV methods is robust against topical influence.
In our third experiment, we investigate the question how text lengths affect the results of the examined AV methods. The motivation behind this experiment is based on the observation of Stamatatos et al. BIBREF12 that text length is an important issue, which has not been thoroughly studied within authorship verification research. To address this issue, we make use of the INLINEFORM0 corpus introduced in Section UID28 . The corpus is suitable for this purpose, as it comprises a large number of verification problems, where more than 90% of all documents have sufficient text lengths ( INLINEFORM1 2,000 characters). This allows a stepwise truncation and by this to analyze the effect between the text lengths and the recognition results. However, before considering this, we first focus on the results (shown in Table TABREF46 ) after applying all 12 AV methods on the original test corpus.
As can be seen in Table TABREF46 , almost all approaches perform very well with c@1 scores up to 0.991. Although these results are quite impressive, it should be noted that a large fraction of the documents comprises thousands of words. Thus, the methods can learn precise representations based on a large variety of features, which in turn enable a good determination of (dis)similarities between known/unknown documents. To investigate this issue in more detail, we constructed four versions of the test corpus and equalized the unknown document lengths to 250, 500, 1000, and 2000 characters. Then, we applied the top performing AV methods with a c@1 value INLINEFORM0 on the four corpora. Here, we reused the same models and hyperparameters (including the decision criteria INLINEFORM1 and INLINEFORM2 ) that were determined on the training corpus. The intention behind this was to observe the robustness of the trained AV models, given the fact that during training they were confronted with longer documents.
The results are illustrated in Figure FIGREF47 , where it can be observed that GLAD yields the most stable results across the four corpora versions, where even for the corpus with the 250 characters long unknown documents, it achieves a c@1 score of 0.727. Surprisingly, Unmasking performs similarly well, despite of the fact that the method has been designed for longer texts i. e., book chunks of at least 500 words BIBREF13 . Sanderson and Guenter also point out that the Unmasking approach is less useful when dealing with relatively short texts BIBREF40 . However, our results show a different picture, at least for this corpus.
One explanation of the resilience of GLAD across the varying text lengths might be due to its decision model INLINEFORM0 (an SVM with a linear kernel) that withstands the absence of missing features caused by the truncation of the documents, in contrast to the distance-based approaches AVeer, NNCD and COAV, where the decision criterion INLINEFORM1 is reflected by a simple scalar. Table TABREF48 lists the confusion matrix outcomes of the six AV methods regarding the 250 characters version of INLINEFORM2 .
Here, it can be seen that the underlying SVM model of GLAD and Unmasking is able to regulate its Y/N-predictions, in contrast to the three distance-based methods, where the majority of predictions fall either on the Y- or on the N-side. To gain a better picture regarding the stability of the decision criteria INLINEFORM0 and INLINEFORM1 of the methods, we decided to take a closer look on the ROC curves (cf. Figure FIGREF49 ) generated by GLAD, Caravel and COAV for the four corpora versions, where a number of interesting observations can be made. When focusing on AUC, it turns out that all three methods perform very similar to each other, whereas big discrepancies between GLAD and COAV can be observed regarding c@1. When we consider the current and maximum achievable results (depicted by the circles and triangles, respectively) it becomes apparent that GLAD's model behaves stable, while the one of COAV becomes increasingly vulnerable the more the documents are shortened. When looking at the ROC curve of Caravel, it can be clearly seen that the actual and maximum achievable results are very close to each other. This is not surprising, due to the fact that Caravel's threshold always lies at the median point of the ROC curve, provided that the given corpus is balanced.
While inspecting the 250 characters long documents in more detail, we identified that they share similar vocabularies consisting of chat abbreviations such as “lol” (laughing out loud) or “k” (ok), smileys and specific obscene words. Therefore, we assume that the verification results of the examined methods are mainly caused by the similar vocabularies between the texts.
We highlighted the problem that underlying characteristics of authorship verification approaches have not been paid much attention in the past research and that these affect the applicability of the methods in real forensic settings. Then, we proposed several properties that enable a better characterization and by this a better comparison between AV methods. Among others, we explained that the performance measure AUC is meaningless in regard to unary or specific non-optimizable AV methods, which involve a fixed decision criterion (for example, NNCD). Additionally, we mentioned that determinism must be fulfilled such that an AV method can be rated as reliable. Moreover, we clarified a number of misunderstandings in previous research works and proposed three clear criteria that allow to classify the model category of an AV method, which in turn influences its design and the way how it should be evaluated. In regard to binary-extrinsic AV approaches, we explained which challenges exist and how they affect their applicability.
In an experimental setup, we applied 12 existing AV methods on three self-compiled corpora, where the intention behind each corpus was to focus on a different aspect of the methods applicability. Our findings regarding the examined approaches can be summarized as follows: Despite of the good performance of the five AV methods GenIM, ImpGI, Unmasking, Caravel and SPATIUM, none of them can be truly considered as reliable and therefore applicable in real forensic cases. The reason for this is not only the non-deterministic behavior of the methods but also their dependence (excepting Unmasking) on an impostor corpus. Here, it must be guaranteed that the true author is not among the candidates, but also that the impostor documents are suitable such that the AV task not inadvertently degenerates from style to topic classification. In particular, the applicability of the Caravel approach remains highly questionable, as it requires a corpus where the information regarding Y/N-distribution is known beforehand in order to set the threshold. In regard to the two examined unary AV approaches MOCC and OCCAV, we observed that these perform poorly on all three corpora in comparison to the binary-intrinsic and binary-extrinsic methods. Most likely, this is caused by the wrong threshold setting, as both tend to generate more N-predictions. From the remaining approaches, GLAD and COAV seem to be a good choice for realistic scenarios. However, the former has been shown to be more robust in regard to varying text lengths given a fixed model, while the latter requires a retraining of the model (note that both performed almost equal in terms of AUC). Our hypothesis, which we leave open for future work, is that AV methods relying on a complex model INLINEFORM0 are more robust than methods based on a scalar-threshold INLINEFORM1 . Lastly, we wish to underline that all examined approaches failed in the cross-topic experiment. One possibility to counteract this is to apply text distortion techniques (for instance, BIBREF41 ) in order to control the topic influence in the documents.
As one next step, we will compile additional and larger corpora to investigate the question whether the evaluation results of this paper hold more generally. Furthermore, we will address the important question how the results of AV methods can be interpreted in a more systematic manner, which will further influence the practicability of AV methods besides the proposed properties.
This work was supported by the German Federal Ministry of Education and Research (BMBF) under the project "DORIAN" (Scrutinise and thwart disinformation).

Figure and table captions:
Figure 1: The three possible model categories of authorship verification approaches. Here,U refers to the instance (for example, a document or a feature vector) of the unknown author. A is the target class (known author) and ¬A the outlier class (any other possible author). In the binary-intrinsic case, ρ denotes the verification problem (subject of classification), and Y and N denote the regions of the problem feature space where, according to a training corpus, the authorship holds or not.
Table 1: All training and testing corpora used in our experiments. Here, |C | denotes the number of verification problems in each corpus C and |DA | the number of the known documents. The average character length of the unknown document DU and the known document DA (concatenation of all known documents DA) is denoted by avg_len(DU ) and avg_len(DA), respectively.
Table 2: All 12 AVmethods, classified according to their properties.
Table 3: Evaluation results for the test corpus CDBLP in terms of c@1 and AUC. TP, FN, FP and TN represent the four confusion matrix outcomes, while UP denotes the number of unanswered verification problems. Note that AUC scores for the non-optimizable and unary AV methods are grayed out.
Table 4: Evaluation results for the test corpus CReddit.
Figure 2: Evaluation results for the four versions of the test corpus CPerv in terms of c@1.
Table 5: Evaluation results for the test corpus CPerv.
Table 6: Confusionmatrix outcomes for the 250 characters version of the test corpus CPerv.
Figure 3: ROC curves forGLAD,Caravel andCOAV (applied on the four corpora versions of CPerv). The circles and triangles depict the current and maximum achievable c@1 values on the corpus, respectively. Note that Caravel’s thresholds always lie along the EER-line.

Questions about this paper:
What are the 12 AV approaches which are examined?

Answers:
MOCC, OCCAV, COAV, AVeer, GLAD, DistAV, Unmasking, Caravel, GenIM, ImpGI, SPATIUM and NNCD