Authorship attribution and link ranking using a clustering/neural network sandwich

Faced with the PAN 2016’s problem of having an unknown number of authors writing an unknown number of documents that might but do not necessarily have the same topic, we came up with a pipeline of a K-Means clustering algorithm informing a neural network to establish document similarity, which then informs a Meanshift algorithm to output the final clusters (each cluster representing an author and enveloping one or more documents). The data provided by PAN consists of 18 problems that each consist of 50-100 documents from various authors. Every problem has either news articles or reviews, and is written in English, Dutch or Greek. Document length differs from 130 to 1000 words. This is not enough training data for a neural network, so character based features were created for training. Also, the task itself limited feature selection to character-only as well; if words or word n-grams would have been used, the initial assumption was that the system would be tricked easily into clustering on topic instead of author. As all documents within a problem have one genre, it is not unimaginable that documents from different authors have the same topic, and are therefore grouped together. Preventing the system from topic clustering and aiming it at author clustering was one of the biggest challenges in this task. The most promising feature for this task were character based skipgrams (Mikolov, Chen, Corrado, & Dean, 2013), pairing every character with either a neighboring one (positive sample) or one further away (negative sample). The embeddings thus created informed the neural network with underlying information regarding character sequences and structures. Although the task is to cluster all documents of a single author, the chosen approach was to train the neural network with every possible document pair, like the PAN 2015 task for author identification. The baseline thus created was very high (94%), resulting in a default decision to most frequent class (negative) for all samples. The K-Means clusterer was added to see if the number of pairs could be cut back, as a set of 50 documents already results in over 1200 possible pairs. KMeans was run with an iterative setting of [1:n-1] clusters, after which the total of document clusters was counted. If two documents were never clustered together, they were stripped from the input for the neural network. This lead to a 50% reduction in document pairs, but also a 25% reduction of correct pairs. Although this improved the baseline slightly, the data was still to biased for the neural network to be able to detect correct pairs in the training data, creating useless output of the network, which gave the Meanshift no additional features to work with. During development and tweaking of the neural network, the initial deadline for the shared task expired without submitting a working system. When the overview paper was published, the used method (on development set) could still be compared to the test set results of the other teams. Based on only character based preprocessed data, as this lead to the best result in the PAN task of 2015 (Stamatatos et al.), Meanshift output had a precision of 0.12 on average. Using a Scikit-learn Countvectorizer for preprocessing the data with both word Ngram counts and normal word counts, the BCubed precision was 0.256, with a BCubed F-score of 0.376. This measure was also applied by the task committee and although the system was not submitted, results of the experiment on the development set could be compared with other participants’s results on the test set afterwards (Stamatatos et al., 2016). Of course, this comparison must be interpreted with the difference between the two sets in mind. The second part of the task, the ranking of links within clusters resulted in a mean average precision of 0.014.