Question: Do all methods they experimented with outperform the random baseline in average rank?
Evidence: A valuable source of information in matching user accounts, one used in traditional stylometry tasks, is the way in which people use language. A speaker or writer's choice of words depends on many factors, including the rules of grammar, message content and stylistic considerations. There is a great variety of possible ways to compare the language patterns of two people. However, first we need a method for modelling the language of a given user. Below we explain how this is done.
Most statistical language models do not attempt to explicitly model the complete language generation process, but rather seek a compact model that adequately explains the observed linguistic data. Probabilistic models of language assign probabilities to word sequences $w_1$ . . . $w_\ell $ , and as such the likelihood of a corpus can be used to fit model parameters as well as characterize model performance.
N-gram language modelling BIBREF18 , BIBREF19 , BIBREF20 is an effective technique that treats words as samples drawn from a distribution conditioned on other words, usually the immediately preceding $n-1$ words, in order to capture strong local word dependencies. The probability of a sequence of $\ell $ words, written compactly as $w_1^\ell $ is $\Pr (w_1^\ell )$ and can be factored exactly as $\Pr (w_1^\ell ) = \Pr (w_1) \prod _{i=2}^\ell \Pr (w_i|w_1^{i-1})$ 
However, parameter estimation in this full model is intractable, as the number of possible word combinations grows exponentially with sequence length. N-gram models address this with the approximation $\tilde{\Pr }(w_i|w_{i-n+1}^{i-1}) \approx \Pr (w_i|w_1^{i-1})$ using only the preceding $n-1$ words for context. A bigram model ( $n=2$ ) uses the preceding word for context, while a unigram model ( $n=1$ ) does not use any context.
For this work, we used unigram models in Python, utilizing some components from NLTK BIBREF21 . Probability distributions were calculated using Witten-Bell smoothing BIBREF19 . Rather than assigning word $w_i$ the maximum likelihood probability estimate $p_i = \frac{c_i}{N}$ , where $c_i$ is the number of observations of word $w_i$ and $N$ is the total number of observed tokens, Witten-Bell smoothing discounts the probability of observed words to $p_i^* = \frac{c_i}{N+T}$ where $T$ is the total number of observed word types. The remaining $Z$ words in the vocabulary that are unobserved (i.e. where $c_i = 0$ ) are given by $p_i^* = \frac{T}{Z(N+T)}$ .
We experimented with two methods for measuring the similarity between n-gram language models. In particular, we tried approaches based on KL-divergence and perplexity BIBREF22 . We also tried two methods that do not rely on n-gram models, cosine similarity of TF-IDF vectors BIBREF23 , as well as our own novel method, called the confusion model.
The performance of each method is shown in Table 2 . Note that all methods outperform the random baseline in both accuracy and average rank by a great margin. Below we explain each of these metrics.
The first metric used for measuring the distance between the language of two user accounts is the Kullback-Leibler (KL) divergence BIBREF22 between the unigram probability distribution of the corpus corresponding to the two accounts. The KL-divergence provides an asymmetric measure of dissimilarity between two probability distribution functions $p$ and $q$ and is given by: $KL(p||q) = \int p(x)ln\frac{p(x)}{q(x)}$ 
We can modify the equation to prove a symmetric distance between distributions: $KL_{2}(p||q) = KL(p||q)+KL(q||p)$ 
For this method, the similarity metric is the perplexity BIBREF22 of the unigram language model generated from one account, $p$ and evaluated on another account, $q$ . Perplexity is given as: $PP(p,q) = 2^{H(p,q)}$ 
where $H(p,q)$ is the cross-entropy BIBREF22 between distributions of the two accounts $p$ and $q$ . More similar models lead to smaller perplexity. As with KL-divergence, we can make perplexity symmetric: $PP_{2}(p,q) = PP(p,q)+PP(q,p)$ 
This method outperformed the KL-divergence method in terms of average rank but not accuracy (see Table 2 ).
Perhaps the relatively low accuracies of perplexity and KL-divergence measures should not be too surprising. These measures are most sensitive to the variations in frequencies of most common words. For instance, in its most straightforward implementation, the KL-divergence measure would be highly sensitive to the frequency of the word “the". Although this problem might be mitigated by the removal of stop words and applying topic modelling to the texts, we believe that this issue is more nuanced than that.
Different social media (such as Twitter and Facebook) are used by people for different purposes, and thus Twitter and Facebook entries by the same person are likely to be thematically different. So it is likely that straightforward comparison of language models would be inefficient for this task.
One possible solution for this problem is to look at users' language models not in isolation, but in comparison to the languages models of everyone else. In other words, identify features of a particular language model that are characteristic to its corresponding user, and then use these features to estimate similarity between different accounts. This is a task that Term Frequency-Inverse Document Frequency, or TF-IDF, combined with cosine similarity, can manage.
TF-IDF is a method of converting text into numbers so that it can be represented meaningfully by a vector BIBREF23 . TF-IDF is the product of two statistics, TF or Term Frequency and IDF or Inverse Document Frequency. Term Frequency measures the number of times a term (word) occurs in a document. Since each document will be of different size, we need to normalize the document based on its size. We do this by dividing the Term Frequency by the total number of terms.
TF considers all terms as equally important, however, certain terms that occur too frequently should have little effect (for example, the term “the"). And conversely, terms that occur less in a document can be more relevant. Therefore, in order to weigh down the effects of the terms that occur too frequently and weigh up the effects of less frequently occurring terms, an Inverse Document Frequency factor is incorporated which diminishes the weight of terms that occur very frequently in the document set and increases the weight of terms that occur rarely. Generally speaking, the Inverse Document Frequency is a measure of how much information a word provides, that is, whether the term is common or rare across all documents.
Using TF-IDF, we derive a vector from the corpus of each account. We measure the similarity between two accounts using cosine similarity: $Similarity(d1,d2) = \frac{d1 \cdot d2}{||d1||\times ||d2||}$ 
Here, $d1 \cdot d2$ is the dot product of two documents, and $||d1||\times ||d2||$ is the product of the magnitude of the two documents. Using TD-IDF and cosine similarity, we achieved significantly better results than the last two methods, with an accuracy of $0.21$ and average rank of 999.
TF-IDF can be thought of as a heuristic measure of the extent to which different words are characteristic of a user. We came up with a new, theoretically motivated measure of “being characteristic" for words. We considered the following setup :
The whole corpus of the $11,224$ Twitter and Facebook accounts was treated as one long string;
For each token in the string, we know the user who produced it. Imagine that we removed this information and are now making a guess as to who the user was. This will give us a probability distribution over all users;
Now imagine that we are making a number of the following samples: randomly selecting a word from the string, taking the true user, $TU$ for this word and a guessed user, $GU$ from correspondent probability distribution. Intuitively, the more often a particular pair, $TU=U_{1}, GU=U_{2}$ appear together, the stronger is the similarity between $U_{1}$ and $U_{2}$ ;
We then use mutual information to measure the strength of association. In this case, it will be the mutual information BIBREF22 between random variables, $TU=U_{1}$ and $GU=U_{2}$ . This mutual information turns out to be proportional to the probabilities of $U_{1}$ and $U_{2}$ in the dataset, which is undesirable for a similarity measure. To correct for this, we divide it by the probabilities of $U_{1}$ and $U_{2}$ ;
We call this model the confusion model, as it evaluated the probability that $U_{1}$ will be confused for $U_{2}$ on the basis of a single word. The expression for the similarity value according to the model is $S\times log(S)$ , where $S$ is: $S=\sum _{w} p(w)p(U_{1}|w)p(U_{2}|w)$ 
Note that if $U_{1}=U_{2}$ , the words contributing most to the sum will be ordered by their “degree of being characteristic". The values, $p(w)$ and $p(u|w)$ have to be estimated from the corpus. To do that, we assumed that the corpus was produced using the following auxiliary model:
For each token, a user is selected from a set of users by multinomial distribution;
A word is selected from a multinomial distribution of words for this user to produce the token.
We used Dirichlet distributions BIBREF24 as priors over multinomials. This method outperforms all other methods with an accuracy of $0.27$ and average rank of 859.
We experimented with two methods for measuring the similarity between ngram language models. In particular, we tried approaches based on KL-divergence and perplexity [5]. We also tried two methods that do not rely on n-gram models, cosine similarity of TF-IDF vectors [17], as well as our own novel method, called the confusion model. The performance of each method is shown in Table 2. Note that all methods outperform the random baseline in both accuracy and average rank by a great margin. Below we explain each of these metrics.
Answer: Yes.