SlowGuess/NAACL_2021_OCR huggingface dataset:
While the ranking experiments in §4.2 were designed to measure the quality of the learned embedding, they do not directly measure task performance: moderation applications require decisions rather than rankings. To this end, we propose an account linking benchmark modeled after the problem of enforcing account bans, in which a fixed number of accounts are linked against novel accounts at test-time. Compared to the ranking experiments, the key difference is that we introduce a distinguished subset of authors from which we have accumulated a significant number of previously published documents to serve as queries. The procedure is illustrated in Figure 4b.
Because the subreddit feature serves as a proxy for topic, restricting to a single subreddit results in a more challenging problem by increasing the likelihood that the comments considered deal with similar topics. To this end, we repeat the following procedure for each of the five most popular subreddits. Each result of the experiment reported in Tables 2 and 9 is the average over the five subreddits of the corresponding results calculated using those subreddits individually.
Given a specified subreddit, we first randomly select 100 distinguished accounts, each publishing at least 100 posts to that subreddit in November 2016. The queries in the experiment consist of the 100 most recently published posts to the subreddit by each of the distinguished accounts in November 2016. In addition, the distinguished ac
counts must have published at least 16 posts to the subreddit between December 2016 and May 2017 to serve as the corresponding targets, as described below.
Next we randomly select 4900 accounts distinct from the distinguished accounts, each publishing at least 16 posts to the subreddit between December 2016 and May 2017. The targets in the experiment consist of the 4 most recently published posts to the subreddit by each of the 5000 accounts.
Performance metrics. For every query and target, each model considered returns a score, with smaller scores associated with a higher likelihood that the query and the target have the same author. For example, the proposed model returns the distance between their embeddings under the model. A decision rule to predict an author match is obtained by thresholding this score with respect to a chosen operating point. In production settings, one adjusts the operating point to obtain acceptable rates of false positives and false negatives. In our running application of ban enforcement, these types of errors correspond respectively with mistakenly banning an innocent user and failing to ban a new account of a banned user. Because the severity of these types of errors are different, we consider the detection cost function
$$
C _ {\det} = \pi C ^ {-} P ^ {-} + (1 - \pi) C ^ {+} P ^ {+}
$$
proposed by Van Leeuwen and Brümmer (2007), where  $P^{-}$  and  $P^{+}$  are empirical probabilities of false negatives and false positives,  $C^{-}$  and  $C^{+}$  are the costs of false negatives and false positives, and  $\pi$  is the a priori probability of a match. We take  $\pi = 0.05$  and we set  $C^{-} = 1$  and  $C^{+} = 2$ , reflecting our presumption that banning an innocent account is more severe than failing to recognize a banned user. Our choices of  $C^{-}$  and  $C^{+}$  are only meant to reflect the asymmetric nature of the problem, although in practice these costs would be highly platform-specific.
We report the minimum value of  $C_{\mathrm{det}}$  over all operating points (minDCF) and the value of  $P^{+}$  at the operating point for which  $P^{-} = P^{+}$ , also known as the equal error rate (EER).
Baseline models. We compare the proposed method with three baselines. First we consider TF-IDF vector representations of the concatenated text content of a sample, which are compared using cosine similarity. Next, we consider universal sentence encodings (Cer et al., 2018), which
Model	Training Length	EER	minDCF
TF-IDF	-	0.341	0.971
Universal	-	0.363	0.981
IUR	16	0.247	0.999
TP	1–8	0.169	0.848
TP	1–16	0.132	0.792
Table 2: Linking evaluation, averaged over 5 subreddits. The proposed model was trained using triplet loss. Smaller scores are better for both metrics.
are compared using angular distance. We experimented with two versions of this baseline, namely embedding the concatenation of the text content of the documents in a sample, and averaging the embeddings of the individual documents. Since we found the concatenated version to perform better, we only report on this variation. Finally, we consider IUR (Andrews and Bishop, 2019). Because this model only embeds samples of size 16, we pad samples containing fewer than 16 posts. To handle samples containing more than 16 posts, we organize the sample into contiguous groups of at most 16 posts, apply the embedding to each group, and average the embeddings.
Results. Table 2 compares the linking performance of the three baseline models along with two of variations of the proposed model arising from varying the sizes of the training samples. Note that both variations of the proposed model outperform the baselines. A further variation on this experiment is reported in Table 9 in which the queries are drawn from the training dataset, better reflecting the context of the motivating example.
Figure 6 shows the effect of the sizes of the targets on linking performance. Note that performance rapidly improves with larger samples, relative to the baselines. This trend is promising for our motivating application of ban enforcement, where it is desirable to recognize banned users as early as possible. Figure 7 shows receiver operator curves (ROC), which plot false positive rates against true positive rates as the operating points vary.
4.4 Embeddings of variable-sized samples
Our experiments in §4.3 show that linking samples from newly created accounts to those of distinguished authors is more successful when using as much historical data from the distinguished authors
Figure 6: Equal error rate (EER) as the length of the target samples varies. Results are averaged over 5 subreddits. Smaller is better for EER. IUR is omitted as it does not admit variable-sized samples.
Figure 7: Reciever operator curves of models evaluated on a single subreddit and averaged over the five most popular subreddits. A point on a ROC near the northwest corner corresponds with an operating point with low FPR and high TPR, so a model is better if its ROC stretches further to the northwest.
as possible. However, computational constraints typically inhibit embedding full account histories during training. Instead, in §4.3 we embed large samples of distinguished authors' histories using models trained on samples of sizes up to a maximum tractable length  $S$ . We take  $S = 16$  in our experiments as described in §2.3. Here, we examine the ability of a model trained on samples of sizes at most  $S$  to generalize to samples of sizes greater than  $S$ .
We also compare to a further baseline that averages single-post embeddings produced by a variation of the proposed model trained on single posts, which we denote by Avg. This is in contrast with the proposed model, which aggregates embeddings of multiple posts using an attention mechanism.
Table 3 shows the ranking performance of a num
ber of variations of the proposed model trained with triplet loss and all features (TPS) on samples of fixed or varying sizes as specified. These results demonstrate that a model trained on variable-sized samples appears to generalize well to much longer samples. We observe substantially better performance compared to the simple averaging baseline, and only a slight decrease in performance compared to the fixed length models as the evaluation sample size increases beyond the lengths seen at training time.
Train Size	Test Size	Model	MRR	R@4
1	8	Avg	0.205	0.243
1–8	8	Prop	0.349	0.411
8	8	Prop	0.358	0.420
1	12	Avg	0.282	0.330
1–8	12	Prop	0.497	0.568
12	12	Prop	0.516	0.589
1	16	Avg	0.347	0.397
1–8	16	Prop	0.603	0.672
16	16	Prop	0.632	0.700
4.5 Selecting the distribution of sizes
As mentioned in §2.3, we use Beta (3, 1) to select sample sizes during training. We hypothesize that a negatively skewed distribution tends to improve training efficiency by supplying longer samples most of the time, while retaining the ability to handle shorter samples. To evaluate this claim, we investigate several distributions of varying degrees of negative skew.
Table 4 shows the ranking performance of variations of the proposed model trained on samples of sizes varying between 1 and 16 posts and evaluated on samples of size 16. These models differ only in the distribution used to select sample sizes. Indeed, the negatively skewed distributions do improve ranking performance over the uniform distribution, although the choice of negatively skewed distribution appears to be mostly immaterial.
Table 3: Performance of the proposed model (Prop) and a simple averaging baseline (Avg), both trained and evaluated on fixed or variable-sized samples as shown.
Model	Skew	MRR	R@4
Unif (0,1)	0	0.604	0.673
Beta (2,1)	-0.566	0.626	0.692
tPois (16)	-0.786	0.632	0.703
Beta (3,1)	-0.861	0.633	0.701
Beta (4,1)	-1.049	0.633	0.704
Table 4: Ranking evaluations of models trained on samples of sizes selected with various distributions. tPois (16) is derived from the Poisson distribution with mean 16 by truncating its support to  $\{1,2,\ldots ,16\}$ .
This work motivates a number of interesting research questions. First, the proposed model makes use of publication times, but only avails of the hour of the day. It would be interesting to examine continuous-time variants of our encoder that incorporate relative time differences between actions when aggregating their embeddings, in light of the fact that patterns of user activity might be highly discriminative. For example, bots and spammers typically post at certain times of day and with particular frequencies. Separately, the proposed data augmentation methods we use to handle variable-sized samples may also be applicable in other settings, such as multi-document summarization (Liu and Lapata, 2019). Finally, the scores we use to determine author matches could be calibrated, providing confidence estimates associated with the account linking decisions.
To our knowledge, this work is the first to demonstrate the feasibility of a general-purpose account linking framework at web scale. Indeed, Figure 6 shows that performance improves as the size of the target increases, suggesting a speed-accuracy trade-off that can be tuned for different application settings. Expanding on an idea above, if confidence estimates were available, they could be used to inform the necessary sample sizes to achieve an acceptable level of risk.
Finally, we note that the generality of the proposed approach makes it potentially applicable to a wide range of applications, including source code attribution (Burrows et al., 2009; Yang et al., 2017; Kalgutkar et al., 2019), plagiarism detection (Pothast et al., 2010; Meuschke et al., 2018; Foltynek et al., 2019), and authorship attribution in collaborative documents (Flöck and Acosta, 2014; Dauber et al., 2017).
H. Alvari, S. Sarkar, and P. Shakarian. 2019. Detection of violent extremists in social media. In 2019 2nd International Conference on Data Intelligence and Security (ICDIS), pages 43-47.
Nicholas Andrews and Marcus Bishop. 2019. Learning invariant representations of social media users. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1684-1695, Hong Kong, China. Association for Computational Linguistics.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset.
David A Broniatowski, Amelia M Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen, Adrian Benton, Sandra C Quinn, and Mark Dredze. 2018. Weaponized health communication: Twitter bots and russian trolls amplify the vaccine debate. American journal of public health, 108(10):1378-1384.
Steven Burrows, Alexandra L Uitdenbogerd, and Andrew Turpin. 2009. Application of information retrieval techniques for source code authorship attribution. In International Conference on Database Systems for Advanced Applications, pages 699-713. Springer.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023.
Walter Daelemans, Mike Kestemont, Enrique Manjavacas, Martin Potthast, Francisco Rangel, Paolo Rosso, Gunther Specht, Efstathios Stamatos, Benno Stein, Michael Tschuggnall, Matti Wiegmann, and Eva Zangerle. 2019. Overview of PAN 2019: Author Profiling, Celebrity Profiling, Cross-domain Authorship Attribution and Style Change Detection. In 10th International Conference of the CLEF Association (CLEF 2019). Springer.
Edwin Dauber, Rebekah Overdorf, and Rachel Greenstadt. 2017. Stylometric authorship attribution of
collaborative documents. In International Conference on Cyber Security Cryptography and Machine Learning, pages 115-135. Springer.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
George R Doddington, Mark A Przybocki, Alvin F Martin, and Douglas A Reynolds. 2000. The nist speaker recognition evaluation-overview, methodology, systems, results, perspective. Speech communication, 31(2-3):225-254.
Wenfei Fan. 2012. Graph pattern matching revised for social network analysis. In Proceedings of the 15th International Conference on Database Theory, pages 8-21.
Fabian Flock and Maribel Acosta. 2014. Wikiwho: Precise and efficient attribution of authorship of revisioned content. In Proceedings of the 23rd international conference on World wide web, pages 843-854.
Tomáš Foltýnek, Norman Meuschke, and Bela Gipp. 2019. Academic plagiarism detection: a systematic literature review. ACM Computing Surveys (CSUR), 52(6):1-42.
Shunichi Ishihara. 2011. A forensic authorship classification in sms messages: A likelihood ratio based approach using n-gram. In Proceedings of the Australasian Language Technology Association Workshop 2011, pages 47-56.
Vaibhavi Kalgutkar, Ratinder Kaur, Hugo Gonzalez, Natalia Stakhanova, and Alina Matyukhina. 2019. Code authorship attribution: Methods and challenges. ACM Computing Surveys (CSUR), 52(1):1-36.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. 2020. Proxy anchor loss for deep metric learning.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Yang Liu and Mirella Lapata. 2019. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
Jing Lu, Chaofan Xu, Wei Zhang, Ling-Yu Duan, and Tao Mei. 2019. Sampling wisely: Deep image embedding by top-k precision optimization. In The IEEE International Conference on Computer Vision (ICCV).
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Leland McInnes, John Healy, and James Melville. 2020. Umap: Uniform manifold approximation and projection for dimension reduction.
Norman Meuschke, Vincent Stange, Moritz Schubotz, and Bela Gipp. 2018. Hyplag: a hybrid approach to academic plagiarism detection. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1321-1324.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017. Mixed precision training.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. 2020. A metric learning reality check. arXiv preprint arXiv:2003.08505.
Gordon Pennycook, Jonathon McPhettes, Yunhao Zhang, Jackson G Lu, and David G Rand. 2020. Fighting pandemic: A critical appraisal of social media. Experimental evidence for a scalable accuracy-nudge intervention. Psychological Science, 31(7):770-780.
Martin Potthast, Benno Stein, Alberto Barrón-Cedeno, and Paolo Rosso. 2010. An evaluation framework for plagiarism detection. In *Coling* 2010: Posters, pages 997-1005.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823.
Roy Schwartz, Oren Tsur, Ari Rappoport, and Moshe Koppel. 2013. Authorship attribution of micromessages. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1880-1891.
Giuseppe Silvestri, Jie Yang, Alessandro Bozzon, and Andrea Tagarelli. 2015. Linking accounts across social networks: the case of stackoverflow, github and twitter. In KDWeb, pages 41-52.
Estée Van Der Walt and Jan Eloff. 2018. Using machine learning to detect fake identities: Bots vs humans. IEEE Access, 6:6540-6549.
David A Van Leeuwen and Niko Brümmer. 2007. An introduction to application-independent evaluation of speaker recognition systems. In Speaker classification I, pages 330-353. Springer.
Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R. Scott. 2019. Cross-batch memory for embedding learning.
Xinyu Yang, Guoai Xu, Qi Li, Yanhui Guo, and Miao Zhang. 2017. Authorship attribution of source code by using back propagation neural network based on particle swarm optimization. PloS one, 12(11):e0187204.
A Hyperparameter selection
Top-  $k$  loss involves a number of hyperparameters. In addition to  $k$  and the margin penalty  $m$ , one must also select the number  $n_{+}$  of targets from each class presented in every batch. We found the values of  $k, m, n_{+}$  suggested by (Lu et al., 2019) to not perform well in our setting. We therefore conducted a small grid search for these values, selecting the optimal configuration based on validation scores. We considered  $k \in \{4,8\}$ ,  $n_{+} \in \{4,16\}$ , and  $m \in \{0.05,0.40\}$ , resulting in  $k = 4$ ,  $n_{+} = 8$ , and  $m = 0.25$ .
In experiments with the triplet loss, we use a fixed margin  $m = 0.2$ . We arrived at this value through the grid search illustrated in Table 5. We do not use dropout regularization, as previous work has shown dropout can degrade performance when training with larger datasets (Lan et al., 2019).
Margin	MRR	R@8
0.1	0.59	0.73
0.2	0.62	0.75
0.3	0.59	0.74
0.4	0.61	0.74
0.5	0.60	0.73
0.6	0.59	0.73
0.8	0.58	0.72
Mixed precision. We use mixed precision training (Micikevicius et al., 2017). This reduces the GPU memory consumed at training time by about half through the use of half precision floats, enables faster forward and backward pass computations, and allows for a larger batch size.
Multi-GPU training. Using 2 V100 GPUs to train the model significantly speeds up the process by quadrupling the effective batch size. Our models have an average training time of around 72 hours.
Simplified text encoding. We limit our convolutional text encoders to windows of 2, 3, and 4 subwords, excluding the largest window of 5 used by other models. Surprisingly, this did not impact ranking performance, which suggests that a small receptive field is sufficient for purposes of comparing authorship. As further support for this claim,
we also experimented with larger receptive fields than 5, which reduced ranking performance.
Model Hyper-parameters. Our reported models are trained with an embedding dimension of  $D = 1024$ , 512 convolutional filters, and an attention mechanism producing outputs of dimension 512. Additionally the subword and subreddit embedding dimensions are both  $N = 512$ .
Model Parameters Table 6 shows the numbers of parameters of the proposed model when using various combinations of text content (T), publication time (P), and subreddit (S).
Table 5: Validation ranking results for various triplet loss margins.
Model	Parameters
T	44.0M
TP	44.1M
TPS	45.5M
Table 6: Numbers of parameters in various trained models.
C Text encoding
We consider two methods to encode raw text content into integer arrays, namely taking the integer values of the corresponding UTF-8 encoded bytes directly, and using the SentencePiece unigram subword model (Kudo, 2018). SentencePiece tokenizes select character groupings according to a pretrained vocabulary of a specified size, which can be orders of magnitudes larger than the vocabulary of size  $2^{8}$  used by the byte encoding. For experiments conducted in  $\S 4$ , we used SentencePiece with a vocabulary size of  $2^{16}$ .
The potentially large disparity in vocabulary size between encoding methods can result in text encoded as integer arrays of significantly different lengths. In light of the need to truncate these arrays at training, we hypothesize that subword encoding with a large vocabulary results in better model performance as more textual information is captured after truncation.
To evaluate this claim, we fit additional SentencePiece subword models with vocabulary sizes  $2^{12}$  and  $2^{14}$  on text content from MUD. Table 7 provides ranking performance of variations of the proposed model trained with only the text feature on samples of fixed sized 16 and evaluated on samples of fixed sized 16. We remark that increases in vocabulary size do correspond with increases
Model	Size	MRR	R@4	R@8
SP	216	0.372	0.439	0.512
SP	214	0.352	0.412	0.483
SP	212	0.307	0.360	0.433
Byte	28	0.105	0.124	0.163
Table 7: Ranking performance of models using various subword vocabulary sizes. All models use Sentence-Piece (SP) or the byte encoding (Byte) described in Appendix C.
in ranking performance. We found that increasing the vocabulary size beyond  $2^{16}$  did not increase performance further.
D Further statistics about MUD
Table 8 shows some further details of the MUD dataset introduced in §4.1.
Number of users contributing	1,071,477
Number of posts	321,659,421
Mean post length	42.5 tokens
Mean number of posts contributed by a user	300.2
Mean number of subreddits accessed by a user	22.1
Mean number of months a user was active	9.9
Percentage of posts containing more than 64 tokens	17.37%
We repeat the linking experiment from §4.3 using queries that are observed at training time. We expect this to improve performance, something which is confirmed in Table 9.
Table 8: Some statistics of the Million User Dataset.
Model	Training Length	EER	minDCF
TF-IDF	-	0.362	0.967
Universal	-	0.371	0.989
IUR	16	0.223	0.951
TP	1-8	0.145	0.791
TP	1-16	0.136	0.765