SlowGuess/NAACL_2022_OCR huggingface dataset:
To illustrate the amount of overlapping content in CoDA and DUTA, we compare each document with all other documents from the same corpus, and denote the maximum Jaccard distance as its maximum similarity. As shown in Figure 1, more than half of the documents in DUTA share almost completely overlapping content, whereas CoDA exhibits much lower similarities overall. Although DUTA consists of data from 10367 onion services which is larger than the number of onion services collected for use in CoDA, this shows that the data in CoDA is more uniquely varied and thus has higher information density.
4.2.4 Mask ID Distribution and TF-IDF
To gain insight into some of the lexical characteristics of each category in the Dark Web, we evaluate the mask ID distribution and TF-IDF (term frequency-inverse document frequency) for CoDA. The mask ID distribution is calculated by dividing the frequency of a particular mask ID (listed in Table 3) in a document by the number of all mask IDs in that document (we exclude ID_NUMBER in our data for this analysis as it accounts for the majority of all masks in every category). This is done for every document, and we take the average distribution by category. Similar methods are used for TF-IDF using scikit-learn (Pedregosa et al., 2011). We exclude English stopwords as defined in NLTK (Bird et al., 2009), but preserve the mask IDs to capture
$^{14}$ https://scikit-learn.org/stable/
Word frequency distribution is shown in Figure 4.
important mask IDs in each category.
We find that some mask IDs are particularly representative in some categories. For example, the drugs category has a high proportion of IDWeight, which is reasonable as webpages in drugs usually specify the weight of the drug in their listings. The TF-IDF measurements also show some interesting results<sup>16</sup>; for example, the majority of terms with the highest TF-IDF in the electronics category are related to Apple products (iPhones, iPads, MacBooks, etc.) which may suggest a high popularity of these products in the Dark Web.
5 Classification Experiments
We build several classifiers to investigate the performance of existing classification models on Dark Web text. Although deep neural network models are widely used today (Minaee et al., 2021), simple machine learning models such as SVM and naive Bayes (NB) have been reported to perform reasonably well on Dark Web texts, often outperforming deep models (Choshen et al., 2019). Therefore, we evaluate both types of models to see which is adequate for Dark Web text classification. We split CoDA into training and test sets (7:3 ratio) after stratified random shuffling with the same random seed for all experiments. The preprocessing method used for document similarity (Section 4.2.3) is applied here as it empirically works best across models.[17]
Multi-class SVM: We train a multi-class SVM classifier with TF-IDF features, and tune its hyperparameters by grid search. We build our classifier using TfidfVectorizer, LinearSVC, and GridSearchCV classes in scikit-learn.
CNN: Convolutional Neural Networks have been established as one of the popular choices for text classification for the ability to recognize position-invariant patterns such as text phrases (Minaee et al., 2021). Using PyTorch, we build a CNN model with a GloVe embedding layer (6B.300d), 2D convolution layers, and a fully-connected layer (Pennington et al., 2014).
BERT: To benefit from contextual representations and transfer learning, we use BERT (Devlin et al., 2019), a state-of-the-art language
Precision	Recall	F1-score
SVM	91.59	91.17	91.19
CNN	88.08	87.30	87.23
BERT	92.51	92.50	92.49
Table 7: Classifier performance on CoDA (weighted avg.). Boldface represents best performance.
model widely adopted across many NLP and machine learning tasks. We use the pretrained bert-base-uncased model in the PyTorch version of the HuggingFace library (Wolf et al., 2020) with a fully-connected classification layer on top of the [CLS] token.
Table 7 summarizes the performance of the three classifiers on CoDA. BERT exhibits the best results possibly due to its capability to model unknown words and utilize contextual information, despite the lexical differences of the Dark Web as shown in Section 4.2. SVM produces comparable results, suggesting that the relatively simple bag-of-words approach is still very effective at modeling topics of such domain-specific text. In contrast, CNN fares relatively worse, which is likely due to the specialized vocabulary of the Dark Web being poorly covered by the pretrained word embedding as seen in Choshen et al. (2019).
Table 8 shows the detailed results of classification using BERT. The classifier works relatively well for categories that exhibit a smaller specialized vocabulary such as arms, electronics, and gambling, whereas it performs worse for categories that cover diverse subtopics such as cryptocurrency and financial. We also observe that the classifier often confuses hacking with cryptocurrency or financial especially when documents contain phrases such as "hacked PayPal" or "hacked Bitcoin wallets", which are not categorized in the hacking category by our guidelines (the hacking category relates to hacking services and professional hacking techniques).
6 Use Cases
In this section, we elaborate on the use cases of our corpus and the classifiers trained on CoDA and DUTA.
(1) Synonym Inference: Dark Web users tend to use words differently from their original meaning to conceal or disguise their intents. For example, we observed that car company names (e.g., Tesla,
$^{16}$ Detailed measurements are provided in Appendix E.  
 $^{17}$ Detailed training configurations of each classifier are provided in Appendix B.
Category	Precision	Recall	F1-score
Arms / Weapons	96.70	97.78	97.24
Cryptocurrency	90.45	87.28	88.84
Drugs	93.90	92.29	93.08
Electronics	94.66	96.88	95.75
Financial	90.71	94.02	92.33
Gambling	99.15	98.31	98.72
Hacking	87.50	89.74	88.61
Pornography	94.20	94.46	94.33
Violence	93.15	93.79	93.47
Others	90.45	89.73	90.09
Weighted avg.	92.51	92.50	92.49
Toyota) are often used in drug-related documents in the Dark Web to refer to synthetic drugs with brand logos imprinted on each pill.
We test the above scenario by training two simple Word2vec models (Mikolov et al., 2013; Rehurek and Sojka, 2010), one using CoDA and another using DUTA. For each model, we query Tesla and Toyota and retrieve the most similar words to the queried terms. In this case, both models output drug-related words such as methoxphenidine, testosterone, and alprazolam. We also query another word, Wasabi, which originally refers to a plant but is also used to refer to a Bitcoin wallet service. In the Dark Web, Wasabi is more likely to be used as a cryptocurrency term rather than the plant itself. When Wasabi is queried, the model trained on CoDA returns cryptocurrency-related words, while the model trained on DUTA does not have the word in its vocabulary. We list the top 10 most similar words to Wasabi as reported by the model trained on CoDA, most of which are related to cryptocurrency services: mustard, electrum, samourai, pools, trustless, mycelium, rpc, xapo, hijacker, converter.
(2) Topic Classification: We assess the efficacy of CoDA and DUTA in classifying document categories in the Dark Web. For this scenario, we manually compile a list of 34 forum / marketplace websites on the Dark Web across three different topics: drugs, weapons, and finance, and create an extra benchmark dataset consisting of 2236 webpages from the list<sup>18</sup>. To remove possible overlap between the CoDA / DUTA corpora and the benchmark dataset, we exclude any documents crawled from the same URL as those from the benchmark dataset, or documents that mention any of the names from
Table 8: Per-category performance of BERT on CoDA
Model	Category	# pages	Precision	Recall	F1-score
CoDA	Drugs	936	99.87	83.44	90.92
Weapons	674	100.00	98.37	99.18
Finance	626	96.96	76.36	85.43
All	2236	99.09	85.96	91.87
DUTA	Drugs	936	99.46	78.74	87.90
Weapons	674	99.84	94.96	97.34
Finance	626	98.89	71.09	82.71
All	2236	99.42	81.48	89.29
Table 9: Classification performance of the BERT-based classifier on the benchmark dataset
the benchmark websites in their content. This excludes 246 and 220 documents from CoDA and DUTA, respectively. We then train a BERT-based classifier for each Dark Web corpus on the remaining documents with the same configuration used in the classification experiments, and evaluate them on the benchmark dataset.[19]
Table 9 shows the classification performance measured on the benchmark dataset, in which the CoDA-trained classifier consistently outperforms the DUTA-trained classifier. We conjecture that this is because CoDA contains less duplicate text with more diverse domain-specific words in the same number of documents, allowing the trained classifier to generalize better to unseen documents.
In this work, we introduced CoDA, a Dark Web text corpus collected from various onion services divided into topical categories. Using CoDA, we conducted a thorough analysis of the linguistic properties of the Dark Web and found that there are clear lexical differences from the Surface Web including abbreviations and lexical structure such as PoS distribution. We also found lexical characteristics of categories through mask ID distribution and TF-IDF.
Our text classification results showed that SVM and BERT perform well in the Dark Web domain, even with the language differences that the Dark Web exhibits compared to that of the Surface Web. Finally, we have demonstrated the practicality of CoDA through two use cases with NLP methods. We speculate that the lack of duplicate content in
<sup>19</sup>Since DUTA uses a different category set, we employ the following category mapping: counterfeit-credit-cards, counterfeit-money, and counterfeit-personal-identification to finance, and drugs and violence to drugs and weapons categories, respectively.
18Refer to Appendix F for the full list of website names and URLs.
CoDA compared to DUTA may aid in the performance of such applications.
We hope that our dataset and our work motivates further research in the field of language-based Dark Web analysis.
Masking Sensitive Information
Due to the nature of anonymous networks such as Tor, raw data posted on the Dark Web may contain private or illegal information. Such information includes (but is not limited to) Bitcoin addresses, credit card information, and social security numbers. Since CoDA was compiled by randomly selecting web documents from the Dark Web, the dataset may contain such information. Therefore, it is important that a public Dark Web dataset such as CoDA addresses ethical issues regarding sensitive information.
To prevent the use of CoDA for malicious purposes such as the extraction of sensitive information, we identify types of potentially sensitive data (such as email, IP, URL, crypto addresses, and social security numbers) which are subsequently masked (refer to Section 3.6 for the detailed description on mask identifiers used). These identifiers are matched using regular expressions and each page has been manually double-checked on whether such content has been properly masked by the authors. During this time, the authors did not find sensitive content outside of the masked information.
As mentioned in Section 4, some data analysis methods are conducted with unmasked versions of the Dark Web to prevent bias. However, we only use the unmasked version of CoDA for a fair analysis between the Dark and the Surface Web data, and do not utilize or disclose information found in the unmasked data in any way.
Handling Illegal Content
A significant portion of the Dark Web deals with explicit, pornographic content (violence, child pornography, torture, etc.). The act of accessing or viewing such media is illegal by law in many parts of the world. To prevent the access of such media, we collect crawled Dark Web pages in the form of HTML and parse HTML tags to retrieve only the text data. In addition, URL addresses and onion addresses that may link to such illegal media are also masked as previously mentioned (note that
all URL addresses and onion addresses have been masked, regardless of their content). Consequently, the authors and the annotators do not have access to media that are illegal by law.
It is worth noting that CoDA still contains texts of various activities that occur in the Dark Web, some of which are illegal in nature (drug trade, counterfeit products, etc.). However, the inclusion of text in the dataset that describes potentially illegal activities is not of ethical concern. Therefore, we do not censor text data that correspond to illegal activities.
Ethics on Annotation
Dark Web content often contains sensitive and illicit activities. Dealing with such content during the annotation process may be unsettling for some people, so we chose annotators who are experienced with the Dark Web and has given consent to being exposed to such content. The annotators recruited for classifying CoDA were specialists who work at a cyber threat data analytics & intelligence company specializing in Dark Web data. To ensure that the annotation process is fair, each of the ten annotators handled the same number of pages and were given equal compensations.
Preventative Measures to Discourage Non-Academic Use of CoDA
The content of CoDA includes text descriptions of various Dark Web activities. A potential harm of releasing this dataset is bringing increased attention to these activities. We strongly believe that our research should be used for scientific purposes only and discourage non-academic use of CoDA. We take a preventative approach by only permitting access to CoDA to researchers with research purposes that abide by the ACL Code of Ethics. The terms of use agreement can be found at https://s2w.inc/resources/coda.
This research was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921).
Mhd Wesam Al-Nabki, Eduardo Fidalgo, Enrique Alegre, and Laura Fernández-Robles. 2019. Torank:
Identifying the most influential suspicious domains in the tor network. Expert Systems with Applications, 123:212-226.
Mhd Wesam Al Nabki, Eduardo Fidalgo, Enrique Alegre, and Ivan de Paz. 2017. Classifying illegal activities on tor network based on web textual contents. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 35-43, Valencia, Spain. Association for Computational Linguistics.
Ron Artstein. 2017. Handbook of linguistic annotation. Springer, Dordrecht.
Georgia Avarikioti, Roman Brunner, Aggelos Kiayias, Roger Wattenhofer, and Dionysis Zindros. 2018. Structure and content of the visible darknet.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.
Alex Biryukov, Ivan Pustogarov, Fabrice Thill, and Ralf-Philipp Weinmann. 2014. Content and popularity analysis of tor hidden services. In 2014 IEEE 34th International Conference on Distributed Computing Systems Workshops (ICDCSW), pages 188-193.
Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. ConvoKit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 57-60, 1st virtual meeting. Association for Computational Linguistics.
Hsinchun Chen. 2011. Dark web: Exploring and data mining the dark side of the web, volume 30. Springer Science & Business Media.
Leshem Choshen, Dan Eldad, Daniel Hershcovich, Elior Sulem, and Omri Abend. 2019. The language of legal and illegal activity on the Darknet. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4271-4279, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Shalini Ghosh, Ariyam Das, Phil Porras, Vinod Yegneswaran, and Ashish Gehani. 2017. Automated categorization of onion sites for analyzing the darkweb
ecosystem. KDD '17, page 1793-1802, New York, NY, USA. Association for Computing Machinery.
Clement Guitton. 2013. A review of the available content on tor hidden services: The case against further development. Computers in Human Behavior, 29(6):2805-2815.
J. Tuomas Harviainen, Ari Haasio, and Lasse Hämäläinen. 2020. Drug traders on a local dark web marketplace. AcademicMindtrek '20, page 20-26, New York, NY, USA. Association for Computing Machinery.
Siyu He, Yongzhong He, and Mingzhe Li. 2019. Classification of illegal activities on the dark web. In Proceedings of the 2019 2nd International Conference on Information Science and Systems, ICISS 2019, page 73-78, New York, NY, USA. Association for Computing Machinery.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomás Mikolov. 2016a. Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomás Mikolov. 2016b. Bag of tricks for efficient text classification. CoRR, abs/1607.01759.
J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159-174.
Seunghyeon Lee, Changhoon Yoon, Heedo Kang, Yeonkeun Kim, Yongdae Kim, Dongsu Han, Sooel Son, and Seungwon Shin. 2019. Cybercriminal minds: An investigative study of cryptocurrency abuses in the dark web. In 26th Annual Network and Distributed System Security Symposium (NDSS 2019).
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Damon McCoy, Kevin Bauer, Dirk Grunwald, Tadayoshi Kohno, and Douglas Sicker. 2008. Shining light in dark places: Understanding the tor network. In Privacy Enhancing Technologies, pages 63-76, Berlin, Heidelberg. Springer Berlin Heidelberg.
Per Håkon Meland, Yara Fareed Fahmy Bayoumy, and Gutterm Sindre. 2020. The ransomware-as-a-service economy within the darknet. Computers & Security, 92:101762.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning-based text classification: A comprehensive review. ACM Comput. Surv., 54(3).
Daniel Moore and Thomas Rid. 2016. Cryptopolitik and the darknet. Survival, 58(1):7-38.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Steven T Piantadosi. 2014. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21(5):1112-1130.
Leonardo Ranaldi, Aria Nourbakhsh, Arianna Patrizi, Elena Sofia Ruzzetti, Dario Onorati, Francesca Fallucchi, and Fabio Massimo Zanzotto. 2022. The dark side of the language: Pre-trained transformers in the darknet.
Radim Rehurek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In Pro
ceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45-50.
Iskander Sanchez-Rola, Davide Balzarotti, and Igor Santos. 2017. The onions have eyes: A comprehensive structure and privacy analysis of tor hidden services. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, page 1251-1260, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Kyle Soska and Nicolas Christin. 2015. Measuring the longitudinal evolution of the online anonymous marketplace ecosystem. In 24th USENIX Security Symposium (USENIX Security 15), pages 33-48, Washington, D.C. USENIX Association.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Changhoon Yoon, Kwanwoo Kim, Yongdae Kim, Seungwon Shin, and Sooel Son. 2019. Doppelgangers on the dark web: A large-scale assessment on phishing hidden web services. In The World Wide Web Conference, WWW '19, page 2225-2235, New York, NY, USA. Association for Computing Machinery.
Hengrui Zhang and Futai Zou. 2020. A survey of the dark web and dark market research. In 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pages 1694-1705.
A Inter-Annotator Agreement
To obtain high-quality annotations, we ran a training session with the first 150 documents in three 50-document intervals. Each stage, we asked 10 annotators to annotate the same documents with the guidelines, and measured inter-annotator agreements using Fleiss' Kappa coefficient (Landis and Koch, 1977; Artstein, 2017). After each interval, we held a discussion session to resolve disagreements and revised the guidelines to accommodate feedback from the annotators. For each interval, the agreement coefficients were 0.67, 0.72, and 0.88. Note that the coefficients greater than 0.60 and 0.80 can be interpreted as "substantial" and "almost perfect" agreements, respectively (Landis and Koch, 1977). This suggests that the training sessions helped the annotators gradually reach a common consensus and familiarize themselves with the guidelines. We then assigned the remaining documents so that each document is assigned to a single annotator to speed up the annotation. The whole process took about three months, including one month of the training session.
The data analysis experiments were performed on a machine with Intel Xeon E5-2630 v4 CPU @ 2.2 GHz (with no GPU usage), and the classification experiments were performed on a machine with Intel Xeon Gold 6258R CPU @ 2.70 GHz and Nvidia GeForce RTX 3090.
SVM: To fine-tune the parameters, we exhaustively generated all the candidate combinations of the two parameter pairs: tolerance and regularization. From a grid of their values, we applied 10-fold cross validation and found that the model with tolerance of 0.1 and regularization of 1.0 work best when we used 'balanced_accuracy' as the scoring strategy. Since this model is a multi-class classifier, we used the OVR (one-versus-rest) multi-class strategy.