SlowGuess/NAACL_2022_OCR huggingface dataset:
Youngjin Jin<sup>1</sup> Eugene Jang<sup>2</sup> Yongjae Lee<sup>2</sup> Seungwon Shin<sup>1</sup> Jin-Woo Chung<sup>2*</sup>
$^{1}$ KAIST, Daejeon, South Korea
$^{2}$ S2W Inc., Seongnam, South Korea
1{ijinjin,claude}@kaist.ac.kr
2{genesith,lee,jwchung}@s2w.inc
The hidden nature and the limited accessibility of the Dark Web, combined with the lack of public datasets in this domain, make it difficult to study its inherent characteristics such as linguistic properties. Previous works on text classification of Dark Web domain have suggested that the use of deep neural models may be ineffective, potentially due to the linguistic differences between the Dark and Surface Webs. However, not much work has been done to uncover the linguistic characteristics of the Dark Web. This paper introduces CoDA, a publicly available Dark Web dataset consisting of 10000 web documents tailored towards text-based Dark Web analysis. By leveraging CoDA, we conduct a thorough linguistic analysis of the Dark Web and examine the textual differences between the Dark Web and the Surface Web. We also assess the performance of various methods of Dark Web page classification. Finally, we compare CoDA with an existing public Dark Web dataset and evaluate their suitability for various use cases.
The World Wide Web contains a vast, non-indexed part of the Internet (known as the Deep Web) which is hidden from traditional web search engines. The Dark Web, which refers to the small portion of the non-indexed pages that require specific routing protocols such as Tor<sup>1</sup> for access, has become a safe haven for users wanting to conceal their identity and preserve their anonymity.
A consequence of the properties of the Dark Web (limited methods of access and the volatility of its onion services) is that it is difficult to grasp the general topology and the overall content of the Dark Web. A number of past academic studies have tried to unravel the Dark Web through methods such as page classification (Al Nabki et al., 2017;
Ghosh et al., 2017; He et al., 2019; Choshen et al., 2019) and content analysis (Biryukov et al., 2014; Avarikioti et al., 2018). However, not much work has been done on the linguistic analysis of the Dark Web (Choshen et al., 2019).
In addition, the Dark Web has been studied and analyzed in the security research community to uncover malicious activities including phishing (Yoon et al., 2019), illicit online marketplace activity (Soska and Christin, 2015), terrorism (Chen, 2011), cryptocurrency abuse (Lee et al., 2019), and ransomware ecosystems (Meland et al., 2020). We believe that the lack of a comprehensive work on the language of the Dark Web from the NLP community mainly stems from the lack of Dark Web datasets publicly available for research. Therefore, a new dataset on the Dark Web may prove to be very useful not only for the NLP community, but also for other research communities devoted to cybersecurity and cybercrime investigation through methods such as page classification and malicious activity detection.
To the best our knowledge, the only currently publicly available Dark Web dataset is DUTA (Al Nabki et al., 2017), which has been extended to hold over ten-thousand unique onion addresses as DUTA-10K (Al-Nabki et al., 2019). DUTA $^2$  has become a baseline dataset for many Dark Web related works such as Choshen et al. (2019), which investigates the characteristics of language used in various illegal and legal onion services.
Nevertheless, DUTA has its shortcomings. For example, the category distribution of DUTA is highly skewed, with some categories such as human trafficking accounting for only 3 out of 10367 total onion services. In addition, the use of DUTA as a means of language analysis may not be ideal as it contains many duplicate data, with only  $51\%$  of the texts being unique (Al-Nabki et al., 2019).
2We will refer to the DUTA-10K dataset from here on as DUTA, unless otherwise specified.
$^{1}$ Tor Project: https://www.torproject.org/
Human Language Technologies, pages 5621 - 5637
To provide a better understanding of the Dark Web (and thus motivate more research on the Dark Web), we introduce CoDA<sup>3</sup> (Comprehensive Darkweb Annotations), a text corpus of 10,000 web documents from the Dark Web (primarily in English) which have been manually classified according to their topic into ten categories. To ensure that the quality of the classification is not overlooked, we develop detailed annotation / tagging guidelines (Section 3) to guide our annotators. Using CoDA, we conduct a thorough text-based data analysis (Section 4) to uncover some of the linguistic properties of the Dark Web, and gain insight into differences in how language is used in the Surface Web and the Dark Web. We build several text classifier models and train them using CoDA, and verify which classification methods perform particularly well with the Dark Web (Section 5). Finally, to evaluate the use of CoDA compared to DUTA, we introduce use cases and compare the performances of classifiers trained on each dataset (Section 6).
The Dark Web is commonly crawled using Tor, which relies on onion routing to enable encrypted communications over a computer network (McCoy et al., 2008). Several works use Dark Web search engines such as Ahmia and web directories such as The Hidden Wiki to recursively search for content on the Dark Web (Guitton, 2013; Al Nabki et al., 2017; He et al., 2019). This method of crawling works surprisingly well as the visible part of the Dark Web is suggested to be well-connected via hyperlinks (Sanchez-Rola et al., 2017; Avarikioti et al., 2018).
To facilitate the research on Dark Web content analysis, a text-based, manually labeled dataset collected from the active domains in the Tor network called DUTA was made publicly available by Al Nabki et al. (2017). In a subsequent work, the original DUTA dataset was extended to 10367 unique domains with minor changes to the labeling procedure (Al-Nabki et al., 2019). To the best of our knowledge, DUTA is the first and only publicly available Dark Web text dataset.
Past works have analyzed the Dark Web through topical classification of texts in onion services. Many have approached text-based page classifica
tion with machine learning methods such as SVM (Support Vector Machine), NB (Naïve Bayes), and LR (Logistic Regression) (Moore and Rid, 2016; Al Nabki et al., 2017; Ghosh et al., 2017; Avarikioti et al., 2018; He et al., 2019) using various information retrieval weighting schemes like TF-IDF (Term Frequency-Inverse Document Frequency) and BOW (Bag-of-Words) (Al Nabki et al., 2017; Ghosh et al., 2017; Choshen et al., 2019; He et al., 2019).
Choshen et al. (2019) have suggested that deep neural models may fare poorly with Dark Web classification as language in the Dark Web is lexically and syntactically different compared to that of the Surface Web. Their work demonstrated that representation methods such as GloVe (Pennington et al., 2014) and contextualized pre-trained language representations such as ELMo (Peters et al., 2018) resulted in a subpar performance compared to traditional machine learning methods, suggesting that the small size of training data and the specialized vocabulary in the Dark Web domain may not be suitable with such methods. Nevertheless, transformer-based pre-trained language models like BERT (Devlin et al., 2019) showed promising results in text classification tasks, although it is not often the case that such models adapt with ease in the Dark Web domain (Ranaldi et al., 2022).
3 The CoDA Corpus
In this section, we introduce our categorization approach and the methods used to construct our Dark Web dataset, CoDA.
3.1 CoDA Category Set
CoDA is comprised of ten categories (described in detail in Section 3.5) as shown in Table 1. We arrive at our ten categories through an extended discussion with the dataset annotators (see Section 3.4) on a suitable method to categorize the various activities in the Dark Web and refer to the high-level taxonomy from Moore and Rid (2016). Unlike DUTA, we do not subdivide each category into legal and illegal (or normal and suspicious as labeled in DUTA) activities because in many cases it is difficult to clearly distinguish between categories using only the surface information available from websites. For example, while human annotators easily agreed that most 'counterfeit money' services are highly likely to be illegal, they found it difficult to determine the legality of 'bitcoin wallet'
$^{3}$ CoDA is available upon request at https://s2w.inc/resources/coda.  $^{4}$ https://ahmia.fi/
Category	Document Count	Ratio	Short guideline description
Pornography	1195	12.0%	general / child pornography and other explicit content
Drugs	1172	11.7%	various types of legal / illegal drugs such as medications, steroids, pain killers, viagra, cannabis, hashish, meth, benzos, ecstasies, opioids, and psychedelics
Financial	1003	10.0%	counterfeit / cloned / stolen money or identifications (e.g., bills, credit cards, certificates, passports), money transfers (e.g., PayPal), fiat money, ATM skimmers, magnetic card readers, etc.
Gambling	787	7.87%	any type of gambling, betting, casinos, lotteries, etc.
Cryptocurrency	763	7.63%	cryptocurrency-specific services or technologies such as wallets, generators, mining, laundering, mixing, multiplying, doubling, scamming, and escrow
Hacking	649	6.49%	hacking tools, hacking guides, hacking groups, hacking services, ransomware, malware, exploits, DDoS attacks, cracking, botnet, etc.
Arms / Weapons	599	5.99%	any type of non-lethal / lethal weapons such as guns, ammunition, explosives, knives, missiles, and chemical weapons
Violence	485	4.85%	human trafficking, hitman, kidnapping, poisoning, torture, extortion, sextortion, sex slavery, blackmail, etc.
Electronics	426	4.26%	sale of or information on (stolen / hacked) mobile phones, laptops, tablet computers, etc.
Others	2921	29.2%	all other content that does not fit the above categories, including log-in pages, error messages, etc.
Total	10000	100.0%
services. This also applies to the drugs category, as the sale of certain drugs may be illegal in some countries, but can be legal in others.
Moreover, unlike DUTA, we exclude non-topical categories such as forum and marketplace as they are orthogonal to topical categories; e.g., hacking forums, which frequently appear in the Dark Web, can be categorized as both forum and hacking. We argue that such categories are more relevant to the structure of webpages rather than topics, and thus need to be annotated independently of topical categories. We leave this for future work, and do not further split our ten categories into sub-categories. Nonetheless, our category set still covers a wide range of activities on the Dark Web.
Finally, we point out that about  $30\%$  of data is categorized into others. During data collection, we observed that many webpage documents contain various pages not related to the categorized activities (such as blogs, news sites, search engines, wiki pages, etc.). Since the content of such pages is not necessarily attributed to a specific activity, we categorize them into others.
We collected onion addresses from Ahmia and repositories of onion domain lists<sup>5</sup>. Starting from these seed addresses, we crawled the Dark Web and extracted unseen onion addresses from crawled webpage documents to gradually expand our onion address list.
The Dark Web contains large amounts of nearly
Table 1: Categories in CoDA with document count and a short description of annotation guidelines
Language	Document count	Language	Document count
English	8855	Portuguese	54
Russian	542	Chinese	38
German	129	Italian	28
French	100	Japanese	27
Spanish	61	Dutch	14
Table 2: Top 10 language distribution of documents in CoDA. The full language distribution statistics is given in Appendix C.
identical websites since no expense is required for maintenance due to free hosting services such as "Freedom Hosting" (now defunct) (Al-Nabki et al., 2019). These cloned website farms serve to provide stable services for illegal activities (Al-Nabki et al., 2019), or to attract users and deceive them into disclosing sensitive information (Yoon et al., 2019). In order to construct a quality corpus, we analyze the content of each document and refrain from collecting redundant webpages (i.e., keeping only one copy of such pages), using the document similarity measure described in Section 4.2.3.
3.3 Data Size and Language Distribution
Using the crawled HTML webpage documents, we compiled a Dark Web corpus consisting of exactly 10K web documents from a total of 7101 onion services. The user statistics for Tor shows that clients connect to the Dark Web from various countries<sup>6</sup>, which is reflected in the language distribution of CoDA (as seen in Table 2). We observe that about  $88\%$  of documents in CoDA is in English. This is
5Including but not limited to https://github.com/ alecmuffett/real-world-onion-sites
https://metrics.torproject.org/userstats-bridge-table.html
in line with the language distribution of DUTA in which  $84\%$  of the samples are in English (Al-Nabki et al., 2019). We argue that the dataset reflects the various biases of the Dark Web, which should be taken into account for future research.
We recruited 10 annotators from a cyber threat analytics company specializing in the Dark Web for manual page-level annotation, i.e., assigning a single category to each webpage document from one of the ten categories achieving an inter-annotator agreement Fleiss' Kappa of  $0.88^{7}$ . This is in contrast to DUTA, which concatenates multiple pages from a single onion domain into one document to assign a category (Al Nabki et al., 2017). Since onion services such as wikis and forums usually contain discussions on a wide range of topics across different pages, page-level annotation was deemed to be the most suitable choice for our category set. We leveraged Prodigy $^{8}$  for an efficient annotation process.
3.5 Annotation Guidelines
A set of comprehensive annotation guidelines was constructed for the annotators to consult when labeling each document to ensure the quality of labels. While the annotation guidelines are extensive with illustrative examples and methods to deal with borderline cases, we present a brief summary of our guidelines in Table 1. Each category is determined based solely on the topic of page content, and not by its type (marketplaces, services, forums, news, blogs, wikis, search results, etc.).
Note that webpages sometimes cover more than one specific topic on a single page (such as a marketplace selling drugs and weapons at the same time). We exclude such multi-topic pages from our corpus and leave multi-label datasets and classification for future work. Finally, we also exclude webpages that contain malicious information on personally identifiable individuals.[9]
No	Mask ID	Description (example)
1	ID_IP_ADDRESS	IPv4 address (xxx.xxx.xxx.xxx)
2	ID_EMAIL	Email address (xxx@yyy.zzz)
3	ID_ONION_URL	Onion URL
4	ID_NORMAL_URL	Non-onion URL
5	ID_BTC_ADDRESS	Bitcoin address
6	ID_ETH_ADDRESS	Ethereum address
7	ID_LTC_ADDRESS	Litecoin address
8	ID_general-money	Fiat money (10 USD, ¥500)
9	ID_CRYPTOMoney	Cryptocurrency (0.01 BTC, 10 mBTC)
10	ID.Weight	Weight (10kg, 10lbs)
11	ID_LENGTH	Length (10cm, 10mm)
12	ID_volume	Volume (10ml, 5L)
13	ID_TIME	Widely used date/time format (2000-01-01 09:00:00, 01-Jan-2020)
14	ID_PERCENTAGE	Percentage (50%)
15	ID_FILENAME	File names with popular extensions (xxx.zip, yyyy.pdf)
16	ID_FILESIZE	File size (10MB, 16GB)
17	ID_VERSION	Version names (version 5.0, v1.0.0)
18	ID_NUMBER	All the other number tokens
Table 3: Mask identifiers in CoDA
3.6 Additional Processing & Text Masking
As the Dark Web contains webpages in various languages, we label each of the documents in CoDA with the language of its content using fastText (Joulin et al., 2016a; Joulin et al., 2016b). To generalize unnecessary details and anonymize sensitive information in the Dark Web, we process each document by masking appropriate information with mask identifiers. A total of 18 types of mask identifiers are used to mask each document, as shown in Table 3. We utilize simple keywords, regular expressions, and a cryptocurrency address detection library $^{10}$  to detect such phrases for masking. This prevents personal information such as email addresses from being included in our public dataset. Finally, to filter out noisy data and optimize text for linguistic analysis, we remove punctuations and words that are over 50 letters long, lemmatize words, and convert all text to lowercase.
To assess the linguistic properties of the Dark Web, illustrate the characteristics of textual content in each category, and better understand the differences in the use of language in the Dark / Surface Web, we conduct an in-depth exploratory data analysis. We analyze the text data of CoDA and compare measurements with that of the other datasets (see Section 4.1) as shown in Table 4. $^{11}$
7All annotators participated in a training session to reach agreement with a small set of documents and annotation guidelines. The detailed process is described in Appendix A.
<sup>8</sup>https://prodi.gy/
Ransomware and extortionware cybercriminals deliberately publish private and harmful information of victims to demand a ransom for its removal. We did not find such content in our dataset, possibly because our data only collects texts without downloading files or media.
10 https://pypi.org/project/ cryptoaddress/
<sup>11</sup>Some detailed methods and results obtained from our analyses are provided in Appendix D.
Data Analysis Methods / Statistics	Dataset
CoDA (ours)	DUTA-10K	Surface Web Aggregate
Analysis Using Raw Data
In-vocab / out-of-vocab words (\$4.2.1)	○	○	○
PoS distribution (\$D.1)	○	○
Content / function word ratio (\$D.1)	○	○
Word frequency distribution (\$4.2.2)	○	○	○
Analysis Using Masked Data
Document similarity (\$4.2.3)	○	○
Mask token distribution (\$4.2.4)	○
TF-IDF (\$4.2.4)	○
4.1 Datasets for Comparison
We compare CoDA with DUTA to look for any significant differences between the two Dark Web corpora, and use documents labeled as English for our analysis. We also aggregate three existing text datasets in English with Surface Web content (we consider each of these datasets as a single subcategory within the aggregate Surface Web data) from here on to compare between the Dark / Surface Web domains. These categories are chosen to encompass the various topics and language styles (formal / informal) used throughout the Surface Web.
The aggregate Surface Web dataset consists of the following categories: the IMDb Large Movie Review Dataset (Maas et al., 2011), the Wikitext-2 Dataset (Merit et al., 2016), and the Reddit Corpus (Chang et al., 2020), to represent review texts, wiki articles, and online forum discussions, respectively. To match the size of the dataset with its Dark Web counterparts, the aggregate Surface Web dataset is trimmed by randomly sampling a portion of documents from each category. The total raw word count of each dataset is shown in Table 5.
It is worth noting that we use raw text data instead of the masked data for some analyses to reduce bias; for example, the Surface Web aggregate dataset is not masked, so we use the non-masked versions of the Dark Web datasets for some comparisons.
4.2 Analysis Methods & Results
4.2.1 In-vocab / Out-of-vocab Word Analysis
It is known that Dark Web users intentionally use obscure slang and words to refer to specific items (Harviainen et al., 2020; Zhang and Zou, 2020). To verify if this behavior affects the types
Table 4: List of data analysis methods/statistics and comparisons between datasets. Some analyses are presented in the Appendix.
Dataset	Total Word Count	In-vocab	Out-of-vocab
CoDA	9.51M	39.6%	60.4%
(45670)	(69696)
DUTA-10K	7.60M	44.7%	55.3%
(37676)	(46651)
Surface Web Aggregate	7.41M	44.9%	55.1%
(46817)	(57361)
Table 5: The total raw word count and the number of unique in-vocab / out-of-vocab words in each dataset. We only consider words that are sequences of purely alphabetic characters separated by whitespace.
of words seen in the Dark Web, we analyze the invocabulary and out-of-vocabulary words by building a list of unique words in each dataset and determining the presence of each word in a spellchecker library $^{12}$ . Words not listed in the library's dictionary are defined to be out-of-vocabulary. Note out-of-vocab words do not necessarily correspond to incorrect or nonexistent words; for example, there are many abbreviations that are out-of-vocab but are widely used online.
The results are shown in Table 5. Due to the limited number of in-vocab words in the dictionary, it follows that in-vocab word ratio decreases with higher total word counts. Therefore, not much can be said about the lexical characteristics of the Dark Web from the ratios. To see which out-of-vocab words frequently appear in each corpus, we rank them by their frequency. In the Surface Web, common abbreviations, well-known companies, and celebrity names rank high in the list, while explicit slangs, malicious activity-specific abbreviations, and misspellings (cannibalism, pedofilia, shcool) manifest the Dark Web. A sample comparison of abbreviations found in each web domain is shown in Table 6. The Surface Web mainly exhibits commonly used abbreviations such as measurement units, household products, and colloquial Internet language, while the Dark Web exhibits abbreviations related to financial services, drugs, and pornography.
4.2.2 Word Frequency Distribution
It is well known that large text corpora tend to follow Zipf's law (Piantadosi, 2014), which states that the word frequency distribution is proportional to a power law of the form:
$$
f (r) \propto r ^ {- \alpha}
$$
$^{12}$ https://github.com/barrust/ pyspellchecker
Common Abbreviations
CoDA (Dark Web)	Surface Web
btc	cp	cvc	btw	dvd	idk
cvv	hd	irc	imo	km	lmao
lsd	mg	pthc	mph	pc	st
ssn	vpn	xxx	tv	vs	wtf
Table 6: Some common abbreviation examples of CoDA and Surface Web Aggregate. These abbreviations do not show up on pyspellchecker's dictionary and are marked as out-of-vocab words.
for  $\alpha \approx 1$ , where  $r$  is the frequency rank of the word (most frequent word has  $r = 1$  and so on) and  $f(r)$  is the frequency of a word with rank  $r$ . To verify whether the characteristics of language used in the Dark Web affect the power law distribution of words, we compare the word frequency distribution between Dark and Surface Web corpora. We aggregate all texts in each category into a single file, lemmatize each word using  $\mathrm{spaCy}^{13}$ , and use scikit-learn $^{14}$  (Pedregosa et al., 2011) to retrieve the word frequency per category.
We find that, as far as word frequency distribution is concerned, there is no significant difference between the Dark Web and the Surface Web<sup>15</sup>. As the Dark Web contains many phishing sites which are near identical copies of each other (Yoon et al., 2019), we believed that some words may have abnormally high frequencies, which would affect the overall distribution. However, the results suggest that word frequency distribution is largely domain-invariant.
4.2.3 Document Similarity
As mentioned in Section 1, about half of DUTA's documents contain duplicate data. CoDA addresses this problem by crawling web pages whose textual content is less similar to one another. To this end, we measure the document similarity between the two Dark Web corpora to validate the uniqueness of documents in CoDA. Through manual inspection, we find that some pages share the same exact content but with slight variations in details such as numbers. To prevent such differences from affecting the document similarity, we mask and preprocess documents in DUTA in the same manner as CoDA and convert each document into a bag of lowercase words. The similarity is measured by
Figure 1: Maximum similarity (Jaccard distance) graph of documents in CoDA and DUTA. The similarities between every document in the same dataset are measured and the maximum similarity is taken for each document.
taking the Jaccard distance on the bags of words, with distance of 1 indicating complete similarity between two documents.