DarkBERT: A Language Model for the Dark Side of the Internet
Year: 2023

Abstract:
Recent research has suggested that there are clear differences in the language used in the Dark Web compared to that of the Surface Web. As studies on the Dark Web commonly require textual analysis of the domain, language models specific to the Dark Web may provide valuable insights to researchers. In this work, we introduce DarkBERT, a language model pretrained on Dark Web data. We describe the steps taken to filter and compile the text data used to train DarkBERT to combat the extreme lexical and structural diversity of the Dark Web that may be detrimental to building a proper representation of the domain. We evaluate DarkBERT and its vanilla counterpart along with other widely used language models to validate the benefits that a Dark Web domain specific model offers in various use cases. Our evaluations show that DarkBERT outperforms current language models and may serve as a valuable resource for future research on the Dark Web.

heading: 1 Introduction
text: The Dark Web is a subset of the Internet that is not indexed by web search engines such as Google and is inaccessible through a standard web browser. To access the Dark Web, specialized overlay network applications such as Tor (The Onion Router) (Dingledine et al., 2004) are required. Tor also hosts hidden services (onion services) — web services in which the client and the server IP addresses are hidden from each other (Biryukov et al., 2013).
This sense of identity obscurity provided to the Dark Web users comes with a catch; many of the underground activities prevalent in the Dark Web are immoral/illegal in nature, ranging from content hosting such as data leaks to drug sales (Al Nabki et al., 2017; Jin et al., 2022). As such, the popularity of the Dark Web as a platform of choice for malicious activities has garnered interest from researchers and security experts alike.
To handle the ever-changing landscape of modern cyber threats, cybersecurity experts and researchers have started to employ natural language processing (NLP) methods. Gaining evidencebased knowledge such as indicators of compromise (IOC) to mitigate emerging threats is an integral part of modern cybersecurity known as cyber threat intelligence (CTI) (Liao et al., 2016; Bromiley, 2016), and modern NLP tools have become an indispensable part of CTI research. As such, the use of NLP techniques has also been extended to the Dark Web (Jin et al., 2022; Yoon et al., 2019; Choshen et al., 2019; Al Nabki et al., 2017; AlNabki et al., 2019; Yuan et al., 2018). The continued exploitation of the Dark Web as a platform of cybercrime makes it a valuable and necessary domain for CTI research.
Recently, Jin et al. (2022) observed that using a BERT-based classification model achieves state-ofthe-art performance among available NLP methods in the Dark Web. However, BERT is trained on Surface Web1 content (i.e., Wikipedia and BookCorpus) (Devlin et al., 2019), which has different linguistic characteristics from that of the Dark Web (Choshen et al., 2019). In the context of CTI, this implies that popular pretrained language models such as BERT are not ideal for Dark Web research in terms of extracting useful information due to the differences in the language used in the two domains. Consequently, an NLP tool that is suitable for application in Dark Web domain tasks would prove to be valuable in the ongoing efforts of Dark Web cybersecurity.
In this paper, we propose DarkBERT, a new language model pretrained on a Dark Web corpus. To measure the usefulness of DarkBERT in handling cyber threats in the Dark Web, we evaluate DarkBERT in tasks related to detecting underground activities. We compare DarkBERT to other widely
1Web services and content that are readily available and indexed in common search engines such as Google
7515
used pretrained language models BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) that are trained on data found in the Surface Web to verify the efficacy of DarkBERT in Dark Web domain texts. Our evaluation results show that DarkBERT-based classification model outperforms that of known pretrained language models. Furthermore, we present potential use cases to illustrate the benefits of utilizing DarkBERT in cybersecurityrelated tasks such as Dark Web forum thread detection and ransomware leak site detection.
Our contributions are summarized as follows: • We introduce DarkBERT, a language model
pretrained on the Dark Web which is capable of representing the language used in the domain compared to that of the Surface Web. • We illustrate the effectiveness of DarkBERT in the Dark Web domain. Our evaluations show that DarkBERT is better suited for NLP tasks on Dark Web specific texts compared to other pretrained language models. • We demonstrate potential use case scenarios for DarkBERT and show that it is better-suited for tasks related to cybersecurity compared to other pretrained language models. • We provide new datasets used for our Dark Web domain use case evaluation.

heading: 2 Related Work
text: The recent availability of Dark Web resources (Jin et al., 2022; Al Nabki et al., 2017; Al-Nabki et al., 2019) has made it possible to explore the differences between the languages used in the Dark Web and the Surface Web. Choshen et al. (2019) explored the differences in the illegal and legal pages in the Dark Web and found a number of distin-
guishing features between the two domains such as named entity, vocabulary, and syntactic structure. Their analyses using standard NLP tools have also suggested that processing text in the Dark Web domain would require considerable domain adaptation. The linguistic differences between the Surface Web and the Dark Web were further examined by Jin et al. (2022) through linguistic features such as part-of-speech (POS) distribution and vocabulary usage between the texts in the two domains.
Recently, Ranaldi et al. (2022) explored the use of pretrained language models over Dark Web texts to examine the effectiveness of such models, and suggested that lexical and syntactic models such as GloVe (Pennington et al., 2014) outperform pretrained models in some specific Dark Web tasks. Meanwhile, Jin et al. (2022) demonstrated that pretrained language models in some Dark Web tasks such as Dark Web activity classification perform better than simple lexical models, suggesting that language models like BERT show promising results in the Dark Web. Either way, a domain-specific pretrained language model would be beneficial in that it would be able to represent the language used in the Dark Web, which may effectively reduce the performance issues faced in previous experiments.

heading: 3 DarkBERT Construction
text: In this section, we describe the process for building our Dark Web domain-specific pretrained language model, DarkBERT. We begin by collecting pages to build the text corpus used for pretraining DarkBERT (Section 3.1). Then, we filter the raw text corpus and employ text preprocessing methods for pretraining purposes (Section 3.2). Finally, we pretrain DarkBERT using the text corpus (Section 3.3).
An overview of the DarkBERT construction process is illustrated in Figure 1.

heading: 3.1 Data Collection
text: A massive text corpus consisting of pages from the Dark Web is necessary for pretraining DarkBERT. We initially collect seed addresses from Ahmia2 and public repositories containing lists of onion domains. We then crawl the Dark Web for pages from the initial seed addresses and expand our list of domains, parsing each newly collected page with the HTML title and body elements of each page saved as a text file. We also classify each page by its primary language using fastText (Joulin et al., 2016a,b) and select pages labeled as English. This allows DarkBERT to be trained on English texts as the vast majority of Dark Web content is in English (Jin et al., 2022; He et al., 2019). A total of around 6.1 million pages was collected. The full statistics of the crawled Dark Web data is shown in Table 8 of the Appendix.

heading: 3.2 Data Filtering and Text Processing
text: While the text data collected in Section 3.1 is of considerable size, a portion of the data contains no meaningful information such as error messages or duplicates of other pages. Therefore, we take three measures — removal of pages with low information density, category balancing, and deduplication — to retain useful page samples in the pretraining corpus and remove unnecessary pages. In addition, it is critical that the model does not learn representations from sensitive information. Although a previous study stated that language models pretrained with sensitive data are unable to extract sensitive information with simple methods, the possibility cannot be ruled out using more sophisticated attacks (Lehman et al., 2021). To this end, we preprocess the pretraining corpus to address ethical considerations using identifier masks or removing texts entirely, depending on the type of the target text. The details of filtering and text preprocessing are described in Sections B and C of the Appendix.
2https://ahmia.fi/

heading: 3.3 DarkBERT Pretraining
text: In order to observe the impact of text preprocessing on DarkBERT’s performance, we build two versions of DarkBERT: one with raw text data (whitespace removal applied) and the other with preprocessed text following Section 3.2. The size of each pretraining corpus is shown in Table 1, and the training losses for the two models are shown in Figure 2.
We leverage an existing model architecture instead of starting from scratch for pretraining. This is done to reduce computational load and retain the general English representation learned by the existing model. We choose RoBERTa (Liu et al., 2019) as our base initialization model as it opts out of the Next Sentence Prediction (NSP) task during pretraining, which may serve as a benefit to training a domain-specific corpus like the Dark Web as sentence-like structures are not as prevalent compared to the Surface Web.
The Dark Web pretraining text corpus is fed to the roberta-base model in the Hugging Face 3 library as an initial base model. For compatibility between DarkBERT and RoBERTa, we use the same BPE (byte-pair encoding) tokenization vocabulary used in the original RoBERTa model, with each page in the pretraining corpus separated using RoBERTa’s separator token 
. The two versions of DarkBERT only differ in the corpus used for pretraining (raw vs. preprocessed); all other factors such as training hyperparameters are equally set. The models are pretrained using a script written in PyTorch (Paszke et al., 2019). Additional
3https://huggingface.co/
details on pretraining including hyperparameters and training equipment are listed in Table 11 and Section D of the Appendix.

heading: 4 Evaluation: Dark Web Activity
text: Classification
In this section, we describe the methods of evaluation and the datasets used to evaluate DarkBERT and other language models. Since page classification has often been performed in past works (Al Nabki et al., 2017; Choshen et al., 2019; Ranaldi et al., 2022), we also choose Dark Web activity classification as the main Dark Web domain benchmark experiment for evaluation. We additionally conduct experiments on multiple use case scenarios, which is described in detail in Section 5.

heading: 4.1 Datasets
text: The distribution of various activities has been studied at large, resulting in publicly available Dark Web text datasets known as DUTA (Al Nabki et al., 2017; Al-Nabki et al., 2019) and CoDA (Jin et al., 2022). We use english texts in the latest version of DUTA (also known as DUTA-10K) and CoDA in our experiments. Since DUTA and CoDA use different categorization methods, we train separate classifiers for each dataset. Since DUTA contains certain categories that are very small in size (for example, there are only 3 pages under the Human Trafficking category), we remove categories that have a low page count (under 1% of total page count). We also remove the Empty category in DUTA as these pages are mostly empty, which is not ideal for text classification. No modifications are made to the CoDA dataset. Finally, we prepro-
cess texts in both DUTA and CoDA following Section 3.2. Per-category statistics for the two datasets used for our activity classification experiment are shown in Table 2.

heading: 4.2 Experimentation
text: The classification experiment is conducted on the two versions of DarkBERT and two widely used language models: BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). Although RoBERTa (and the two variants of DarkBERT which use RoBERTa as their base model) is a cased language model which distinguishes between capitalized words and uncapitalized words, BERT comes in two versions: a cased model and an uncased model. To observe if letter case has any effect on classification performance, we build a separate, uncased version of DUTA and CoDA in which every character is converted to lowercase. In summary, we evaluate the Dark Web activity classification task using DUTA and CoDA — each with two variants: cased and uncased corpus — on two versions of DarkBERT (raw and preprocessed), two versions of BERT (cased and uncased), and RoBERTa.

heading: 4.3 Results and Discussion
text: The result of Dark Web activity classification is shown in Table 3. We observe that DarkBERT outperforms other language models for both datasets and their variants. However, it is also worth noting that both BERT and RoBERTa exhibit relatively similar performances to DarkBERT. This is in line with previous classification experiments with CoDA, which have shown that BERT is able to adapt relatively well to other domains (Jin et al., 2022). RoBERTa also performs slightly better compared to BERT, which reflects the advantages in performance that RoBERTa has over BERT as mentioned in the original paper (Liu et al., 2019).
We also observe that all language models perform significantly better for the CoDA dataset compared to DUTA. Upon closer inspection on the DUTA dataset, we find that some of the included categories in DUTA may not be suitable for classification tasks. For example, many of the pages in the Hosting & Software category contain duplicate texts, which may overfit the model during fine-tuning (DUTA in general has duplicate texts as mentioned by Al-Nabki et al. (2019)). In addition, some of the pages seem to be ambiguous in terms of classification the DUTA dataset; for eaxmple, we observe pages classified as Hosting & Software
Table 3: Dark Web activity classification evaluation results. Boldface indicates best performance.
Dataset Model Precision Recall F1 score Dataset Model Precision Recall F1 score
DUTAcased
BERTcased 77.31 76.91 77.09
CoDAcased BERTcased 92.12 92.16 92.13 BERTuncased 78.21 78.20 78.19 BERTuncased 92.83 92.67 92.75 RoBERTa 78.54 78.79 78.63 RoBERTa 93.36 93.27 93.31 DarkBERTraw 80.11 79.94 80.01 DarkBERTraw 94.15 94.35 94.25 DarkBERTpreproc 79.90 80.08 79.98 DarkBERTpreproc 94.26 94.33 94.29
DUTAuncased
BERTcased 78.11 77.97 77.99
CoDAuncased BERTcased 92.86 92.85 92.85 BERTuncased 78.21 78.20 78.19 BERTuncased 92.83 92.67 92.75 RoBERTa 78.42 78.36 78.37 RoBERTa 93.30 93.40 93.34 DarkBERTraw 79.47 79.49 79.47 DarkBERTraw 94.46 94.45 94.46 DarkBERTpreproc 79.65 79.77 79.69 DarkBERTpreproc 94.31 94.53 94.42
that do not contain any activities related to hosting or software related terms.
We take a deeper look at the activity classification results on the CoDA (cased) dataset by constructing confusion matrices (Figure 8 of the Appendix) to check for misclassifications. We find that in general, the two versions of DarkBERT show the best classification performance for most categories. The highest number of correct classifications for every category occurs in either one of the DarkBERT models. However, some categories such as Drugs, Electronics, and Gambling show very similar performances across all four models. This is likely due to the high similarity of pages in such categories, making classification easier even with the differences in the language used in the Dark Web. Finally, we inspect the language models using their predictions through error analysis, which is described in Section E.1 of the Appendix.

heading: 5 Use Cases in the Cybersecurity Domain
text: In this section, we introduce three Dark Web domain use cases for DarkBERT and demonstrate its effectiveness over existing language models in cybersecurity / CTI applications. We list details on the experimental setup for each use case in Sections E.2 and E.3 of the Appendix.

heading: 5.1 Ransomware Leak Site Detection
text: One type of cybercrime that occurs on the Dark Web is the selling or publishing of private, confidential data of organizations leaked by ransomware groups. This can occur in the form of leak sites that expose victims and threaten to release sensitive data (such as financial information, private assets, and personal identification) of uncooperative victims (Yuste and Pastrana, 2021). It would thus be
(a) A ransomware leak site sample
(b) A noteworthy thread sample
Figure 3: A ransomware leak site and noteworthy thread samples that DarkBERT correctly classified but are misclassified by other language models.
beneficial for security researchers to automatically identify such websites. We formulate the task of leak site detection as a binary classification problem of predicting whether a given page is a leak site or not. We compare the effectiveness of this task using the pretrained language models used for evaluation in Section 4 (BERT, RoBERTa, and DarkBERT). Datasets: We monitor leak sites of 54 popular ransomware groups for two years (from May 2020
to April 2022), and periodically download HTML files from these sites especially when new victims are revealed4. Leak sites typically contain the victim organization name, descriptions of leaked data, and threat statements with sample data (refer to Figure 3a for an example leak site page).
We collect pages by randomly choosing a maximum of three pages with different page titles from each of the 54 leak sites, and label them as positive examples. To create negative data, rather than collecting random pages in the Dark Web, we consider pages with content similar to that of leak sites to make the task more challenging. To select such pages, we utilize the activity category classifier from Section B used for balancing the pretraining corpus. The intuition behind using the activity classifier to select negative data is that the text content of certain categories like Hacking are more similar to that of leak sites than other less relevant categories such as Pornography and Gambling. Our pilot study suggests leak sites are mostly classified by the activity classifier as Hacking, followed by Cryptocurrency, Financial, and Others. Thus, we only collect Dark Web pages that are classified into one of these four categories and treat them as negative examples. Our training text data consists of 105 positive and 679 negative examples (pages). Training is done using 5-fold cross validation. Results and Discussion: As shown in Table 4, DarkBERT outperforms other language models, demonstrating the advantages of DarkBERT in understanding the language of underground hacking forums on the Dark Web. Figure 3a shows a leak site sample correctly classified by DarkBERT but
4URLs of such leak sites can be found in cybersecurity news, social media, open-source repositories, and so on. We used URLs taken from https://github.com/fastfire/deepdarkCTI/ blob/main/ransomware_gang.md.
misclassified by other models. We also observe that while DarkBERT uses RoBERTa as a base model, RoBERTa itself shows a sharp drop in performance compared to the other models.
In addition, DarkBERT with preprocessed input performs better than the one with raw input, which highlights the importance of the text preprocessing step in terms of reducing superfluous information. As lengthy words or cryptocurrency addresses have been replaced with mask identifier tokens in the preprocessed input, such words present in the raw input may cause the tokenizer to produce uninformative tokens and affect task performance.

heading: 5.2 Noteworthy Thread Detection
text: Dark Web forums are often used for exchanging illicit information, and security experts monitor for noteworthy threads to gain up-to-date information for timely mitigation. Since many new forum posts emerge daily, it takes massive human resources to manually review each thread. Therefore, automating the detection of potentially malicious threads can significantly reduce the workload of security experts. Identifying noteworthy threads, however, requires a basic understanding of Dark Web-specific language. Similar to the aforementioned leak site detection, we can formulate this task as a binary classification problem to predict whether a given forum thread is noteworthy. We compare the performance of noteworthy thread detection for DarkBERT and the baseline models: BERT and RoBERTa. Datasets: Identifying a thread as noteworthy is a highly subjective task. While there can be many different definitions for noteworthiness, we focus on activities in hacking forums that can potentially cause damage to a wide range of victims. To incorporate perspectives from the cybersecurity industry and ensure the quality of the dataset, we recruit
two researchers from a cyber threat intelligence company specializing in the analysis of hacking forums on the Dark Web to discuss types of noteworthy threads, and set annotation guidelines accordingly. We consider a thread of hacking forums to be noteworthy if it describes one of the following activities:
1. Sharing of confidential company assets such as admin access, employee or customer information, transactions, blueprints, source codes, and other confidential documents. 2. Sharing of sensitive or private information of individuals such as credit information, medical records, political engagement, passports, identifications, and citizenship. 3. Distribution of critical malware or vulnerabilities targeting popular software or organizations.
In particular, we place emphasis on activities targeting large private companies, public institutions, and industries. We choose RaidForums, one of the largest hacking forums, as our data source (together with its mirror and follow-up sites5). We collect 1,873 forum threads posted from July 2021 to March 2022 and work with the recruited annotators to select noteworthy threads. They first annotate the same 150 threads and achieve an interannotator agreement of 0.704 as measured by Cohen’s Kappa, which indicates substantial agreement. All disagreements in the annotated dataset are then discussed and resolved by both annotators. The final dataset contains 249 positive (noteworthy) and 1,624 negative threads. We use the title and body text of each thread from the HTML source as input to the classifier and exclude any thread replies to simulate the practical scenario in which we categorize the noteworthiness of threads as soon as they are posted, and training is done using 5-fold cross validation. Results and Discussion: As seen in Table 5, DarkBERT outperforms other language models in terms of precision, recall, and F1 score for both inputs. Similar to ransomware leak site detection, we see a noticeable performance drop for RoBERTa compared to the other models. Figure 3b shows a noteworthy thread sample that is correctly classified by DarkBERT but misclassified by other models. Due to the difficulty of the task itself, the overall performance of DarkBERT for real-world noteworthy
5http://raidforums.com, http://rfmirror. com, http://breached.co
thread detection is not as good compared to those of the previous evaluations and tasks. Nevertheless, the performance of DarkBERT over other language models shown here is significant and displays its potential in Dark Web domain tasks. By adding more training samples and incorporating additional features like author information, we believe that detection performance can be further improved.
It should also be noted that the performances for both raw and preprocessed inputs are similar for DarkBERT. Unlike data used for ransomware leak site detection, thread content is generally shorter than general webpage content, and sensitive information such as URLs and email addresses often influences the noteworthiness of threads (e.g., whether a victim is a leading global company or not). Since such information is masked for preprocessed inputs, contents of noteworthy threads and non-noteworthy threads may look similar from the viewpoint of the language models, which in turn deteriorates the performance of this task.

heading: 5.3 Threat Keyword Inference
text: In this section, we describe how we utilize the fillmask function to derive a set of keywords that are semantically related to threats and drug sales in the Dark Web. Fill-mask is one of the main functionalities of BERT-family language models, which finds the most appropriate word that fits in the masked position of a sentence (masked language modeling). It is useful for capturing which keywords are used to indicate threats in the wild. In order to show that DarkBERT is robust in handling this task, we compare DarkBERT and BERTReddit, a BERT variant fine-tuned on a subreddit corpus whose topic is drugs (Zhu et al., 2021).
Figure 4 shows a sample drug sales page from the Dark Web in which a user advertises a Dutch MDMA pill with the Philipp Plein logo6. We then mask MDMA in the title phrase: 25 X XTC 230 MG DUTCH MDMA PHILIPP PLEIN, and let DarkBERT and BERTReddit suggest the most semantically related words. In Table 6, we list the suggested candidate words by the two language models, respectively. The result shows that DarkBERT suggests drug-related words (i.e., Oxy and Champagne) and a word closely related to drugs (i.e., pills). On the other hand, BERTReddit mainly
6While Philipp Plein normally refers to a German fashion brand, in this case, it indicates an MDMA pill on which the brand logo is imprinted. Well known car brands such as Tesla, Rolls Royce, and Toyota are also used in a similar manner.
suggests professions such as singer, sculptor, and driver, which are not relevant to drugs. This comes from the fact that the preceding word, Dutch, is usually followed by a vocational word in the Surface Web. We evaluate how each language model produces keyword sets semantically related to drugs in a quantitative fashion. Datasets: To evaluate the language models, we use the sample dataset provided by Zhu et al. (2021). This dataset is composed of ground truth data (i.e., drug names and their euphemisms) and sentences containing the drug names7. Experimental Setting: We compare three language models: DarkBERTCoDA, BERTCoDA, and BERTReddit. The first two language models are fine-tuned on a subset of CoDA documents classified as drugs, whose base model is DarkBERT and BERT, respectively. BERTReddit is a BERT variant fine-tuned on a subreddit corpus whose topic is drugs. To compare them quantitatively, we also use
7The ground truth data are from the DEA Intelligence Report: https://www.dea.gov/sites/default/ files/2018-07/DIR-022-18.pdf
precision at k (P@k) following Zhu et al. (2021). Here, precision at k is the proportion of inferred keywords that are semantically related to a given drug name in the top-k set that are synonymous. Results and Discussion: The measured P@k values are presented in Table 7. DarkBERTCoDA outperforms BERTReddit for k ranging from 10 to 20, but is overtaken for higher values of k. Although DarkBERTCoDA shows better performance when k is small, the ground truth dataset contains euphemisms mainly derived from the Surface Web, and the words that DarkBERTCoDA infers as semantically related words are not contained in the dataset. For instance, Tesla and Champagne are drug names frequently seen in the Dark Web, but are not recognized as such in Zhu et al. (2021). On the other hand, crystal and ice are detected by both DarkBERTCoDA and BERTReddit because they are used in both the Surface Web and the Dark Web.

heading: 6 Conclusion
text: In this study, we propose DarkBERT, a Dark Web domain-specific language model based on the RoBERTa architecture. To allow DarkBERT to adapt well to the language used in the Dark Web, we pretrain the model on a large-scale Dark Web corpus collected by crawling the Tor network. We also polish the pretraining corpus through data filtering and deduplication, along with data preprocessing to address the potential ethical concerns in Dark Web texts related to sensitive information. We show that DarkBERT outperforms existing language models with evaluations on Dark Web domain tasks, as well as introduce new datasets that can be used for such tasks. DarkBERT shows promise in its applicability on future research in the Dark Web domain and in the cyber threat industry. In the future, we also plan to improve the performance of Dark Web domain specific pretrained language models using more recent architectures and crawl additional data to allow the construction of a multilingual language model.

heading: Ethical Considerations

heading: Crawling the Dark Web
text: While crawling the Dark Web, we take caution not to expose ourselves to content that should not be accessed. For example, illicit pornographic content (such as child pornography) are easily found on the Dark Web. However, our automated web crawler takes the approach of removing any non-text media and only stores raw text data. By doing so, we do not expose ourselves to any sensitive media that is potentially illegal.
Sensitive Information Masking
Since the Dark Web harbors many activities considered to be malicious in nature, it is of utmost importance that sensitive data be left out of the text corpus used for pretraining. In particular, it is possible that some contents in the Dark Web may include private information such as e-mails, phone numbers, or IP addresses. To prevent DarkBERT from learning representations from sensitive texts as mentioned above, we mask our data before feeding it to our language model. While we have used both DarkBERT pretrained on preprocessed text and raw text for our experiments, we have used both of the models only for evaluation purposes. In addition, we only release the preprocessed version of DarkBERT in order to avoid any malpractices 8. Through extensive testing on fill-mask and synonym inference tasks, we observe that it is infeasible to infer any characteristics or data that might be considered sensitive or private in nature using the preprocessed version of DarkBERT.

heading: Annotator Ethics
text: For the task of noteworthy thread detection, we recruited two researchers from a cyber threat intelligence company as mentioned in Section 5.2, who agreed to assist us in our research methods. For a fair annotation process in the discussion of noteworthy threads, both recruited annotators handled the same set of thread data and were given equal compensations.

heading: Use of Public Dark Web Datasets
text: Both DUTA and CoDA are available upon request by the respective authors, and due to the sensitive nature of the Dark Web domain, these datasets are
8Researchers can request access to DarkBERT and related use case datasets by filling out the request form in the following url: https://s2w.inc/resources/darkbert/
only to be used for academic research purposes. We adhere to this guideline and only utilize the provided data in the context of research for this work. On the other hand, we do not plan to publicly release the Dark Web text corpus used for pretraining DarkBERT for similar reasons.

heading: Limitations

heading: Limited Usage for Non-English Tasks
text: As mentioned in Section 3, DarkBERT is pretrained using Dark Web texts in English. This is mainly our design choice as the vast majority (around 90%) of Dark Web texts is primarily in English (Jin et al., 2022). We believe that with the limited number of collected pages in non-English languages in our pretraining corpus, building a multilingual language model for the Dark Web domain would pose additional challenges, such as downstream task evaluations becoming more difficult to perform as they would require high-quality annotations of taskspecific datasets in multiple languages. As such, while our language model is suitable for Dark Web tasks in English, further pretraining with languagespecific data may be necessary to use DarkBERT for non-English tasks.

heading: Dependence on Task-Specific Data
text: Although DarkBERT is a useful tool that can be directly applied to many existing Dark Web domainspecific tasks, some tasks may require further fine-tuning through task-specific data (as seen in Ransomware Leak Site Detection and Noteworthy Thread Detection use case scenarios in Section 5). However, there is a shortage of publicly available Dark Web task-specific data. While we provide the datasets used to fine-tune DarkBERT in this paper, additional research on tasks that do not have readily available datasets for use may require further manual annotation or handcrafting of necessary data to leverage DarkBERT to its maximum potential.

heading: Acknowledgements
text: This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT). (No. 2022-0-00740, The Development of Darkweb Hidden Service Identification and Real IP Trace Technology)

heading: A Appendix
text: We list some additional details such as example figures from the DarkBERT evaluation and cybersecurity use case experiments mentioned in Sections 4 and 5. Select portions of figures have been blurred out to comply with the ethical guidelines to hide sensitive information.

heading: B Data Filtering Details
text: Removal of pages with low information density: Initially, we decide to leave out pages that have an abnormally high or low character count. This is done to exclude content that is not seemingly useful in the representation of the Dark Web. For example, most of the pages containing an abnormally low character count are error messages such as “404 not found” or “Captcha error” and log-in messages such as “Sign In” or “Already have an account?”. On the other hand, the pages that contain an abnormally high character count are mostly large lists of keywords or continuous repetitions of certain strings. These texts are not very useful as they contain low information density of Dark Web content, and are therefore removed from the pretraining corpus.
To decide on the minimum and the maximum threshold of character counts to remove from the crawled data, we measure the per-page character count statistics as shown in Table 8, and use approximately half the character count value from the 25th quartile (500 characters) and double the character count value from the 75th quartile (10,000 characters). This is done so that the majority of the pages are still included in the pretraining corpus while also serving as a generous threshold for pages containing unwanted data as shown above. By filtering out pages below the minimum and above the maximum threshold for their character count, we are left with 5.43 million pages out of the initial 6.1 million.
Category balancing: Previous studies (Al Nabki et al., 2017; Jin et al., 2022) have found through their web crawling that pornographic content is one of the most common activities found in the Dark Web. One of the challenges in pretraining DarkBERT is to use text data that consists of various content found in the Dark Web while avoiding skewness in which certain activities constitute a significant fraction of the entire dataset. If these activities (that take up a large portion of the corpus) exist, then the learned representation of the language model would be more biased towards such activities through pretraining.
To address the issue of balancing content in the pretraining corpus, we attempt an automated categorization of every page. A general categorization of various activities and the guidelines for each activity were addressed by Jin et al. (2022), where each page in the Dark Web was sorted into a total of 10 categories. Following this classification methodology, we train a simple page classification model using BERT (Devlin et al., 2019). Although the use of vanilla BERT may seem contradicting due to the domain differences between the Surface Web (the domain of origin for the texts that BERT was pretrained with) and the Dark Web, it is not necessary for this classification model to achieve high performance since our goal is to obtain a general grasp of the pretraining corpus category distribution.
We implement the model by finetuning the bert-base-uncasedmodel from the Hugging Face library (Wolf et al., 2020) with the CoDA Dark Web text corpus (Jin et al., 2022). This model is then run through the entire pretraining corpus to output a specific category for each page. We use 9 of the 10 predefined categories from CoDA and exclude the Others category, because most of the pages that fit in this category (log-in pages, error pages, etc.) have already been filtered out from the pretraining corpus through character count filtering. In addition, we found that pages are more likely to be misclassified as Others category compared to other categories, meaning that the exclusion of Others category would yield a more accurate category distribution.
The page category statistics resulting from classification is shown in Table 9. We observe from our data that pornography accounts for the highest fraction of all categories in the Dark Web, making up 41.7% of all pages. Meanwhile, categories such as gambling and arms / weapons make up less than 1% of all pages each. Even with the use
of vanilla BERT and the exclusion of the Others category taken into consideration, it is evident that the variation of content in the pretraining corpus is unbalanced. To this end, we take a rather simple approach of random removal of pages from overrepresented categories until all categories have similar amounts of content. Deduplication: A significant portion of the Dark Web is duplicate content. Since pretraining language models requires considerable resource and time, reducing the pretraining corpus size through deduplication is beneficial. This process is handled by minhashing (Broder et al., 2000) each page in the corpus and removing duplicate pages until all remaining minhash values are unique.
The pretraining corpus statistics after applying random removal of over-represented pages and deduplication is shown in Table 9. The deduplication rate represents the reduction in data size as a result of deduplication only, while the total reduction rate represents the reduction in data size as a result of both deduplication and random removal for category balancing. Both are based on the ratio between the initial data size (Table 9) and the final data size of each category. We observe that most categories have deduplication rates of less than 10%. However, categories such as drugs and hacking exhibit high deduplication rates. In addition, the deduplication rate and the total reduction rate of gambling and arms / weapons categories are equal, since we did not perform random removal of pages as these categories were already initially small in terms of data size. Finally, the size difference between the smallest category (arms / weapons) and the largest category (pornography) is 7-fold in the final pretraining corpus, compared to the 70-fold difference in size observed from the initial data.
C Identifier Mask Details
Here, we give an extended discussion on each of the identifier masks used for text processing men-
tioned in Section 3.2. The types of identifier masks used for preprocessing the pretraining corpus is illustrated in Table 10. Implementation: Some identifier types such as URLs and IP addresses always contain distinct patterns. These identifiers are searched and undergo substitution using regular expressions. Other identifier types such as emails and phone numbers are masked using the text preprocessing API provided by textacy 9. Email Addresses: Email addresses are often seen in the Dark Web as a means of communication. Unlike the contacts commonly seen in the Surface Web, many of the email addresses listed in the Dark Web are those that provide end-to-end encryption services such as ProtonMail 10 to prioritize privacy. However, some email addresses can include strings that can be traced to a single individual, so all email addresses are masked. URLs: There are two identifier types for URLs: onion domain addresses and non-onion domain addresses. While URLs do not necessarily expose personal information themselves, it is possible that links to some URLs may be contain harmful information or data. To eliminate the possibility of such URLs from being learned as a representation of the Dark Web, we mask all URLs. IP Addresses: Although the Dark Web is used to hide IP addresses, some pages contain IP addresses in their texts. Many of the pages that contain IP addresses are Tor relay sites, which show information such as Tor exit relay node addresses (the IP addresses listed in the Tor relay sites can also easily be found on the Surface Web at https://metrics. torproject.org/rs.html). Given the frequent illegal activities occurring in the Dark Web,
9https://textacy.readthedocs.io/en/ latest/
10https://proton.me/mail
it is possible that some IP addresses listed in these pages may exist for malicious purposes. For example, Winter et al. (2014) has shown that some malicious exit relays have been engaging in HTTPS man-in-the-middle attacks. Therefore, we found it necessary to mask all IP addresses (both IPv4 and IPv6 addresses are masked). Lengthy Words: While exploring some of the unpreprocessed text in the pretraining corpus, we found that certain pages contain words (string of characters separated by whitespace) that are extremely long in length. On closer inspection, most of these lengthy words are URLs, code snippets, hash values, file names, cryptocurrency addresses, and even binaries. While URLs and cryptocurrency addresses can be removed through the preprocessing mask identifiers, other types such as hashes and file names are not separately processed in advance. Hashes in particular would incur overhead in building meaningful vocabulary through tokenization as they do not have specific lexical patterns. In addition, since we do not want executable content such as binaries or detailed file names to be learned by
our language model, we decide to mask all lengthy words. To this end, we define lengthy words by studying the word length distribution, as well as manual inspection of example words for some notable word lengths.
The unique word length distribution for the pretraining corpus is shown in Figure 5. The word length distribution shows a steep upward trend at shorter word lengths (peaking at length of 7) similar to the English language word distribution, and gradually decreases with longer word lengths. As observed in the figure, some specific word lengths appear in much greater frequencies at higher levels. Upon inspection, we find that this is due to some of the commonly used string formats that happen to have specific lengths. For example, many words of length 59 found in our corpus are content identifier (CIDv1) hashes commonly used in IPFS 11, which is a decentralized, hypermedia distribution protocol. Similarly, words of length 64 are mostly SHA-256 hashes.
Our manual inspection of some of the vocabularies present for each word length shows that at around length of 38 to 40, the majority of words take the form of hash-like values and meaningless noisy strings. Therefore, we classify words with lengths of 38 or more characters as lengthy words, and mask them from the pretraining corpus. Note that the masking process of lengthy words is performed after masking all other identifiers mentioned previously such as email addresses, URLs, and cryptocurrency addresses. Since texts belonging to such identifiers are lengthy (ex. onion V3 addresses are 56 characters long, and Ethereum addresses consist of 40 digit hexadecimal strings), masking these texts with their associated mask identifiers beforehand prevents them from being misclassified as lengthy words.
11https://ipfs.io/
Uncommon Characters: As mentioned in Section 3.1, we collect pages that are classified as “English”. However, some of these collected pages contain multilingual characters that are not standard English. The inclusion of such nonstandard characters results in noisy tokens during the tokenization process and produces unnecessary token vocabularies, so we remove all the characters that are “uncommon” in contemporary English. Specifically, we remove all Unicode characters that are not one of the 256 characters in the Basic Latin (ASCII characters) and the Latin-1 Supplement (accented alphabets that are often seen in English) category. Cryptocurrency Addresses: Decentralized digital assets like cryptocurrencies are used to make unidentifiable transactions. As many cryptocurrencies are secure by design and provide pseudonymity, the synergy with the anonymous nature of the Dark Web makes them the preferred method of choice for transactions. Studies show that cryptocurrencies have been involved in illegal underground operations (Lee et al., 2019) in the Dark Web and underground marketplaces in general (Soska and Christin, 2015). While cryptocurrencies are known for their pseudonymous properties, many of the transactions are traceable as the entire blockchain is public (for some cryptocurrencies). In particular, we mask Bitcoin, Ethereum, and Litecoin addresses as these three cryptocurrencies are among the most popular in the Dark Web with transparent transaction details (Monero and
Dash are also popular in the Dark Web, but they incorporate added layers of anonymity to further conceal their transactions). (Barysevich and Solad, 2018).

heading: D DarkBERT Pretraining Details
text: Both versions of DarkBERT are pretrained on a machine with Intel Xeon Gold 6348 CPU @ 2.60GHz and 4 NVIDIA A100 80GB GPUs. All 4 GPUs were used to run the pretraining process, and each version of DarkBERT took about 15 days to run (up to 20K training steps — we stopped the pretraining process at training loss convergence). Both versions of DarkBERT share relatively similar training losses over the 20K training steps. Since training loss for both versions of DarkBERT stopped decreasing at around 20K steps, we use the models saved at 20K steps for evaluation.

heading: E Evaluation Details

heading: E.1 Dark Web Activity Classification
text: We implement a classification pipeline using the language models available in the Hugging Face library (bert-base-cased, bert-base-uncased, and roberta-base) and add a fully-connected classification layer on top of the [CLS] token with PyTorch. Evaluation is performed for each model using k-fold cross validation (k = 10), which is implemented using scikit-learn’s StratifiedKFold module (Pedregosa et al., 2011). Each fold is run up to 10 epochs with a learning rate of 2e-5. Error Analysis: We further scrutinize model performance by looking at specific pages in the CoDA dataset that are correctly classified by DarkBERT but are misclassified by the other models. We find that most pages that have been misclassified by BERT and RoBERTa but correctly classified by DarkBERT contain many domain-specific jargons or key phrases seen in that particular activity in the Dark Web. For example, one of the pages under the Financial category that is misclassified by both BERT and RoBERTa as Others contains the name of a credit card seller service (we choose not to reveal the service name for ethical considerations). Another page under the Pornography category contains the phrase red room which is highly correlated to this category of pages in the Dark Web, but is misclassified by both BERT and RoBERTa as Others. Finally, a page under the Crypto category contains blockchain and cryptocurrency terms, but
is misclassified by BERT and RoBERTa as Others. As shown in the above examples, DarkBERT is able to correctly classify pages that contain phrases mostly seen in the Dark Web but are not commonly used in the Surface Web, whereas BERT and RoBERTa tend to misclassify such pages in the Others category as these models consider such words and phrases as generic attributes rather than activity-specific terms.

heading: E.2 Ransomware Leak Site Detection
text: We use the same classification pipeline as activity classification in Section 4 with k-fold crossvalidation (k = 5) and connect fully-connected classification layers on top of the [CLS] token. Similarly, the evaluation is performed on both raw and preprocessed inputs. An early stopping strategy using validation loss is utilized to avoid overfitting. Due to the limited size of the dataset, we choose to repeat k-fold validation 5 times to mitigate the variations in performance per run and average the results. An example data sample used for this task can be seen in Figure 6 and additional details on used hyperparameters can be found in Table 12.

heading: E.3 Noteworthy Thread Detection
text: Similar to ransomware leak site detection, we adopt k-fold cross validation (k = 5) for each model and employ early stopping strategy. Due to the limited size of the dataset, we again use repeated k-fold validation, where the number of repetitions is set to 5. An example data sample used for this task can be seen in Figure 7 and additional details on used parameters can be found in Table 12.

heading: Electronic Financial Gambling

heading: Electronic Financial Gambling
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: DarkBERT shows promise in its applicability on future research in the Dark Web domain and in the cyber threat industry. In the future, we also plan to improve the performance of Dark Web domain specific pretrained language models using more recent architectures and crawl additional data to allow the construction of a multilingual language model