How to Locate Dark Web Hacker Forums for Security Research | by Mitchell Telatnik | Towards Data Science

To most users, Google is the gateway to exploring the internet. However, the deep web contains pages that cannot be indexed by Google. Within this space, lies the dark web — anonymized websites, often called hidden services, dealing in criminal activity from drugs to hacking to human trafficking.
Conducting security research on the dark web can be difficult. Website URLs on the dark web do not follow conventions and are often a random string of letters and numbers followed by the .onion subdomain. These websites require the TOR browser to resolve, and cannot be accessed through traditional browsers such as Chrome or Safari.
In addition, traditional search engines such as Google do not exist on the dark web. Instead, website URLs are either exchanged person to person (online or in-person) or collected into a simple Html directory. This can make it difficult to find hacker forums on the dark web — specifically forums where serious individuals gather.
Some forums can be found on the dark web directories listed above. These can be found on the dark web or even the surface web (the web we all know and that this article is hosted on). However, because these forums are easy to find, they often attract amateurs in the community. In order to locate more relevant forums for security research, the snowball sampling method can be used to crawl the darknet.
Snowball sampling is a method that can be used to locate hidden services on the dark web for security research including data collection and CTI streams. Snowball sampling is a web crawler architecture that takes a root URL and crawls the website for outgoing links to other websites. This process is then continued for each gathered link for a set depth. This crawler will return a large list of URLs to dark web websites that it gathered.
This method is very similar to the way early search engine web crawlers worked. A key example of this can be found in the 1998 paper “The Anatomy of a Large-Scale Hypertextual Web Search Engine” by Sergey Brin and Lawrence Page — the founders of Google.
Snowball sampling works well when applied to forums on the dark web. Users will often link to other forums in forum posts and comments — an example of the exchange on URLs from person to person that was mentioned earlier. By starting with a hacker forum found on a directory (or one you already know of), more serious and security-relevant forums can be quickly located.
In order to develop the dark web crawler, you need to set up your environment. It may be useful to read my article on how to scrape the dark web to better understand the process. It will also be useful in scraping data from the dark web forums you have identified. This article assumes you are using the OSX operating system. However, if you are using Linex or Windows many aspects should still be applicable.
The TOR browser is a browser that uses the TOR network and will allow us to resolve websites using a .onion subdomain. The TOR browser can be downloaded here.
Running a VPN while crawling the dark web can provide you additional security. A virtual private network (VPN) is not required but highly recommended.
For this article, I assume you already have python installed on your machine with an IDE of your choice. If not, many tutorials can be found online.
Pandas is a data manipulation Python package. Pandas will be used to store and export the data scraped to a csv file. Pandas can be installed using pip by typing the following command into your terminal:
pip install pandas
Selenium is a browser automation Python package. Selenium will be used to crawl the websites and extract data. Selenium can be installed using pip by typing the following command into your terminal:
pip install selenium
For selenium to automate a browser, it requires a driver. Because the TOR browser is running off of Firefox, we will be using Mozilla’s Geckodriver. You can download the driver here. After downloading, extract the driver and move it to your ~/.local/bin folder.
The location of the TOR browser’s Firefox binary will also be needed. To find this, right-click on the TOR browser in your applications folder and click on show contents. Then navigate to the Firefox binary and copy the full path. Save this path somewhere for later use.
This implementation will get you started creating a snowball sampling dark web crawler of depth 1. Because the website structure of the forums differs from each other, it can be difficult to automate the crawler beyond depth 1.
First, import the web driver and FirefoxBinary from selenium. Also import pandas as pd and re.
from selenium import webdriverfrom selenium.webdriver.firefox.firefox_binary import FirefoxBinaryimport pandas as pdimport re
Create a variable “binary” and set it to the path of the Firefox binary you saved earlier.
binary = FirefoxBinary(*path to your firefox binary*)
Set the web driver up to use Firefox and pass the binary variable.
driver = webdriver.Firefox(firefox_binary = binary)
Create a variable “starting_node” and set it to the URL of the starting forum.
starting_node = *your url*
Create an empty list “found_nodes”.
found_nodes = []
You will now need to automate your crawler to scrape the text of the forum posts, comments, or both. This process will differ from forum to forum. I recommend reading my article “How to Scrape the Dark Web” for some guidance and the Selenium documentation can be found here.
Compile a regex expression that can be used to identify onion links in the strings.
p = re.compile('\S+onion')
For each string (post text or comment text) search for onion URLs and add them to the list “found_nodes”.
for post in posts:  nodes = p.findall(post)  found_nodes.append(nodes)
The list “found_nodes” will contain the onion links found from the starting node. This algorithm can then be either repeated for each found link or stopped at a depth of 1.
Snowball sampling is a good method for discovering dark web hacker forums and other hidden services. In addition, services found through this method are generally more useful for security research than those found on directories since serious members of hacker communities generally do not congregate on well-advertised sites.
There are some areas of improvement in this implementation, however. Following this implementation of snowball sampling requires each found node to be manually visited and inspected for content. Not only can this be a time-consuming process, but it can also be dangerous to visit unknown onion links. Automatically determining website content through post context would greatly improve this crawler.
I want to reiterate that scraping the dark web can be dangerous. Make sure you take the necessary safety precautions. Please continue to research safe browsing on the dark web. I am not responsible for any harm that occurs.