###### Abstract  
This paper investigates, applies, and evaluates state-of-the-art Large Language Models (LLMs) for the classification of posts from a dark web hackers' forum into four cybersecurity categories. The LLMs applied included Mistral-7B-Instruct-v0.2, Gemma-1.1-7B, Llama-3-8B-Instruct, and Llama-2-7B, with zero-shot learning, few-shot learning, and fine-tuning. The four cyber-security categories consisted of \"Access Control and Management\", \"Availability Protection and Security by Design Mechanisms\", \"Software and Firmware Flaws\", and \"not relevant\". The hackers' posts were also classified and labelled by a human cyber-security expert, allowing a detailed evaluation of the classification accuracy per each LLM and customization/learning method. We verified LLM fine-tuning as the most effective mechanism to enhance the accuracy and reliability of the classifications. The results include the methodology applied and the labelled hackers' posts dataset. large language model; mistral; gemma; llama; cybersecurity; dark web; text classification; few-shot learning; fine-tuning +
Footnote †: journal: _Article_  
0000-0002-4000-0000-0000-0000-0000-0000-00000-0000-00000-00000-00000-00000-00000-00000-00000-00000-00000-000000-000000-000000-000000-0000000-000000000  
efforts on proactive security, involving sources from the dark web and employing state-of-the-art AI tools, have been intensified, both from an academic and commercial perspective, involving white hats, government agencies, and independent labs. Specifically, AI and LLMs are increasingly used in cybercrime forums [4] to extract cyber threat intelligence and support detection [5], and there have been initial efforts to understand content from opaque and not easily accessible sources [6]. Additionally, there have been efforts based on the collection and consolidation of CTIs [7]. These approaches are aligned with legislative efforts, including NIS2 [8] and the resilience of critical entity directives [9] in the EU. Furthermore, the market currently offers real-time CTI info from the dark web, including the monitoring of attack surfaces, threat intelligence feeds, and insights into dark web activities. The present research aims to enhance existing analyst-driven intelligence strategies with an open, LLM-based methodology. Specifically, the use of open-source LLMs is employed to categorize posts from hacker forums into four distinct cybersecurity vulnerability groups. The posts are collected by the University of Arizona [10], and they are of an informal and unstructured format, involving security and other thematic areas. In this context, zero- and few-shot, as well as parameter-efficient fine-tuning strategies (LoRA), are evaluated in scenarios with limited resources and labelled data. The LLM behaviour is investigated in small, high-quality datasets where data collection is constrained by domain specificity, sensitivity, or access. This integration enhances our ability to analyze and categorize complex informal communications more accurately. Our methodology enables the scalable classification of such content into cybersecurity-relevant categories, supporting tasks like vulnerability tracking, trend mapping, or risk detection, contributing to the \"Identify\" function within the NIST cybersecurity framework [11], enhancing threat intelligence and improving the identification and categorization of cybersecurity threats. These advancements can yield actionable insights for future threat detection and prevention. Additionally, our work bridges cybersecurity and AI by advancing methodologies, which typically rely on pre-trained embedding models and cosine similarity metrics [12]. The main contributions of this work include the following:  
* **The novel classification of dark web resources for proactive security**: We introduce a novel, automatic classification mechanism involving posts from dark web hackers into specific cybersecurity categories identified by a domain expert.