## 3 Materials and Methods
### Dataset and Categories
#### 3.3.2 Selection of Categories and Post Categorization  
The cybersecurity vulnerabilities considered, as detailed in [35], focus on critical infrastructures and draw upon the following resources: (a) the NISTIR [36] hierarchical structure enriched with vulnerabilities from NESCOR electric sector failure scenarios, (b) the Common Vulnerabilities and Exposures (CVEs) Dictionary of publicly known vulnerabilities, (c) the National Vulnerability Database, and the (d) ISO/IEC 27005 risk management standard [37]. These vulnerabilities have different levels of granularity, informational depth, and levels of specificity. Vulnerability description examples include the \"System relies on communications that are easy to jam\", \"insufficient maintenance\", \"Weaknesses in Authentication Process or Authentication Keys\", and \"Use of weak SSL cipher and other cryptographic design flaws\".  
To establish a practical and manageable set of categories, we initially defined a broader set of ten categories, as depicted in Table 2. The initial expert-based classification of posts produced uneven results, and early experiments proved challenging for LLMs to handle this level of granularity (as presented in Table 2).  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Vulnerability Categories** & **Num of Distinct Posts** & **\% of Distinct Posts** \\ \hline SW and firmware flaws & 151 & 21.51\% \\ Availability protection and & 228 & 32.48\% \\ security by design mechanisms & 113 & 16.10\% \\ Access control and management & 123 & 17.52\% \\ Not relevant & 18 & 2.56\% \\ Data integrity mechanisms & 12 & 1.71\% \\ Patch management & 13 & 1.85\% \\ Security awareness and training & 14 & 1.99\% \\ Data encryption at rest, in motion, & 14 & 1.99\% \\ and in use & 11 & 1.57\% \\ Misconfiguration & 14 & 1.99\% \\ Governance, ICT security policies, & 11 & 1.57\% \\ and procedures & & \\ Security information/log & 10 & 1.42\% \\ management and monitoring & & \\ \hline
**Total** & **702** & **100\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The initial set of vulnerability categories.  
Considering the above, we focused on the first three vulnerability types, namely \"Software and firmware flaws\", \"Availability protection and security by design mechanisms\", and \"Access control and management\", which together account for most posts (approximately 70%). Including the \"Not relevant\" category brings the total coverage to approximately 88%. Consequently, our classification relies on these four categories.  
As presented in Figure 1, each selected post is categorized by the LLMs employing zero-shot, one-shot and three-shot prompt engineering, as well as fine-tuned LLM variants. The resulting inferences are then compared to those provided by a human expert.  
The manual assignment of posts to the categories (labelling) was performed empirically through the gradual filtering of possible categories. Mapping posts to the categories has not always been straightforward. Criteria included the content, the keywords included (as mapped according to dark web slang) and most importantly, the identification of the intent of the writer. The concept of confidence level was included in our experimentation, but it proved that it did not improve the accuracy of the LLMs. Proof by contradiction logic was also employed by an expert if assigning a post to other categories led to contradictions.  
The experiments, including both training and inference, were conducted on a predefined computational setup with limited resources, specifically using an NVIDIA A4500 GPU (manufactured by TSMC, Hsinchu, Taiwan) of 20 GB VRAM.