# Exploring CyBOK with Topic Modeling Techniques

Ana I. González-Tablas, Mohammed Rashed

► To cite this version:

Ana I. González-Tablas, Mohammed Rashed. Exploring CyBOK with Topic Modeling Techniques. 16th International Symposium on Human Aspects of Information Security and Assurance (HAISA), Jul 2022, Mytilene, Lesbos, Greece. pp.49-65, 10.1007/978-3-031-12172-2_5. hal-05324943

HAL Id: hal-05324943

https://inria.hal.science/hal-05324943v1

Submitted on 21 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Exploring CyBOK with Topic Modeling Techniques

Ana I. González-Tablas(✉) and Mohammed Rashed

Universidad Carlos III de Madrid, Leganés, Spain
{aigonzal,mrashed}@inf.uc3m.es

**Abstract.** Several frameworks that cover cyber security education and professional development have been introduced as a guidance for learners, educators and professionals to the different knowledge areas of the field. One of the most important frameworks is the Cyber Security Body of Knowledge (CyBOK). In this paper, we apply the BERTopic topic modeling technique to CyBOK. We aim, by using this technique, to identify the most relevant topics related to each CyBOK's knowledge area in an automated way. Our results indicate that it is possible to find a meaningful topic model describing CyBOK and, thus, suggests the possibility of applying related techniques to texts to identify their main themes.

**Keywords:** Natural language processing · NLP · Topic modeling · Cyber security framework · CyBOK · Cyber security body of knowledge

## 1 Introduction

As time passes, cyber security is more recognised as a field on its own. Hence, there is more focus on areas like creating specific standards, developing new educational programs and industrial certificates for cyber security among others. Yet, such recognition is just a step to fulfill the needs of the industry. As per Furnell et al. [11], the shortage in the number of qualified cyber security personnel has been alerted for several years now. One of the causes of such shortage is the ambiguity in the skills required for cyber security related job offers. Furnell [10] highlighted how such vacancies required security related certifications. However, in a single vacancy, recruiters may ask for certifications that cover distinct groups of skills and areas of knowledge.

This research was supported by the Ministerio de Ciencia, Innovación y Universidades (Grant No. PID2019-111429RB-C21), by the Region of Madrid grant CYNAMON-CM (P2018/TCS-4566), co-financed by European Structural Funds ESF and FEDER, and the Excellence Program EPUC3M17. The opinions, findings, conclusions, or recommendations expressed are those of the authors and do not necessarily reflect those of any of the funders.
To fill the gap between the knowledge required and the existing roles in cyber security in the job market, existing cyber security frameworks [1, 3, 7, 23, 24] may be used as a reference. Even though the list of underlying topics in these frameworks is not definitive, they establish a basis that designers of both academic programmes and professional certificates can depend on for preparing their course materials. Yet, as discussed by Furnell and Collins in [12], it might be difficult to understand how the different frameworks cover the wide spectrum of topics addressed in the cyber security field or to identify what can be expected when some material is said to be ‘about security’. Moreover, the analysis that can be undertaken to identify the specific topics needs to be done mainly by manual means nowadays [14].

Topic modeling techniques use Natural Language Processing (NLP) as a means to find hidden semantics in a collection of documents through clustering the themes as topics [17]. Within the NLP field, a topic is usually represented by a set of words. Hence, topic modeling helps identifying keywords that maybe used as a reference by educators as well as researchers and professionals to find out what the focus of any text is.

In this paper, we explore how topic modeling techniques can help identify the most significant topics in cyber security documents. Particularly, as the main contribution of this work and a proof-of-concept of the previous idea, we apply the BERTopic algorithm to CyBOK [24]. On one hand, BERTopic [13] is a flexible and competitive topic modeling algorithm, that leverages modern pre-trained language models. On the other hand, CyBOK is already a clear reference in the field that counts with the CyBOK Mapping Framework<fnref n="1" />. This framework facilitates finding the CyBOK knowledge areas related to a document by making use of an extensive keyword list. By no way our method aims at replacing the method in the CyBOK Mapping Framework but paving the way to developing a complementary and automated method leveraging NLP techniques.

In summary, the main contributions of our work are:

- Exploring how topic modeling techniques can be used to identify the main topics in CyBOK; specifically through applying BERTopic [13].

- Paving the way to a complementary and automated method for determining the knowledge area of a text in addition to the CyBOK Mapping Framework.

The paper is organized as follows. In Sect. 2, we address related work. Next, in Sect. 3, we discuss the focus of this paper and explain the system design along with the experiments. Then, Sect. 4 addresses the results of the experiments. Afterwards, we extensively discuss the results in Sect. 5. Finally, we conclude our paper and address future work in Sect. 6.

## 2 Related Work

Because our work is related to several areas, we split this section into three parts covering the areas of *topic modeling*, *cyber security frameworks* and *text mining in cyber security*.

<footnote n="1">¹ https://www.cybok.org/resources/.</footnote>
## 2.1 Topic Modeling

It is a class of unsupervised learning techniques that, given a collection of documents, analyses the latent topics that appear in the corpus and best identify each of the given documents; within this context, a topic is usually represented by a set of words carefully selected by leveraging statistics and word semantics. Some techniques return a weight that represents the relevance of each word in the topic. Good models have coherent and interpretable topics with a small number of overlapping words [8]. Topic modeling is mainly used to understand underlying themes in text collections within varied fields. However, it can be combined with another machine learning technique to make possible or enhance the results of other text mining tasks such as document classification.

Reference to topic modeling goes back to 1990 [9]. Deerwester et al. treated the unreliability emerging from term-document association data as a statistical problem. Such approach, in which they assumed that data had an underlying latent semantic structure, helped them to resolve term-matching retrieval deficiencies. To estimate the structure, which is hidden partially because of the random word choice with regards to retrieval, they used a statistical analysis known as Latent Semantic Indexing (LSI). In this analysis, which leverages singular-value decomposition, a large matrix is constructed and includes term-document association data for building 'semantic space'. The more the related the documents and terms, the closer they are to each other in the space. Afterwards, Hofmann introduced Probabilistic LSI (PLSI) [15], largely influenced by [9], which is considered to be the first genuine topic model [18].

Later, the Dirichlet Multinomial Mixture (DMM) and the Latent Dirichlet Allocation (LDA) [6] topic models were proposed to enhance the accuracy of LSI and PLSI [8]. Since then, many variants were introduced. These variants aim at improving the capabilities to model the evolution of topics and to process modern text formats, while others focus on dealing with noise. We refer interested readers to the in-depth analysis on topic modeling by Churchill and Singh in [8].

State-of-the-art models mix traditional topic models with NLP methods by leveraging prior knowledge on natural language. An NLP technique that stands out is word embeddings which represents words as real-value vectors; words that are semantically similar are close to each other in the vector space. This technique can be used also to represent phrases or documents in the same semantic space. State-of-the-art embeddings are even capable of word sense disambiguation and providing contextually-meaningful word embeddings, like BERT and GPT-2 [26]. Two recent topic modeling algorithms that incorporate contextual-word embeddings are Top2Vec<fnref n="2" /> [5] and BERTopic<fnref n="3" /> [13].

## 2.2 Cyber Security Frameworks

In the last decades several cyber security frameworks have been published by public and private organizations worldwide. One of the early frameworks

<footnote n="2">² https://github.com/ddangelov/Top2Vec.</footnote>

<footnote n="3">³ https://maartengr.github.io/BERTopic/index.html.</footnote>
was 2017’s *Cybersecurity Curricula-Curriculum Guidelines for Post-Secondary Degree Programs in Cybersecurity* [1] as an output of the Joint Task Force (JTF) formed collaboratively in 2015 by four major international computing societies. The publication aimed to support academic institutions with curricular guidance through a “comprehensive view of the cybersecurity field, the specific demands of the base discipline, and the relationship between the curriculum and cybersecurity workforce frameworks”. Other frameworks include Skills Framework [7], Cyber security Body of Knowledge (CyBOK) [24], Common Body of Knowledge (CBK) [3], National Initiative for Cybersecurity Education (NICE) Cybersecurity Workforce Framework [23], among others. Frameworks’ origins span industry and academic communities. Each framework has its own perspective of the break-down of cyber security topics as per Furnell [11] to which we refer readers to for more detailed discussion on the topic. Among those frameworks, we chose to explore CyBOK because of its availability and the extensiveness of its text.

## 2.3 Text Mining in Cyber Security

In their extensive literature review on text mining in cyber security, Ignaczak et al. [16] divided the type of content analyzed into 16 categories, with electronic documents being the top ranked one. The top 5 categories include social networks, online forums, web pages and security databases. Also, they divided the tasks into 5 different areas including text supervised-learning classification, text clustering, information extraction, topic modeling and finally sentiment analysis. With regards to topic modeling, they consider that it is beneficial for:

- Cyber Threat Intelligence (CTI) as it is capable of identifying the main topics from online communities, and

- Data Leak Prevention (DLP) because it is able to determine if a document’s main topics are connected to confidential subjects.

Sundarkumar et al. [25] used topic modeling for malware detection through leveraging the API call sequences’ types, while Neuhaus et al. [22] used it on the vulnerability reports found in Common Vulnerability and Exposures (CVE) database to find the dominant vulnerabilities and the new vulnerability trends. We aim to extend the use of topic modeling to cyber security education by applying such technique to natural language based texts.

# 3 Materials and Methods

We reviewed the topic modeling literature and tools to find the algorithms that best suit our objective. We selected Top2Vec and BERTopic, as both use contextual-word embeddings and then apply dimensionality reduction and clustering techniques. After preliminary testing, we opted for BERTopic as we obtained better results with this algorithm. With the recommended configuration, BERTopic returned around 50 topics, most of them meaningful in our context, while Top2Vec only returned 2 or 3 topics.
## 3.1 BERTopic

BERTopic algorithm [13] considers three phases:

1. *Document embedding.* BERTopic uses a pre-trained BERT model by default although any other embedding model can be used as well.

2. *Document clusterization.* First, UMAP algorithm [20] is used to reduce the word embeddings' dimensionality. Then, HDBSCAN algorithm [19] is applied to cluster the reduced document embeddings in semantically similar document clusters.

3. *Topic extraction.* Each previously identified cluster is understood as a single topic. Topic representations are modeled from the reduced document embeddings in that cluster. To find the words that best represent each topic, TF-IDF is used within the cluster, referred to as c-TF-IDF by the author.

One limitation of BERTopic is that it only assigns one topic per document. However, we highlight that HDBSCAN is a soft clustering technique. Thus, although it assigns each document to a single topic, it also estimates the probability of a document being associated to all identified topics. Hence, the obtained probability matrix can be used to estimate the distribution of topics in a document.

## 3.2 Corpus Generation

Our corpus is composed of documents that contain the text paragraphs found in CyBOK v1.1.0. In order to build the corpus, we downloaded each of the 22 chapters of CyBOK v1.1.0 from its web page<fnref n="4" /> in PDF format and extracted the text contained in each PDF file using text processing tools. Microsoft Word and Office 365 have been used to open PDF files and save them as UTF-8 text; the result is a text file without headers or footers.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Doc length distribution per chapter</figcaption></figure>

<footnote n="4">⁴ https://www.cybok.org/knowledgebase1.1/.</footnote>
<table><caption>**Table 1.** Corpus. KA refers to Knowledge Area</caption><thead><tr><td>Number</td><td>Title of KA</td><td>Version</td><td>Document count</td></tr></thead><tbody><tr><td>1</td><td>Introduction to CyBOK</td><td>1.1.0</td><td>90</td></tr><tr><td>2</td><td>Risk Management &amp; Governance</td><td>1.1.1</td><td>292</td></tr><tr><td>3</td><td>Law &amp; Regulation</td><td>1.0.2</td><td>948</td></tr><tr><td>4</td><td>Human Factors</td><td>1.0.1</td><td>157</td></tr><tr><td>5</td><td>Privacy &amp; Online Rights</td><td>1.0.2</td><td>161</td></tr><tr><td>6</td><td>Malware &amp; Attack Technologies</td><td>1.0.1</td><td>124</td></tr><tr><td>7</td><td>Adversarial Behaviours</td><td>1.0.1</td><td>169</td></tr><tr><td>8</td><td>Security Operations &amp; Incident Management</td><td>1.0.2</td><td>260</td></tr><tr><td>9</td><td>Forensics</td><td>1.0.1</td><td>237</td></tr><tr><td>10</td><td>Cryptography</td><td>1.0.1</td><td>350</td></tr><tr><td>11</td><td>Operating Systems &amp; Virtualisation Security</td><td>1.0.1</td><td>164</td></tr><tr><td>12</td><td>Distributed Systems Security</td><td>1.0.1</td><td>223</td></tr><tr><td>13</td><td>Formal Methods for Security</td><td>1.0.0</td><td>243</td></tr><tr><td>14</td><td>Authentication, Authorisation &amp; Accountability</td><td>1.0.2</td><td>253</td></tr><tr><td>15</td><td>Software Security</td><td>1.0.1</td><td>182</td></tr><tr><td>16</td><td>Web &amp; Mobile Security</td><td>1.0.1</td><td>212</td></tr><tr><td>17</td><td>Secure Software Lifecycle</td><td>1.0.2</td><td>152</td></tr><tr><td>18</td><td>Applied Cryptography</td><td>1.0.0</td><td>341</td></tr><tr><td>19</td><td>Network Security</td><td>2.0.0</td><td>209</td></tr><tr><td>20</td><td>Hardware Security</td><td>1.0.1</td><td>170</td></tr><tr><td>21</td><td>Cyber Physical Systems</td><td>1.0.1</td><td>202</td></tr><tr><td>22</td><td>Physical Layer and Telecom Security</td><td>1.0.1</td><td>153</td></tr></tbody></table>

The text is then preprocessed so only the meaningful paragraphs remain. Preprocessing is mainly done manually using text processing tools combined with some Python scripts.

First, some sections are removed manually (cover, copyright, changelog, cross-reference of topics vs reference material, further reading, notes —except for the Law & Regulation chapter—, references, acronyms, glossary and index). Then, text retrieved from figures and tables is also manually identified and removed. Lines containing only a list of references<fnref n="5" /> are removed as well.

After training the topic model with the cleaned corpus, we noticed that titles of sections *Introduction* and *Conclusions* introduced too much noise (a specific topic was identified for these documents). Hence, we removed them as well. For similar reasons, we removed the note numbers in the *Law & Regulation* chapter. The resulting corpus contains 5292 documents, each document being a paragraph

<footnote n="5">⁵ E.g. line 3 in page 4 of *Cryptography* chapter is literally '[3, c8-c9, App B] [4, c1-c5]'.</footnote>
from the original chapters. Table 1 specifies the contents of each chapter and the
final number of documents in each of them, while Fig. 1 shows the document
length distribution per chapter.

3.3 Topic Modeling with BERTopic

BERTopic is quite a flexible algorithm that embeds several algorithms. Each of
these algorithms has a set of parameters, and selecting one or another value for
them will influence the results. Next, we describe the values used for the main
parameters. Any not mentioned parameter has been set with the default values.

We used the *all-MiniLM-L6-v2*<sup>6</sup> embedding model for english language
included in the BERT Sentence Transformer framework. It is a general-purpose
model trained on a large and diverse dataset which presents a faster performance
than other models in the framework while still offering a good quality.

Regarding topic extraction, we set the number of words to extract per topic<sup>7</sup> to 10 and the minimum topic size<sup>8</sup> to 20. We used a customized vectorizer model so stop words are removed before finding topic representations with c-TF-IDF; we configured it to work with unigrams. Finally, we set the option that allows to calculate the probabilities of all topics per document to true.

All the experimentation has been done in Google Colaboratory, using the
Python notebook provided by the author of BERTopic<sup>9</sup> as a template.

4 Results

4.1 Topic Model Analysis

We found 51 topics in the built corpus plus the additional topic identified as -1,
which contains the documents that do not belong to any identified topic. The
list of topics, the number of documents assigned to them and their 10-words
representation are shown in Table 2.

We further leverage the BERTopic API as it offers a set of functions that helps
us visualize the obtained topic model and better understand it. Figure 2a shows
the intertopic distance map, in which all the 51 identified topics are represented
by a circle considering their semantic distance in a 2D space. Figure 2b shows
the dendrogram of the hierarchical clustering of the topics.

Besides, not only did we leverage BERTopic to train the model, but we also
used it to compute the probabilities of finding each topic in all documents.

<footnote n="6">⁶ https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2.</footnote>

<footnote n="7">7 BERTopic recommends a value between 10 and 20, but as we obtained similar results with both values, we chose the smaller one to preserve the topic model quality.</footnote>

<footnote n="8">⁸ This parameter establishes the minimum number of documents that must be found in a cluster to be recognized as a topic. The lower this value is, the higher the number of topics are. Default value is 10 but it obtained 107 topics while a value of 20 obtained around 50 topics, which we found more compact and easier to analyze.</footnote>

<footnote n="9">⁹ https://github.com/MaartenGr/BERTopic/blob/master/notebooks/BERTopic.ipynb.</footnote>
<table><caption>**Table 2.** List of topics found in CyBOK. First column contains the topic identification number. Second column contains the number of documents assigned to that topic. Third column contains the 10-word list representing the topic.</caption><thead><tr><th>Topic</th><th>Count</th><th>Top 10 words</th></tr></thead><tbody><tr><td>-1</td><td>1499</td><td>Security, systems, information, data, used, devices, attacks, cyber, control, network</td></tr><tr><td>0</td><td>268</td><td>Malware, criminals, malicious, victims, botnet, crime, online, criminal, money, operations</td></tr><tr><td>1</td><td>225</td><td>Forensic, evidence, data, file, digital, forensics, acquisition, artifacts, storage, files</td></tr><tr><td>2</td><td>168</td><td>Privacy, data, users, information, technologies, techniques, adversary, anonymous, tor, metadata</td></tr><tr><td>3</td><td>156</td><td>Distributed, coordination, consistency, resources, resource, services, service, cloud, consensus, systems</td></tr><tr><td>4</td><td>155</td><td>Properties, proofs, logic, verification, formal, model, methods, language, programs, trace</td></tr><tr><td>5</td><td>139</td><td>Encryption, key, adversary, scheme, block, cipher, algorithm, ae, rsa, pke</td></tr><tr><td>6</td><td>122</td><td>43, concepts, 41, methods, 42, 61, analysis, 31, 21, principles</td></tr><tr><td>7</td><td>114</td><td>Access, control, policies, policy, rights, user, users, capabilities, object, rbac</td></tr><tr><td>8</td><td>114</td><td>Cryptography, cryptographic, algorithms, libraries, developers, fhe, schemes, api, encryption, secure</td></tr><tr><td>9</td><td>113</td><td>Web, mobile, javascript, apps, browser, applications, html, application, security, browsers</td></tr><tr><td>10</td><td>111</td><td>Data, gdpr, personal, protection, law, processing, art, eu, subjects, european</td></tr><tr><td>11</td><td>111</td><td>State, international, law, jurisdiction, states, territory, persons, cyber, conflict, constitute</td></tr><tr><td>12</td><td>110</td><td>Key, keys, certificate, public, certificates, tls, protocol, parties, pki, exchange</td></tr><tr><td>13</td><td>109</td><td>Distance, signal, signals, receiver, attacker, wireless, spoofing, communication, prover, position</td></tr><tr><td>14</td><td>100</td><td>Detection, siem, alerts, anomaly, ids, alert, events, soim, sensors, network</td></tr><tr><td>15</td><td>99</td><td>Security, tasks, task, people, awareness, human, training, behaviour, organisation, employees</td></tr><tr><td>16</td><td>91</td><td>Risk, assessment, management, governance, impact, risks, cyber, outcomes, likely, likelihood</td></tr><tr><td>17</td><td>90</td><td>Copyright, patent, intellectual, trade, infringement, rights, trademark, patents, property, registered</td></tr><tr><td>18</td><td>85</td><td>Authentication, identity, user, authorisation, oauth, token, identities, saml, client, credentials</td></tr><tr><td>19</td><td>81</td><td>Network, networking, sdn, networks, lan, security, ntp, internet, layer, nfv</td></tr><tr><td>20</td><td>77</td><td>Mod, 01k1, k0, rm, keygen, 01k0, k1, bits, true, pks</td></tr><tr><td>21</td><td>75</td><td>Hardware, trusted, secure, attestation, abstraction, processor, software, design, level, trust</td></tr><tr><td>22</td><td>74</td><td>Cpss, control, cps, systems, attacks, physical, industrial, security, power, grid</td></tr><tr><td>23</td><td>62</td><td>Lifecycle, software, practices, development, secure, microsoft, organisation, security, processes, devops</td></tr><tr><td>24</td><td>57</td><td>Operating, systems, domains, security, ring, isolation, kernel, privileged, virtual, hypervisor</td></tr><tr><td>25</td><td>56</td><td>Passwords, password, unlock, authentication, users, biometric, people, device, patterns, features</td></tr><tr><td>26</td><td>56</td><td>Peers, p2p, routing, peer, attacks, overlay, sybil, discovery, eclipse, unstructured</td></tr><tr><td>27</td><td>50</td><td>Liability, tort, tortfeasor, victim, harm, tortious, duty, law, care, negligence</td></tr><tr><td>28</td><td>49</td><td>Signature, signatures, scheme, group, message, key, signing, edса, signer, schnorr</td></tr><tr><td>29</td><td>47</td><td>Memory, address, page, kernel, code, pointer, program, operating, buffer, pages</td></tr><tr><td>30</td><td>46</td><td>Sidechannel, sidechannels, attacks, power, hardware, timing, information, leakage, channel, attacker</td></tr><tr><td>31</td><td>39</td><td>Hash, function, functions, sponge, output, length, construction, mac, oracle, sha3</td></tr><tr><td>32</td><td>38</td><td>Vulnerabilities, software, implementation, objective, categories, coding, vulnerability, programming, prevention, topic</td></tr><tr><td>33</td><td>38</td><td>Program, analysis, static, values, fuzzing, tools, vulnerabilities, execution, technique, taint</td></tr><tr><td>34</td><td>36</td><td>Law, legal, legislation, states, civil, laws, authority, secondary, study, legislative</td></tr><tr><td>35</td><td>35</td><td>Sql, output, injection, structured, input, query, database, statements, generation, command</td></tr><tr><td>36</td><td>34</td><td>Content, origins, qa, shell, cap, notes, pk, experience, authors, world</td></tr><tr><td>37</td><td>33</td><td>Verifier, prover, zeroknowledge, protocol, run, proof, public, apparently, noninteractive, simulator</td></tr><tr><td>38</td><td>33</td><td>Crimes, act, computer, crime, uk, criminal, 1990, 201340, directive, misuse</td></tr><tr><td>39</td><td>32</td><td>Incident, incidents, response, management, organisation, plan, procedures, plans, followup, handle</td></tr><tr><td>40</td><td>32</td><td>Ip, ddos, bgp, traffic, nat, syn, attacks, dos, address, icmp</td></tr><tr><td>41</td><td>29</td><td>Disclosure, vulnerability, disclosures, disclosing, vendor, responsible, finders, practitioners, upstream, benefits</td></tr><tr><td>42</td><td>25</td><td>Warranty, contract, contractual, goods, warranties, exclusions, contracts, liability, quality, vendor</td></tr><tr><td>43</td><td>25</td><td>Syslog, logs, log, format, events, generating, logging, 26, text, applications</td></tr><tr><td>44</td><td>23</td><td>Cybok, knowledge, area, kas, management, security, incident, cyber, human, areas</td></tr><tr><td>45</td><td>23</td><td>Interception, providers, communications, lawful, service, state, states, legal, intercepting, public</td></tr><tr><td>46</td><td>22</td><td>Network, netflow, packet, ppac, packets, monitoring, traffic, ip, interface, capture</td></tr><tr><td>47</td><td>22</td><td>Attack, mitigation, represents, attacker, considered, attacks, proactive, control, actuators, controller</td></tr><tr><td>48</td><td>22</td><td>Design, bullet, security, open, requirements, 211, obscurity, controls, usable, section</td></tr><tr><td>49</td><td>21</td><td>Entropy, random, puf, pufs, numbers, circuits, prng, generator, number, noise</td></tr><tr><td>50</td><td>21</td><td>Logging, log, accountability, logs, audit, policies, collected, evidence, entries, tamper</td></tr></tbody></table>
Recall that BERTopic uses HDBSCAN to compute the clusters (i.e. our topics). Although each document is assigned to a single topic, HDBSCAN can also estimate the probability of a document belonging to each cluster (i.e. the probability of finding a topic in a document). E.g, if we had a certain document with probability values of 0.7 and 0.6 for topics 5 and 20, and very low probability values for the rest of the topics, we could understand that topics 5 and 20 are found in the document, but not the rest of them.

Figure 2c shows the distribution of the probabilities per topic for all the documents in the corpus without outliers and Fig. 2d shows them with outliers. The connection between the topics and the chapter in which they show high probability is generally comprehensible from a cyber security professional’s point of view. We elaborate on this part more in the next section.

To analyze how specific the identified topics are with respect to the documents in the corpus, we count how many topics are found with a probability greater than a certain threshold for each document. Figure 2e presents the results considering a probability $p > 0.05$. Finally, to better grasp the number of documents that are related to each topic, we count how many of them present a probability greater than a different larger threshold. Figure 2f presents the results considering a probability $p > 0.5$.

## 4.2 Topic Model vs CyBOK Chapters

After analysing the topic model, we wanted to get a deeper insight on how the topics were represented in each chapter. Therefore, we depicted the probabilities of finding a topic in a document per chapter. Results are presented in Figs. 3, 4 and 5; we split the chapters’ probabilities into 3 figures for better readability.

# 5 Discussion

## 5.1 Topic Model Analysis

Reading through the terms representing each topic (Table 2), we found that most extracted topics seem coherent and make sense in the context of CyBOK. Topics 6 and 20 together with topic 36 are the exception; they are represented with text left from mathematical equations (topic 6), numbers (possibly originating from the section numbering) and terms that appear in a significant number of section titles or general terms (topics 20 and 36). We believe that a deeper preprocessing of the texts used to build the corpus would generate better results.

Focusing on the topic model structure (Figs. 2a and 2b), we find that some of the groupings (clades) make more sense than others from the point of view of a cybersecurity-knowledgeable person. Some groupings appear to be semantically correct, like {18, 25, 5, 8, 20, 12, 28} and {11, 34, 17, 42}. Others, like {9, 49, 29, 31, 35} and {30, 45, 37, 47, 13, 14}, appear to mix topics that are not understood as semantically close from a cyber security perspective (e.g. topic 35: {sql, output, injection, structured, input, query,
<figure><img src="image_4.png" /><figcaption>(a) Intertopic distance map</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(c) Probability distribution of finding a topic in a document for the whole corpus (without outliers)</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(e) Histogram of the number of topics with $p > 0.05$ found per document</figcaption></figure>

Hierarchical Clustering

<figure><img src="image_7.png" /><figcaption>(b) Hierarchical clustering</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(d) Probability distribution of finding a topic in a document for the whole corpus (with outliers)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(f) Number of documents with a probability $p$ of finding a topic in them $p > 0.5$</figcaption></figure>

Fig. 2. Topic model structure and topic probabilities
<figure><img src="image_10.png" /><figcaption>Fig. 3. Probability distribution of finding a topic in a document per chapter (I)</figcaption></figure>
<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>Fig. 4. Probability distribution of finding a topic in a document per chapter (II)</figcaption></figure>
<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /><figcaption>Fig. 5. Probability distribution of finding a topic in a document per chapter (III)</figcaption></figure>

database, statements, generation, command} and topic 31: {hash, function, functions, sponge, output, length, construction, mac, oracle, sha3}). However, we point out the fact that the clustering has been computed according to the semantic representation of the topics, which has been derived with the support of a general-purpose pre-trained language model. Using a model specifically trained on a large and varied cyber security corpus would for sure output a different hierarchical clustering, probably more coherent from a cyber security point of view.

Regarding the distribution of the probabilities without outliers among the whole corpus (Fig. 2c), we observe that most probabilities are very low (with a mean value of around 0.006), presenting a very similar distribution for all chapters. With regards to the distribution of the outliers as shown in Fig. 2d, we notice that there is a significant number of them and that their values are present in the full range for the majority of the topics. Outliers can be viewed as documents sufficiently related to that topic.

In Fig. 2e, we find that most documents have one or two topics that we might consider as related (notice that we have selected *p* > 0.05, which is a very low value but that will include only probabilities considered as outliers). Hence, we infer that the identified topics are very specific compared to the documents in the corpus, being this result coherent with the documents' lengths. From Fig. 2f, we can observe that only a small fraction of the whole corpus is related to a certain topic with a (significant) probability *p* > 0.5. As expected, the topics more prevalent in the corpus present higher number of documents, although in general, the number is quite homogeneous (around 20 documents).

## 5.2 Topic Model vs CyBOK Chapters

Analysing results shown in Figs. 3, 4 and 5, we find several cases:

a) Chapters that present a single predominant topic, like Chapters 5, 7 or 17.

b) Chapters that seem to have several frequent topics among their documents, like Chapters 3, 10, 15 and 18.

c) The very special case of Chapter 1.
The fact that chapters are related to one or more topics depends in part on how many topics have been identified related to the subject treated in a given chapter. Some subjects are very specific and the terminology is used consistently throughout all the chapter (e.g. Chapter 6 *Malware Attack & Defences* and Chapter 7 *Adversarial Behaviours* are closely related to topic 0: {malware, criminals, malicious, victims, botnet, crime, online, criminal, money, operations}). On the other hand, some subjects have been the source of several topics and the probabilities of their documents reflect this situation (like Chapter 3 *Law & Regulation* and Chapter 10 *Cryptography*, that relate respectively to several topics closely connected to each of the subjects).

Chapter 1 is a very special case as it is the introduction to the whole Cyber Security Body of Knowledge, which makes it reasonable that it is not closely related to any specific topic.

In general, the analysis of the topics related to each chapter seems reasonable and the only unexpected result is that the model is not capable of identifying topic 16: {risk, assessment, management, governance, impact, risks, cyber, outcomes, likely, likelihood} as related to documents in Chapter 2 *Risk Management & Governance*.

## 5.3 Possible Expansions

We understand our work as a first step in a research line that would benefit users of cyber security frameworks. From our point of view, we identify two possible expansions whose results may interest to the cyber security community and that we believe feasible to achieve in the short-term.

In first place, although we have applied a topic modeling technique to CyBOK, we believe that these techniques could be also applied, at the same time, to several frameworks like the ones discussed in Sect. 2.2. Such a model would yield a list of topics covered by all the frameworks and an estimation of its distribution along the frameworks. Hence, results could be used to compare the different topics covered by each framework in an automated way and would serve as support to go further in the line of some recent works such as [11, 14] and [12]. Such research would present additional challenges, as it is uncertain which topic modeling algorithms might provide good results when applied to several frameworks containing different text formats. To illustrate the differences, consider that CyBOK [24] is a book with proper paragraphs, while NICE [23] is mainly a list of really short descriptions of Knowledge, Skills and Abilities.

In second place, our results pave the way to develop a text classification model for cyber security materials with respect to CyBOK. Text classification is a supervised machine learning technique that yields a model that, once trained using a ground-truth labeled corpus, assigns labels to new documents. If there is no ground-truth labeled corpus (or it is not large enough), topic modeling techniques can be used to generate a labeled corpus, that can be used in turn as input in the training phase. Such a model would be really helpful in cases like the one described in [21], where the coverage of two cyber security programs respect
to CyBOK knowledge areas is analysed using CyBOK's knowledge trees and mapping framework. Notice that we believe that text classification models should be seen as a supporting tool and not as a substitution of current approaches.

## 5.4 Limitations

There are several limitations to our work. First, we believe that our technique in cleaning the text from noise is not optimal, which results in the appearance of noisy topics and keywords that do not seem relevant (e.g. topic **6** included among its top 10 words 43, 41, 42, 61, 31, 21). Also, the language model that we applied in the experiments causes some redundancy in the words that appear in the topics e.g. topic **47** has among its top 10 words: *attacker*, *attacks*, *attack*. Similarly, our topic model is not optimum as per [8] given that some topics have overlapping words, e.g. topics 5, 12, 28 include the word *key*. Finally, the language model is not a cyber security related one but rather a general-purpose pre-trained one.

# 6 Conclusion

In this paper, we applied topic modeling techniques using BERTopic's [13] Python library on CyBOK, one of the highly recognized cyber security frameworks. Our objective was to evaluate the possibility of identifying the most common topics for the CyBOK's knowledge areas in an automated way. Generally, the algorithm yielded acceptable results in which the most probable topics for each chapter made sense to a cybersecurity-knowledgeable person in terms of the words that define each topic. Our results pave the way to several interesting extensions, as discussed in Sect. 5.3.

## 6.1 Future Work

We plan to optimize the model by further cleaning the input texts and to formally evaluate the model quality. Additionally, we aim to carry out similar experiments on other frameworks (see Sects. 1 and 2.2) and to compare the results to the ones we obtained in this work. Besides, we would like to evaluate how different topic modeling algorithms perform on individual frameworks. For instance, although we have discarded Top2Vec for this exploratory work, we believe better results can be achieved after a thorough research on possible configurations.

In the medium term, we would like to address the extensions discussed in Sect. 5.3. Finally, we believe that building an enhanced cyber security language model that extends to all cyber security fields and not just vulnerabilities, as is the case of CyBERT [4], would be a great contribution to the scientific community.
References

1. Curriculum Guidelines for Post-Secondary Degree Programs in Cybersecurity. https://www.acm.org/binaries/content/assets/education/curricula-recommendations/csec2017.pdf. Accessed 03 Apr 2022

2. ISO/IEC 27002:2013 Information technology—Security techniques—Code of practice for information security controls. https://www.iso.org/standard/54533.html. Accessed 03 Apr 2022

3. The (ISC)² CBK. https://www.isc2.org/Certifications/CBK. Accessed 03 Apr 2022

4. Ameri, K., Hempel, M., Sharif, H., Lopez, J., Jr., Perumalla, K.: CyBERT: cybersecurity claim classification by fine-tuning the BERT language model. J. Cybersecur. Priv. 1(4), 615–637 (2021)

5. Angelov, D.: Top2Vec: distributed representations of topics (2020)

6. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003)

7. Chartered Institute of Information Security: CIISec Skills Framework Version 2.4. https://www.ciiisec.org/CIISEC/Resources/Capability_Methodology/Skills_Framework/CIISEC/Resources/Skills_Framework.aspx. Accessed 03 Apr 2022

8. Churchill, R., Singh, L.: The evolution of topic modeling. ACM Comput. Surv. (CSUR) (2022). https://doi.org/10.1145/3507900

9. Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41(6), 391–407 (1990)

10. Furnell, S.: The cybersecurity workforce and skills. Comput. Secur. 100, 102080 (2021)

11. Furnell, S., Bishop, M.: Addressing cyber security skills: the spectrum, not the silo. Comput. Fraud Secur. 2020(2), 6–11 (2020)

12. Furnell, S., Collins, E.: Cyber security: what are we talking about? Comput. Fraud Secur. 2021(7), 6–11 (2021)

13. Grootendorst, M.: BERTopic: leveraging BERT and c-TF-IDF to create easily interpretable topics (2020). https://doi.org/10.5281/zenodo.4381785

14. Hallett, J., Larson, R., Rashid, A.: Mirror, mirror, on the wall: What are we teaching them all? Characterising the focus of cybersecurity curricular frameworks. In: 2018 USENIX Workshop on Advances in Security Education (ASE 2018) (2018)

15. Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learn. 42(1), 177–196 (2001)

16. Ignaczak, L., Goldschmidt, G., Costa, C.A.D., Righi, R.D.R.: Text mining in cybersecurity: a systematic literature review. ACM Comput. Surv. (CSUR) 54(7), 1–36 (2021)

17. Kherwa, P., Bansal, P.: Topic modeling: a comprehensive review. EAI Endor. Trans. Scalable Inf. Syst. 7(24), e2 (2020). https://doi.org/10.4108/eai.13-7-2018.159623

18. Liu, L., Tang, L., Dong, W., Yao, S., Zhou, W.: An overview of topic modeling and its current applications in bioinformatics. Springerplus 5(1), 1–22 (2016). https://doi.org/10.1186/s40064-016-3252-8

19. McInnes, L., Healy, J.: Accelerated hierarchical density based clustering. In: 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 33–42. IEEE (2017)

20. McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
21. Nautiyal, L., et al.: The United Kingdom's cyber security degree certification program: a cyber security body of knowledge case study. IEEE Secur. Priv. **20**(1), 87–95 (2022)

22. Neuhaus, S., Zimmermann, T.: Security trend analysis with CVE topic models. In: 2010 IEEE 21st International Symposium on Software Reliability Engineering, pp. 111–120. IEEE (2010)

23. Newhouse, W., Keith, S., Scribner, B., Witte, G.: National initiative for cybersecurity education (NICE) cybersecurity workforce framework. https://nvlpubs.nist.gov/nistpubs/specialpublications/nist.sp.800-181.pdf. Accessed 03 Apr 2022

24. Rashid, A., Chivers, H., Lupu, E., Martin, A., Schneder, S.: The cyber security body of knowledge version 1.1.0 (2021)

25. Sundarkumar, G.G., Ravi, V., Nwogu, I., Govindaraju, V.: Malware detection via API calls, topic models and machine learning. In: 2015 IEEE International Conference on Automation Science and Engineering (CASE), pp. 1212–1217. IEEE (2015)

26. Tripathy, J.K., Sethuraman, S.C., Cruz, M.V., Namburu, A., Mangalraj, P., Vijayakumar, V.: Comprehensive analysis of embeddings and pre-training in NLP. Comput. Sci. Rev. **42**, 100433 (2021)