which shows that our evaluation framework manages to autonomously describe malware families as accurately as a set of strict handcrafted Yara rules.

The *robustness phase* verified that the clustering was robust to noise insertion, having almost no impact on homogeneity and mean similarity, and only mild effect on completeness and silhouette scores due to creation of many singletons (c.f. figure 4.9, page 181). Indeed, the NMI score between the initial and final (highly polluted) clusterings of this phase was 0.974, indicating that information loss due to noise insertion was only marginal.

In addition, our evaluation framework produced a high rate of descriptive cluster prototypes that represent behaviors and can be used to scale up clustering, assist manual analysis or enhance classification models for malware detection.

As **main contributions of this chapter** (i) we revisited the foundations of malware analysis research, defining basic malware analysis primitives, to propose the *AnR paradigm* for reliable evaluation methodologies; (ii) we proposed ECDGs, as a compact call dependency graph that enables more efficient binary similarity computation; and (iii) we proposed a new similarity function for ECDGs that is efficient, accurate and robust.

Few other indirect contributions come from our concrete implementation, namely the study of symbolic execution to trace external calls, the evaluation of gSpan as a practical algorithm for sub-graph isomorphism, and the evaluation of cluster prototypes extraction to represent malware families.

Ultimately, our experiments show that $\sigma^{ECDG}$ can reliably work as cornerstone of multiple types of frameworks, from those who autonomously produce descriptions of malware families as accurately as manually created syntactical signatures to those who target malware search and malware clustering.
<figure><img src="image_59.png" /></figure>
# EMB-DUET: MULTI-F FEATURE
# CLUSTERING BASED ON NUMERICAL
# EMBEDDING

## 5.1 Introduction

Historically malware analysis heavily resorted to experts who manually create signatures for malware detection and classification. These signatures very often target syntactic properties of the files like text strings or regular expressions of byte code [1], which are susceptible to anti-analysis techniques that are effective to evade detection such as obfuscation, packing, polymorphism and metamorphism. Albeit largely used in practice for threat intelligence, malware triage, or other scenarios (e.g. identifying a ransomware family), *the primary short-coming of the signature-based methods is that they entail high precision but low recall* [1].

To work around this hindrance it is necessary to improve the efficacy of machine learning applications in malware analysis, which poses several major challenges. First, malware classification is a *undecidable problem*, which impairs the smooth usage of supervised learning as it depends on the available ground-truth. Second, the features must meaningfully describe the programs, while being at the same time accurate, robust and friendly to machine learning algorithms (e.g. feature vectors). Finally, due to the huge number of instances that need to be analyzed in practice, the whole workflow must be very scalable.

For a malware classification framework to be truly autonomous, it is ideal to take a *data-driven approach* where the analysis outcomes derive from data themselves instead of hing-ing on outsourced labels. Traditional signature-based methods are themselves mostly extrapolations of manually created signature, thus not scalable.

For the issue of feature selection, it can be propitious to base the classification on multi-view learning, as most studies on multi-view learning have often proven to be able to utilize unlabeled data effectively and improve classification accuracy [286]. A good alternative is
to use *hybrid features*, which are derived from a combination of dynamic and static analysis [68], because this:

(i) diversifies tools and techniques used to extract the features, thus making it much more complex for adversaries to bypass all the analysis layers;

(ii) provides multiple views of the dataset that can be interpreted as imperfect partitions which converge to the “true” one once combined.

To tackle the efficiency issue while remaining friendly to machine learning algorithms, our approach leverages the *numerical embedding* technique, which is particularly useful to analyze unstructured data that cannot be organized in predefined structures (e.g. text strings for sentiment analysis) [66]. We use numerical embedding to transform the unstructured data views into numerical vectors that can be efficiently analyzed by different algorithms, which enables to scale up clustering analysis for a large corpus of binary files.

To consolidate all individual views into a single clustering, we use methods of *ensemble clustering* [291]. They allow to improve the quality of the ensembles by combining them into a final unique clustering. Our solution blue print is similar to DUET by X. Hu and K. Shin [124], but we add a feature vectorization step —to compute numerical embedding vectors—that normalizes the input data and boosts the overall performance of the framework.

As main contributions, we:

(i) proposed EMB-DUET as a data-driven framework that is able to handle heterogeneous features, which leverages numerical embedding to boost overall performance [addresses RQ5.1];

(ii) verified the accuracy and robustness afforded by the aggregation of the different data views (i.e. strings, n-grams and e-ECDGs) [addresses RQ5.2];

(iii) analyzed the impact and effects of different hyperparameters involved in EMB-DUET and $\sigma^{ECDG}$-Learning with SVM Regression, so as to provide a guideline for proper parameter setting [addresses RQ5.3].

### 5.1.1 Research Questions

We address the top-level research question **TOP-RQ3: How to improve the arsenal of techniques currently used in malware analysis, in particular for the analysis of binary files?**

For this, we break down this backbone question into the following subordinate research questions:
**RQ5.1** *How to design a data-driven clustering that can systematically work with heterogeneous features?*

**RQ5.2** *Which set of features can afford accuracy and robustness to the data-driven clustering?*

**RQ5.3** *How to optimize the data-driven clustering to make it scalable?*

**5.1.2 Chapter Outline**

The remainder of this chapter is organized as follows:

- Section 5.2 (page 191) presents the specific background of this chapter;

- Section 5.3 (page 204) presents the related works;

- Section 5.4 (page 208) presents our methodology, which introduces $\sigma^{\text{ECDG}}$-learning with SVM Regression and EMB-DUET;

- Section 5.5 (page 212) presents the experimental setup and the evaluation results.

- Section 5.6 (page 252) discusses results.

- Section 5.7 (page 256) concludes.

**5.2 Background**

**5.2.1 Support-Vector Machines (SVMs)**

Support-Vector Machines constitute a family of prediction methods that can be used for supervised classification as well as regression analysis (case referred as *support-vector regression*).

In its simplest form (i.e. as a linear classifier), SVM can be seen as an algorithm that takes a set of data points within a *n*-dimensions hyperspace and finds a (*n* − 1)-dimensional hyperplane that separates both sides of the hyperspace. For this, the algorithm finds a function that approximates the data points within a precision margin, where misclassification on the learning data are tolerated in order to optimize the trade-off between bias and variance error.

More formally, given the training data $C_1 = \{(x_1, y_1), \cdots, (x_n, y_n)\} \subset X \times \mathbb{R}$ (data points belonging to class 1) and $C_{-1} = \{(x'_1, y'_1), \cdots, (x'_m, y'_m)\} \subset X \times \mathbb{R}$ (data points belonging to class -1), where X denotes the input space, the goal is to find $f(x)$ such that the deviation is greater than $\epsilon$ to any $y_i \in \{y_1, \cdots, y_n\} \cup \{y'_1, \cdots, y'_m\}$. In case of linear functions, $f(x)$ can be defined as:
<figure><img src="image_60.png" /><figcaption>Figure 5.1 – Maximum-margin hyperplane and margins computed in SVM.</figcaption></figure>

$$f(x) = \langle w, x \rangle + b, w \in X, b \in \mathbb{R}$$

where $\langle \cdot, \cdot \rangle$ denotes the dot product in $X$ and $w$ is a normal vector to the hyperplane. The data points can be normalized so as to have the following equation to describe the hyperplanes (as shown in figure 5.1):

$$\langle w, x \rangle + b = 1 \qquad (\text{margin for class 1})$$

$$\langle w, x \rangle + b = 0 \qquad (\text{maximum-margin hyperplane})$$

$$\langle w, x \rangle + b = -1 \qquad (\text{margin for class -1})$$

Figure 5.1 shows an example of SVM computation, where the area in blue indicates the region above the hyperplane $\langle w, x \rangle + b = 1$ (i.e. margin of class 1), the area in red indicates the region below the hyperplane $\langle w, x \rangle + b = -1$ (i.e. margin of class -1), and the area in yellow indicates the region between the margins where misclassification is tolerated (without punishment).

It is possible to demonstrate that the distance between the margins in this normalized formulation is $2/||w||$ (c.f. appendix section 6.8.1, page 344). Since the goal of SVM is to maximize the distance between the margins, the SVM problem can be rewritten as a convex optimization problem as follows:
$$ \text{minimize} \quad \frac{1}{2} \|w\|^2 $$

$$ \text{subject to} \quad \begin{cases} y_i - \langle w, x_i \rangle - b \ge \epsilon \\ \langle w, x_i \rangle + b - y_i \ge \epsilon \end{cases} $$

If the training data (of the example) is linearly separable, then the convex optimization problem is feasible. Otherwise, the slack variables $\xi_i, \xi_i^*$ can be introduced to cope with the optimization problem constraints, which corresponds to Vapnik's formulation of the soft margin:

$$ \text{minimize} \quad \frac{1}{2} \|w\|^2 + C \sum_{i=1}^{l} (\xi_i + \xi_i^*) $$

$$ \text{subject to} \quad \begin{cases} y_i - \langle w, x_i \rangle - b \ge \epsilon + \xi_i \\ \langle w, x_i \rangle + b - y_i \ge \epsilon + \xi_i^* \\ \xi_i, \xi_i^* \ge 0 \end{cases} $$

where the constant $C > 0$ determines the trade-off between the flatness of the maximum-margin hyperplane and the total deviation larger than $\epsilon$ that is tolerated.

It is possible to show that finding a solution for this optimization problem depends solely on the dot product of data point pairs [142] (which goes beyond the scope of this thesis). Therefore, given a transformation $k: X \times X \to \mathbb{R}$ whose result is equal to the inner product in another space $V$ (not necessarily linear), it is possible compute the SVM in this new space $V$ without explicitly recomputing the coordinates of the data points in there —which is known as the *kernel trick*, as $k$ is often referred to as the kernel function.

More formally, given the transformation $\phi: X \to V$ of vectors in $X$ onto $V$ the only requirement on $k$ for applying the kernel trick is:

$$ k(x_i, x_j) = \langle \phi(x_i), \phi(x_j) \rangle_V $$

This allows the SVM algorithm to efficiently fit the maximum-margin hyperplane in a transformed feature space, which may be nonlinear and high-dimensional (as depicted in figure 5.2).

Some common kernels include:
<table><tr><td>k(x<sub>i</sub>, x<sub>j</sub>) = &lt;x<sub>i</sub>, x<sub>j</sub>&gt;</td><td>Linear</td></tr><tr><td>k(x<sub>i</sub>, x<sub>j</sub>) = (&lt;x<sub>i</sub>, x<sub>j</sub>&gt; + r)<sup>d</sup></td><td>Polynomial</td></tr><tr><td>k(x<sub>i</sub>, x<sub>j</sub>) = e<sup>-γ||x<sub>i</sub>-x<sub>j</sub>||<sup>2</sup></sup></td><td>Radial Basis Function (RBF)</td></tr><tr><td>k(x<sub>i</sub>, x<sub>j</sub>) = tanh(k&lt;x<sub>i</sub>, x<sub>j</sub>&gt; + c)</td><td>Sigmoid</td></tr></table>

<figure><img src="image_61.png" /><figcaption>Figure 5.2 – Maximum-margin hyperplane and margins computation using the kernel trick<fnref n="1" />.</figcaption></figure>

The case of support-vector regression is almost analogous to the problem exposed above; the difference is that in classification the goal is to find the maximal margins that separate the data points (setting the maximum-margin hyperplane halfway between these margins), whereas in regression the goal is to find the maximal margins such that most of the data points are inside the region comprised between them (again, setting the maximum-margin hyperplane halfway between the margins).

Following this analogy, the support-vector regression problem can be rewritten as a convex optimization problem as follows:

$$ \begin{array}{ll} \text{minimize} & \frac{1}{2} \|w\|^2 + C \sum_{i=1}^{l} (\xi_i + \xi_i^*) \\ \text{subject to} & \begin{cases} y_i - \langle w, x_i \rangle - b \le \epsilon + \xi_i \\ \langle w, x_i \rangle + b - y_i \le \epsilon + \xi_i^* \\ \xi_i, \xi_i^* \ge 0 \end{cases} \end{array} $$

For a given data point P in X, its regression corresponds to the projection of P on the

<footnote n="1">1. Adapted from https://commons.wikimedia.org/wiki/File:Kernel_Machine.svg.</footnote>
maximum-margin hyperplane defined by solving the convex optimization problem above.

Similarly to the classification case, it is possible to use the kernel trick to compute regression in a non-linear space and also to choose parameters that act on the trade-off between the flatness of the maximum-margin hyperplane and the total deviation beyond $\epsilon$, which makes SVM a powerful regression method. For greater details on SVM regression refer to the tutorial by AJ Smola and B Schölkopf [265].

## 5.2.2 Embedding Learning

Numerical embedding is an essential part of modern data analysis, specially used to analyze unstructured data that cannot be organized in predefined structures, such as texts, graphs, videos, objects and chemical compounds [66]. The goal is to map unstructured data onto lower-dimensional numerical vectors that are able to retain key properties of the original data, in opposition to traditional methods (e.g. one-hot encoding), which do not inherent any relationship between the unities of the original data.

Once the numerical embedding for a given unstructured data is computed, any downstream analysis method can be applied. This is referred as the *two-stage method* and it has become a standard tool of machine learning [66].

More precisely, the idea of embedding learning is to reduce the dimension of a *predictor set* $\mathcal{S} = \{s_1, \cdots, s_n\}$ with an embedding $\mathcal{X} = \{\mathcal{X}(s)\}_{s \in \mathcal{S}}$, where $\mathcal{X}: \mathcal{S} \to \mathbb{R}^p$ and $p$ is a tuning parameter (usually) no greater than $|\mathcal{S}|$. It means that the embedding $\mathcal{X}$ works as a digest of $\mathcal{S}$, which provides numerical vectors (in $\mathbb{R}^p$) that are friendly to machine learning methods.

We say that $\mathcal{X}$ is an ideal embedding if it can represent $\mathcal{S}$ in the learning process without information loss. In this situation, $\mathcal{X}$ is said to be a *sufficient learning-adaptive embedding*. Formally, it is defined as [66]:

**Definition 5 (Sufficient learning-adaptive embedding)** An embedding $\mathcal{X} = \{\mathcal{X}(s)\}_{s \in \mathcal{S}}$ is learning-adaptive sufficient to predict the outcome of $Y$ if the conditional distribution of $Y$ given an embedding $\mathcal{X}(s) = \mathbf{x}$ does not depend on $S = s$, i.e. $\mathbb{P}(Y = y | S = s, f = \mathbf{x}) = \mathbb{P}(Y = y | f = \mathbf{x})$.

According to this definition, learning from a sufficient embedding is equivalent to learning from the original unstructured data. In practice, the superiority in terms of empirical performance for numerical embedding for some learning tasks is widely acknowledged over the traditional approach based on manually constructed features [66].
<figure><img src="image_62.png" /><figcaption>Figure 5.3 – Model architectures of CBOW and Skip-gram [191]</figcaption></figure>

embedding matrix of size $|V| \times p^2$ and $W$ is a weighted matrix of size $p \times |V|$ used to compute the word vectors for the context words$^3$. The vectors for the target word $w_t$ and its learning context are computed as follows:

$$ 
\begin{align*}
v_t &= o_t \cdot E \tag{target word} \\
v'_{t,j} &= (o_j \cdot E)^T \cdot W \quad j \in \{t-c, \dots, t+c\}, j \neq t \tag{context words}
\end{align*}
$$

where $o_t$ and $o_j$ are the one-hot encoded vectors for the target word and the context words.

Then, for each context word, the log-linear classification model softmax is used to obtain the posterior distribution of words in a vector $\hat{o}$ of size $|V|$. The training goal model is to maximize the average log probability [190], i.e. the following objective function:

<footnote n="2">2. $p$ is the embedding degree defined as parameter, which defines the embedding $\mathcal{X}: V \to \mathbb{R}^p$</footnote>

<footnote n="3">3. Two different vectors are computed for each word: one for the word when it is a target word and another when it is context word.</footnote>
<figure><img src="image_63.png" /><figcaption>Figure 5.4 – Detailed architecture of the Skip-gram model</figcaption></figure>

$$J_t(\theta) = \frac{1}{|V|} \sum_{t=1}^{|V|} \sum_{-c \le j \le c, j \ne 0} \log p(w_{t,j} = w_{O,j} | w_t) \quad (5.1)$$

where $w_{O,j}$ is the actual word present on the j-th context position, and $p(w_{t,j} = w_{O,j} | w_t)$ is computed using the softmax probability, i.e.:

$$p(w_{t,j} = w_{O,j} | w_t) = \frac{e^{(v'_{t,j} \cdot v_t)}}{\sum_{w'=1}^{|V|} e^{(v'_{w'} \cdot v_t)}} \quad (5.2)$$

## Word2Vec

Word2Vec builds on word embedding to achieve a simple, scalable and fast-to-train model. It was also proposed by Mikolov et al at Google in 2013 [190], basing on Skip-grams but with a new training method called *Negative Sampling*.
Negative sampling was proposed as a possible solution to simplify the maximization of the average log probability of the softmax (equations 5.1 and 5.2), because a naive classifier (e.g. logistic regression) would involve a computational cost proportional to $|V|$ (see equation 5.2). For this, the vector representations are simplified to approximately maximize the log probability of the softmax according to the following objective function:

$$J_t(\theta) = \log \sigma(v_{O,j}^\top \cdot v_t) + \sum_{k \sim P(w)} [\log \sigma(-v_k^\top \cdot v_t)] \quad (5.3)$$

The first term tries to maximize the log-probability for the actual words that lie in the context window, while the second term tries to iterate over random words $k$ that (with very high probability) do not lie in the context window, aiming to minimize the log-probability of the co-occurrence. The word sampling is based on the frequency of occurrence; Mikolov et al propose the distribution $P(w) = U(w)^{3/4}$—experimentally found, but not further explained —where $U(w)$ is a unigram (i.e. 1-gram) distribution.

Furthermore, to counter the imbalance between rare and frequent words, Mikolov et al propose a subsampling approach that discards each word $w_i$ in the training set with probability —again experimentally found, but not further explained:

$$1 - \sqrt{\frac{t}{f(w_i)}} \quad (5.4)$$

where $f(w_i)$ is the frequency of word $w_i$ and $t$ is a chosen threshold (typically ~ $10^{-5}$). This subsampling formula aggressively subsamples words whose frequency is greater than $t$ while preserving frequencies ranking.

Word2Vec trains the Skip-gram model using standard stochastic gradient descent and backpropagation [243], boosting it with the negative sampling and subsampling techniques just presented. For more details on CBOW, Skip-gram and Word2Vec refer to Xin Rong's paper on these topics [238].

## Doc2Vec

Doc2Vec is an extension of Word2Vec that works with text of variable-length (ranging from sentences to documents). This method was proposed by Q. Le and T. Mikolov at Google in 2014 [160], comprising of two models for *paragraph vectors*: a distributed memory model (PV-DM), which considers the ordering of words in a paragraph, and a distributed bag of (PV-DBOW), which ignores word ordering.
Both methods slightly modify Word2Vec to include a vector having the *paragraph id* —corresponding to the “document” terminology, as made popular by Gensim (a widely-used Python library for Machine Learning) [233]. Figure 5.5 shows the model architectures for PV-DM and PV-DBOW: the former method considers the concatenation of the paragraph vector with the word vectors to predict the target word, while the latter predicts words randomly sampled from the paragraph ignoring the context words in the input.

<figure><img src="image_64.png" /><figcaption>Figure 5.5 – Model architectures of PV-DM and PV-DBOW</figcaption></figure>

## Graph2Vec

Graph2Vec was proposed in 2017 by Narayanan *et al* as a method for numerical embedding computation of graphs inspired from Doc2Vec [200]. Graph2Vec was the first numerical embedding to learn the representations of entire graphs, instead of substructures such as nodes, paths and subgraphs.

Figure 5.6 shows the parallel between the models of Doc2Vec and Graph2Vec. In Graph2Vec graphs are considered analogical to documents Doc2Vec, but composed of rooted subgraphs
instead of words. Other graph substructures (e.g. nodes, walks, paths) could be used as
atomic entities instead of rooted subgraph, however the authors point two main advantages
to choosing rooted subgraphs:

* *higher order substructure:* rooted subgraphs encompass higher order neighborhoods than simpler substructures, therefore they are likely to better reflect the graph composition;

* *non-linear substructure:* rooted subgraphs capture better inherent non-linearity in graphs than linear substructures such as walks and paths.

Therefore, to train the model it is necessary to extract the rooted subgraphs and assign them with a unique label for all the rooted subgraphs in the vocabulary. This is accomplished using the Weisfeiler-Lehman (WL) relabeling strategy [74], typically used for testing graph isomorphism.

The WL relabeling strategy uses a coloring refinement method which assigns an initial color to all the nodes of the subgraph (of degree *d* defined as parameter) and at each iteration it aggregates a mixture of colors for each node depending on their neighbors. After *n* iterations (defined as parameter), the feature description for the rooted subgraphs is an ordered vector that counts the number of nodes for each color —if two rooted subgraphs are isomorphic they generate the same feature description. For greater details about Graph2Vec refer to the original paper [200].

<figure><img src="image_65.png" /><figcaption>Figure 5.6 – Parallel models of Doc2Vec and Graph2Vec [200]</figcaption></figure>

**5.2.3 Clustering Ensembles**

The most common approach for the machine learning setting assumes that features are represented in a single vector or graph space. These features are representation (i.e. *views*) of the objects (e.g. binary codes) in different feature spaces. Usually these multiple views are obtained from different vector spaces or different graph spaces or a combination of vector and graph spaces [286].
Unsupervised learning can rely on two strategies to handle data analysis with multiple views: centralized or distributed algorithms [169]. The former consolidates the multiple views into a unique representation before mining hidden patterns in the data. The latter learns hidden patterns individually from each view before learning the optimal hidden patterns of all the views.

Centralized algorithms are much more popular than distributed algorithms, despite the complexity of dealing with multiple representations that could not only be of different forms (e.g. vectors and graphs), but also have very different statistical properties [169]. Distributed algorithms have some advantages over centralized algorithms [169]:

- they do not constrain the individual (unsupervised) learning of the views, treating them as black boxes. This allows the system designer to chose the most appropriate method for each representation;

- they provide greater flexibility to the analysis system, since they do not directly operate on data.

Clustering ensembles (or *clustering aggregation*) is a method for distributed unsupervised learning. It combines multiple clustering models into a single consolidated partition without accessing neither the features nor the algorithms of the underlying partitions [273]. The goal is to produce a consolidated partition that improves the quality of individual clusters.

Cluster ensemble spun off from supervised classifiers that used a combination of different models for learning, after the success of this aggregation method to improve overall learning quality. However, in contrast to supervised learning, unsupervised learning has no clear-cut metric to optimize. Some authors tried to define a set of properties that endorse the use of clustering ensemble methods, but due to the malleable nature of unsupervised clustering no consensus emerged on this matter [291]. Some of these properties include:

- Robustness: The combination process must have better average performance than the single clustering algorithms.

- Consistency: The result of the combination should be somehow very similar to all combined single clustering algorithm results.

- Novelty: Cluster ensembles must allow finding solutions unattainable by single clustering algorithms.

- Stability: Results must have lower sensitivity to noise and outliers.
Vega-Pons and Ruiz-Shulcloper emphasize that it is not possible to claim that any partitioning of the data is (outright) better than another —including the natural organization (i.e. the ground-truth, if any) and the results obtained by a cluster ensemble [291]. In the case of clustering ensemble, it can only be ensured that the final consolidated cluster is a consensus of all the underlying partitions and thus it is expected that the consolidation process can produce a more reliable partition of data as it compensates for possible errors in the individual clusters.

The basic workflow of clustering ensemble methods consists of two steps: Generation and Consensus (c.f. figure 5.7). In the *generation* phase the set of individual partitions are computed; there are no particular constraints on how these partitions must be obtained —it may involve different algorithms, parametrizations, features and any other combination of those. In the *consensus* phase the individual partitions are consolidated in a single final clustering with the help of a *consensus function*, which is the main object of any clustering ensemble algorithm.

<figure><img src="image_66.png" /><figcaption>Figure 5.7 – General workflow of cluster ensemble methods [291].</figcaption></figure>

There are two main approaches for designing a consensus function: *objects co-occurrence* and *median partition*. The former considers the frequency that a data point belongs to one cluster or that pairs of data points appear together in a same cluster; the latter targets the partition that maximizes the similarity with all partitions in the cluster, i.e.:

$$P^* = \arg\max_{P \in \mathbb{P}_X} \sum_{j=1}^{m} \Gamma(P, P_j)$$

where $\mathbb{P} = \{P_1, \dots, P_m\}$ is the set of partitions (computed in the generation phase), $\mathbb{P}_X$ is the set of all possible partitions of the set of objects $X$ (such that $\mathbb{P} \subset \mathbb{P}_X$), $P^*$ denotes the consensus partition and $\Gamma$ is a similarity measure between partitions.
Some methods used for designing consensus functions based on objects co-occurrence
include *Relabeling and Voting*, *Co-association Matrix*, *Graph and Hypergraph* and several
other strategies; consensus functions based on median partition include *Genetic Algorithms*,
*Non-negative Matrix Factorization*, *Kernel Methods* and *Mirkin Distance*.

In this chapter we use the following consensus functions: cluster-based similarity parti-
tioning algorithm (CSPA), Hypergraph-Partitioning Algorithm (HGPA), Meta-Clustering Al-
gorithm (MCLA), Hybrid Bipartite Graph Formulation (HBGF), Nonnegative Matrix Factor-
ization (NMF). For a gentle introduction about these consensus functions, refer to survey
[291] by Vega-Pons and Ruiz-Shulcloper.

**5.3 Related Works**

In this chapter, our work explores the coalescence of *hybrid feature, numerical embed-
ding* and *ensemble learning* for malware classification. For this reason, this section reviews
works on malware analysis that fall in at least one of these areas of interest.

**5.3.1 Hybrid Features for Malware Analysis**

In malware analysis, features can be represented in many different ways (e.g. vectors
having header data, strings, CFGs or ECDGs), therefore the problem of handling multi-view
data arises naturally. Normally, features are derived from representative data (i.e. views) ob-
tained from a file analysis process and are subsequently used in a data analysis step in order
to fulfill the desired task (e.g. malware detection, malware clustering). These views are in
general software birthmarks<fnref n="4" /> extracted either through dynamic or static analysis methods.
Anderson *et al* refer to these extractors as *data sources* [6].

We call *hybrid features* the representations that entail a composition of data views, in par-
ticular when they are obtained through a mix of static and dynamic data sources [68]. These
data sources are in general respectively associated with syntactical and semantic features,
although it is possible to define features composed of multiple data views obtained from a
single data source (e.g. behavior profile [22]) —nonetheless they are not as relevant in our
context due to their shortcomings against targeted anti-analysis techniques.

The key aspect of this composition method is the fusion step, where data views extracted
from different data sources are composed to derive a (unified) hybrid feature. As many real-

<footnote n="4">4. Inherent characteristics that can be used to identify particular software [292].</footnote>
world applications, the most popular approach to obtain multi-views unsupervised learning
is through centralized algorithms. This includes approaches like feature concatenation [65,
95, 131, 179, 180, 194, 284], feature hashing [22, 125], Multiple Kernel Learning (MKL) [6],
feature composition [84] and metric composition [86, 137, 138, 206].

Feature concatenation is the simplest and most commonly used method for feature com-
position. It consists in producing a multi-view data where the multiple views collected from
different data sources are concatenated to produce a comprehensive description of objects.
However, it can be ineffective in practice and can even result in worse clustering results than
single-view clustering [328]. If the number of feature dimensions is overwhelmingly big,
techniques to shrink this representation have to be applied (e.g. PCA); Dahl *et al* use random
projections to map input vectors of hundreds of thousands dimensions into a feature space
of few thousand dimensions [65]. Although still providing good results in practice, such fea-
tures vectors are completely deprived of meaning over the original data views, which hinders
their interpretation.

Feature hashing (a.k.a *hashing-trick*) works by applying a hash function to the input data views and using the output value as feature vector [308]. It is a very fast and space-efficient strategy to vectorize features, however (similarly to feature concatenation) the feature vectors fail to keep a correspondence of representation with the original data views.

Multiple Kernel Learning (MKL) explores the fact that kernels can be linearly combined to obtain new flexible ones. This allows to define individual kernels for each data view and combine them into a single consolidated kernel that takes into account all sets of data views simultaneously. Combining models in kernel machine classifiers is thereby based on combining their kernels [162]. MKL is a very flexible method that can extend SVM methods to work with heterogeneous data, but it is not compatible with many other machine learning methods that do not rely on kernels.

Feature composition refers to combination of data views of different data sources that are directly fused into a feature representation. For instance, Eskandari *et al* define a feature which combines the information gathered by static and dynamic analyses into an enhanced version of the CFG, which is in turn analyzed using different classifiers such as decision trees, naive Bayes and random forest [84]. The utility of this approach is the derivation of a feature that “naturally” combines different data views, however it is applicable only for data views that have some level of innate kinship. This approach (alone) cannot provide the flexibility necessary to benefit from very different data views, therefore it is seldom used.

Metric composition defines custom distances that work with multiple data views. This
allows to define pairwise distances that can be used by many different machine learning al-
gorithms. The drawback of this method is its lack of flexibility, since the custom metric is
defined to work for the (limited) set of selected metrics of a given analysis scenario; more-
over, the topology of the feature space can become counter-intuitive, thus making it harder
to interpret the results.

Alternatively, feature composition methods based on distributed algorithms exist, but
they are much less popular than their centralized counterparts.

De la Rosa *et al* proposed a two-layered model composed of a meta-model and a set of
underlying specialized models, which are ranked with respect to their complexity [69]. Each
specialized model is trained with different features and the meta-model is trained to choose
the best (and simplest) specialized model in order to predict the final class for a given input.
The experiment shows that this strategy is able to speed up the overall classification, however
the authors point out that it is still the most complex (and most expensive) classifier that was
most frequently able to correctly assign malware into their corresponding families. This out-
come could be somehow expected because there is not cooperation between the underlying
specialized models to get the final classification. Therefore the meta-model can be primary
viewed as a speed-up optimization, but it fails to attain an actual learning composition of
the models.

Most commonly, frameworks based on distributed algorithms use standard ensemble
learning. Supervised models generally use voting and stacking ensemble, which consists in
a two-level model where the first level contains the specialized sub-models and the second
level contains a sub-model that produces a prediction based on the result of the first-level
sub-models. These are the most frequent distributed methods used (together or not) to com-
bine heterogeneous data views in enhanced malware detection models [15, 111, 121, 149,
252, 317].

As for unsupervised models, Ye *et al* generate the final clustering from the optimal con-
nectivity matrix, which computes the co-association matrix of the samples in each parti-
tion and use a minimum threshold (with sample-level constraints) to select the samples of
the aggregation clustering [320]. X. Hu and K. Shin proposed the DUEL as a framework for
combining static and dynamic features through clustering ensembles [124]; they defined
cluster-quality measures and also produced the final clustering based on optimal connec-
tivity matrix using three different algorithms. Zhang *et al* defined a mixture model scheme
based on clustering ensemble, which assumes that the data views are modeled as random
variables drawn from a probability distribution described as a mixture of multivariate com-
ponent densities [326].

5.3.2 Numerical Embedding for Malware Analysis

The use of numerical embedding for malware analysis is relatively recent. Inspired by
concepts from natural language processing, Rieck et al were the first to propose the use of
numerical embedding for malware analysis in 2011 [235].

They defined a special representation of behavior denoted as *malware instruction set (MIST)*, in which the n-gram decomposition are transformed into numerical vectors. Roughly, the numerical vectors belong to a vector space where each dimension is associated with one n-gram instruction. Based on this numerical embedding, machine learning algorithms are used for behavioral clustering and classification —as it is usual in machine learning with the *two-stage method* (c.f. section 5.2.2, page 195). The results obtained with this method were sound, achieving a high F-score (~ 0.95) and outperforming the then state-of-the-art framework which was based on feature hashing [22]. Nonetheless, setting n-gram instructions as orthogonal dimensions oversimplifies their geometrical representation —e.g. *why (1,0,0) and (0,1,0) would be equidistant to (1,0,0) and (0,0,1)?* —furthermore, this embedding strategy is not easily transposed to other data types that cannot be easily parsed in n-grams (e.g. graphs).

In 2016, Hashemi *et al* proposed a graph embedding method based on *power iteration*.
The method targets the graph spectrum, which uses the eigenvalues and eigenvector of the
adjacency matrix to deduce useful properties of the graph, in order to compute numerical
embedding. In turn, power iteration is a method that given the adjacency matrix of a graph
computes a vector that is a linear combination of eigenvectors proportionate to their eigen-
values. As usual, the authors use the *two-stage method* to apply supervised machine learning
classifiers (KNN and SVM) for malware detection. The method achieves decent results with
datasets of different sizes and the authors emphasize the low false positive ratio obtained.

As of 2018, embedding learning starts to be used in malware analysis. Awad *et al* apply Word2Vec on the opcodes extracted with binary code disassembling to obtain an embedding vector representing each instance [14]. This method was tested on ~ 11K malware instances of 9 different families using kNN-based classification, achieving a high classification scores reaching over 95%. A similar approach was adopted by R. Lu [170], but it uses a LSTM model for malware detection [100].

Other works adopt a similar methodology, using the *two-stage method* after computing
the numerical embedding. Jiang *et al* used Node2Graph [109] to compute embedded vec-
tors from function-call graphs in parallel with a boolean model of Windows API calls to in-
put them in an auto-encoder neural network for malware detection [134]. Zhang et al used
GloVe [215], which trains a global word co-occurrence matrix and produces a word vector
space model, to compute the embedded vectors and use them to train a CNN for malware
detection [325]. Kale et al used Word2Vec on opcode sequences in conjunction with a fea-
ture vector obtained from hidden Markov models (referred as HMM2Vec), applying them in
several supervised model (SVM, k – NN, RF and CNN) to do (supervised) malware classifi-
cation [139] —S. Paul and M. Stamp followed a very similar approach [213].

5.4 Methodology

5.4.1 EMB-DUET

Our methodology improves the DUET workflow, which combines different data views (from static and dynamic data sources) using ensemble clustering, with the use of numerical embedding learning to normalize data into vectors and speed up the overall workflow. Therefore we refer to our methodology as *EMB-DUET*.

Figure 5.8 shows the general workflow of EMB-DUET. It consists of a multi-phased pro-
cess in which samples are analyzed using either static and/or dynamic analysis to extract
the representative data (i.e. data views), then (i) the numerical embedding vectors of the
data views are computed through unsupervised learning, (ii) the individual specialized clus-
tering (i.e. ensembles) for each embedding is computed, and finally (iii) all ensembles are
consolidated into a final clustering that take all data views into account.

The embedding computation step (i) is dependent on the type of the data views (e.g. text, graph), however a large set of algorithms exist to handle most different data types. Once the embedding vectors have been computed, different clustering algorithms can be used (in step ii); eventually, this phase can use ensemble clustering to combine different clustering strategies (e.g. hierarchical, density, etc) in the process of ensemble computation.

We present our workflow as a versatile approach for malware clustering based on hybrid
features. In the current work, we instantiate the workflow components as follows:

— Data views: n-grams, string and ECDGs

— Clustering: OPTICS and HDBSCAN

— Ensemble Clustering: CSPA, HGPA, MCLA, HBGF and NMF
<figure><img src="image_67.png" /><figcaption>Figure 5.8 – General workflow of EMB-DUET.</figcaption></figure>

Our set of data views are chosen because of their complementary characteristics. n-grams and strings bear syntactic information about the binary files (hence entailing high precision but low recall), whereas ECDG has proven to be a good alternative to represent the semantics of the binary file, providing high accuracy and robustness (c.f. chapter 4, page 153).

In order to improve the quality of ECDGs, we merge the graphs obtained through symbolic execution (c.f. *angr-extractor* in section 3.5.1, page 136) and traditional dynamic analysis using Cuckoo sandbox (detailed later in section 5.5.1, page 212) using disjoint union (c.f. section 3.4.3, page 132); thereby we follow a *feature composition* method for graph-based data view as explained above (c.f. section 5.3.1, page 204). To avoid confusion with the regular version of ECDG, which do not leverage feature composition, we refer to this setting as *enhanced ECDG* (e-ECDG). Figure 5.9 shows the feature composition of the e-ECDGs.

To compute numerical embedding vectors we choose the learning methods according to the data type that most naturally corresponds to the algorithm input data type. Therefore, we use Doc2Vec for n-grams and strings, and Graph2Vec for e-ECDGs.

The clustering workflow follows modular architecture, which is able to work with heterogeneous features and leverages ensemble clustering to consolidate all data views in a single cluster, addresses RQ5.1 <fnref n="5" />.

The choice of complementary features such as n-grams (syntactic), strings (syntactic) and e-ECDGs (structural/semantic) —in addition to the feature composition of the former,

<footnote n="5">5. **RQ5.1:** *How to design a data-driven clustering that can systematically work with heterogeneous features?*</footnote>
<figure><img src="image_68.png" /><figcaption>Figure 5.9 – Feature composition of the enhanced-ECDG.</figcaption></figure>

which combines symbolic execution and sandboxing —targets **RQ5.2** <fnref n="6" />.

Finally, the embedding layer (in addition to DUET) that learns vector representations for the data views, in order to normalize input data to speed up the overall workflow, addresses **RQ5.3** <fnref n="7" />.

### 5.4.2 $\sigma^{ECDG}$-learning with SVM Regression

Using e-ECDG in the EMB-DUET workflow predicates on the transformation of graphs into vectors. Here we benefit from the accuracy and robustness properties previously ascertained on our behavioral graph representation ($\sigma^{ECDG}$) (c.f. chapter 4, page 153). For this, to guarantee the same properties for the vectorized instances (i.e. after the learning process), it is necessary to readjust the similarity measure of the embedded numerical vectors of e-ECDGs with respect to $\sigma^{ECDG}$.

To this end, we consider the inclusion of one additional learning layer that transposes the $\sigma^{ECDG}$-based similarities from graphs onto vectors. More specifically, this additional layer learns a SVM regression model to predict the $\sigma^{ECDG}$-similarity given a pair of vectorized e-ECDGs. Figure 5.10 shows the workflow for e-ECDG clustering based numerical embedding and the additional $\sigma^{ECDG}$-regression layer.

The $\sigma^{ECDG}$-regression model is trained with a set of e-ECDG pairs and the corresponding $\sigma^{ECDG}$-similarities. Since training dataset ($\mathcal{T}$) includes tuples that contain pairs of e-ECDGs, it grows quadratically with respect to the number of graphs in $\mathcal{G}$. However, since there is no obligation to train the SVM regression model with all pairs in $\mathcal{G} \times \mathcal{G}$, we can limite $\mathcal{T}$ to $K$

<footnote n="6">6. **RQ5.2:** How to design a data-driven clustering that can systematically work with heterogeneous features?</footnote>

<footnote n="7">7. **RQ5.3:** How to optimize the data-driven clustering to make it scalable?</footnote>
<figure><img src="image_69.png" /><figcaption>Figure 5.10 – e-ECDG clustering workflow based on numerical embedding with additional $\sigma^{ECDG}$-regression layer.</figcaption></figure>

tuples, where *K* is a value defined as parameter. This asymmetry in the asymptotical curves actually facilitates the composition of $\mathcal{T}$.

More formally, the training data comprises a set of *K* tuples composed of ($G_i$, $G_j$, $\sigma^{ECDG}$ ($G_i$, $G_j$)), where *i*, *j* are the indexes and $G_i$ and $G_j$ are graphs of e-ECDGs randomly chosen from $\mathcal{G}$ = {$G_1, \cdots, G_N$}. Thus, we can notate it as $\mathcal{T}$ = {($G_{i_1}, G_{j_1}, \sigma^{ECDG}(G_{i_1}, G_{j_1}))$, \dots, ($G_{i_K}, G_{j_K}, \sigma^{ECDG}(G_{i_K}, G_{j_K})$)}, where $i_1, \cdots, i_K, j_1, \cdots, j_K \in R$ {1, \cdots, N}.

Figure 5.11 shows the training process and the use case of the SVM regression model; the elements that participate in the learning process are colored in blue, while those that participate in the use case, which rely on the trained model to predict the $\sigma^{ECDG}$-similarity of unknown ECDGs, are colored in orange—the test phase relies on the trained model as in the normal use case.

To guarantee a maximum of diversity of $\mathcal{G}$ in the composition of $\mathcal{T}$, it is possible to use round-robin tournaments generated by the circle method for selecting graphs in $\mathcal{G}$, because this generation method maximizes the carry over effect [158]. Furthermore, since the distribution of $\sigma^{ECDG}$ values for graphs in $\mathcal{G}$ is not uniform (because it is more likely to find dissimilar ECDGs for a pair of graphs randomly chosen), it may be necessary to generate a superset $S_{\mathcal{T}}$ of tuples before composing $\mathcal{T}$ in accordance with the desired distribution.

Beyond $\mathcal{T}$, the success of the learning process also depends on the hyperparameter setting for the SVM regression. Altogether, the entire parametric setting encompasses a great number of variables that contribute to the regression performance; but it also provides great learning flexibility to this additional transformation layer.

This transformation layer aims to produce predictions of the $\sigma^{ECDG}$-similarity —which was demonstrated to be accurate and robust (c.f. chapter 4, page 153)—given the vectorized version of the ECDGs. Although intending to contribute to RQ5.2 8, the outcome of $\sigma^{ECDG}$-learning with SVM Regression is only ancillary to TOP-RQ3: *How to compute the similarity of unknown programs with high accuracy while being friendly to search and clustering algo-

<footnote n="8">8. RQ5.2: Which set of features can afford accuracy and robustness to the data-driven clustering?</footnote>
<figure><img src="image_70.png" /><figcaption>Figure 5.11 – Training process and use case of the SVM regression model.</figcaption></figure>

rithms for malware analysis?

**5.5 Experiments**

**5.5.1 Implementation Setup**

ngrams-extractor

To extract the n-grams of files we extended our analysis framework with a proprietary tool written in the Go language [217]. For this, we prepared a docker image with the appropriate Orqal interface (c.f. section 3.5.1, page 138)

Cuckoo

Cuckoo [267] is a tool to automate the analysis of artifacts in the context of a sandbox<fnref n="9" />. Cuckoo was originally started in 2010 as an open source project in a Google Summer of Code

<footnote n="9">9. Security mechanism isolate the execution of programs (c.f. section 2.3.1).</footnote>
within the Honeynet Project<fnref n="10" />.

Cuckoo has a modular design that endows it with great flexibility and facilitates its in-
tegration in larger frameworks. Figure 5.12 shows a basic model of Cuckoo architecture,
composed of a front-end (i.e. cuckoo host) and a back-end where the automated malware
analysis takes place. Cuckoo host provides interfaces to interact with users as well as other
systems and manages the back-end by dispatching new analyses on the sandbox machines
and retrieving the generated reports.

<figure><img src="image_71.png" /><figcaption>Figure 5.12 – Basic architecture model of Cuckoo.</figcaption></figure>

The most basic deployment of Cuckoo consists of a single machine on which the Cuckoo
Host runs as a service and the sandbox environment is deployed with virtual machines.

<footnote n="10">10. Non-profit security research organization dedicated to investigating the latest attacks and developing open source security tools to improve Internet security. Link: https://www.honeynet.org</footnote>
Cuckoo has built-in integrations with several virtualization hypervisors, including Virtual-Box, KVM, VMware and others. Cuckoo also supports distributed deployment on a cluster of machines, which can contain physical and virtual machines.

Figure 5.13 provides more details about each Cuckoo component. The Cuckoo Host comprises three modules (Core, Web and REST API) and the Sandbox consists of a custom analysis environment whose principal requirement is to have Cuckoo agent.py running —implementing the Cuckoo agent as a Python script enables the Cuckoo sandbox to work on different operating systems (e.g. Windows and Linux).

<figure><img src="image_72.png" /><figcaption>Figure 5.13 – Macroscopic system structure of Cuckoo.</figcaption></figure>

Cuckoo Core is responsible for managing the analysis queue, controlling the task scheduling and launching new analysis tasks; Cuckoo Web provides a GUI that allows users to submit files to be analyzed, check the analysis dashboard and have access to reports of finished
tasks; and Cuckoo REST API provides the same functionalities as Cuckoo Web but through a REST API that is more convenient for integration with other systems.

The Cuckoo agent runs a service that listens to the network awaiting commands (with their respective files and parameters) from the Cuckoo Host to initiate a new analysis. The Cuckoo agent interacts with analysis tools that are activated during the execution of the target file; eventually, it can also interact with 3rd-party applications installed on the sandbox. When the analysis is complete, the Cuckoo agent prepares the final report and makes it ready to be fetched by the Cuckoo Host.

Cuckoo supports many different configurations and is able to integrate many other existing tools. For a complete view refer to Cuckoo official documentation<fnref n="11" />.

**Our Setup** For maximum efficiency and simplicity, we deploy Cuckoo on a single machine having the following specification: 1 processor Intel Xeon W-2155 (10 cores, 13,75 MB de cache, 3,30 GHz), 128GB RAM (DDR4, UDIMM, non-ECC, 2666MHz), with Ubuntu Linux 18.04. A storage of 40TB is available over NFS where the reports are saved and we use MongoDB as database.

In our setup we installed the following modules on Cuckoo: *Pydeep* to compute the ss-deep of the submitted files; *Volatility* to analyze memory dumps obtained in the analysis; *TCPDump* to analyze the data exchanged over network during the file execution.

We integrated our Cuckoo Host with two hypervisors: QEMU-KVM and VirtualBox. We conducted a pre-analysis benchmark to compare both integration setups, which showed almost no difference in the average analysis time (~130s on VirtualBox vs. ~134s on QEMU-KVM) but better stability with QEMU-KVM. Therefore we decided to use QEMU-KVM as hypervisor to the remainder of the experiments. As for the sandboxes, we generated 30 clones of a 4GB RAM VM with a typical Windows 10 installation and a standard setup of the Cuckoo agent (i.e. without any hardening).

We also implemented a custom reporting module to generate (subset of) the ECDGs with the Windows API calls intercepted during the target file analysis. The standard Cuckoo agent setup relies on procmon [188] (c.f. section 2.3.1, page 88) to monitor filesystem activities, registry and process/thread in real time, including these information in the final report. Despite being able to extract many relevant calls—which is certainly enough for a good analysis—procmon hooks Windows at the kernel level and therefore intercepts only the calls to the Windows API; thus, our ECDG report module for Cuckoo can generate only a (reasonably

<footnote n="11">11. Link: https://cuckoo.readthedocs.io/en/latest/</footnote>
good) subset of the entire ECDG.

5.5.2 Dataset

To compose our dataset we took an approach similar to the one of our preceding experi-
ment (c.f. section 4.5.1, page 168). Again we relied on syntactical signatures due to the high
precision they typically provided and also due to the stability in reproducing the experiments
with other datasets. Thus we reused the set of public Yara rules used before (i.e. Yara-Rules,
InQuest and McAfee ATR Team) in section 4.5.1 (page 168).

However, this time we turned to Malpedia [218] to complement our set of Yara rules and to improve the quality of our ground truth. Malpedia is a collaboration platform for curating a malware corpus, which offers “a plethora of possibilities for researchers, including using it as a *testbed for evaluations* on detection and analysis methods, *quality assurance for classification*, and contextualization of new malware”. Malpedia keeps a record of curated malware families with their corresponding Yara rules. Therefore, we guided the composition of our dataset with these Malpedia Yara rules, keeping the Yara rule filename<fnref n="12" /> as label of the malware family ground truth.

Initially we ran the Yara rules on our entire database, which comprises samples collected in previous projects, samples made available by VirusTotal for academic research and a complete dump of the MalwareBazaar database. We could identify over 100K matched files, many of which matched multiple Yara rules. Table 5.1 shows the number of files per number of Yara rules.

<table><caption>Table 5.1 – Number of files per number of Yara rules</caption>
  <tr>
    <th><strong>#Yara Rules</strong></th>
    <td>1</td>
    <td>2</td>
    <td>3</td>
    <td>4</td>
    <td>5</td>
    <td>6</td>
  </tr>
  <tr>
    <th><strong>#Matched files</strong></th>
    <td>53200</td>
    <td>28938</td>
    <td>4641</td>
    <td>1287</td>
    <td>16746</td>
    <td>635</td>
  </tr>
</table>

To simplify the remainder of our analysis we looked for a univocal mapping between
samples and Yara rules. Therefore we closely checked the cases of files matching multiple
Yara rules and found many redundancies.

To find such matches and combine them into a unique label, we defined the *matching graph* as a graph whose nodes represent Yara rules and edges connect Yara rules that match concomitantly at least one file. By constructing this matching graph from our set of matched

<footnote n="12">12. Henceforth we use the terms *Yara rule filename* and *Yara rule* for the sake of brevity, although technically a Yara rule file could include multiple Yara rules.</footnote>
files, we found 223 isolated nodes (37.54% of the total of nodes), i.e. the files matched only a
single Yara rule.

The isolated nodes correspond to our case of interest (i.e. univocal mapping) and there-
fore they do not need to undergo any merging strategy. Thus, for the remainder of this graph
analysis, we promptly remove these nodes.

Figure 5.14 shows the matching graph (after removal of isolated nodes). We can see that besides the very dense component in the center, all other connected components are rather small. Therefore, we decided to merge them in a single label.

<figure><img src="image_73.png" /><figcaption>Figure 5.14 – Matching graph (after removal of isolated nodes).</figcaption></figure>

For most cases the Yara rules were clearly related (e.g. win.globeimposter_g0.yar and win.globeimposter_auto.yar Yara rules). However, this merging strategy provided some surprising results. For instance, we found a connected component that included only the Yara rules for Neutrino<sup>13</sup> and QakBot<sup>14</sup>, which at first sight are two very distinct threats —our final family label for such cases is composed with both families, e.g. *qakbot+neutrino*.

This merging procedure allowed us to obtain a total of 13,065 files —referred as *annotated files* or *annotated dataset* —uniquely matched with 308 Yara rules (including merging).

<footnote n="13">13. Link: https://malpedia.caad.fkie.fraunhofer.de/details/win.neutrino</footnote>

<footnote n="14">14. Link: https://malpedia.caad.fkie.fraunhofer.de/details/win.qakbot</footnote>
Intriguingly, 282 of these files were initially in our cleanware database, from which 168 of the cleanware files were matched with Yara rules associated with the *nymaim*<fnref n="15" /> label. To enable the use of the AnR paradigm (c.f. section 4.4.3, page 165), our dataset was extended with non-annotated files from our cleanware database to obtain a total of 27K files<fnref n="16" />.

### 5.5.3 $\sigma^{ECDG}$ -Learning with SVM Regression

This section presents the experimental analysis of $\sigma^{ECDG}$ -Learning with SVM Regression. Since our goal with this analysis is to readjust the similarity measure of numerical embedding vectors of e-ECDGs with respect to $\sigma^{ECDG}$ as previously analyzed (c.f. section 4, page 153), we reused the former dataset (c.f. section 4.5.1, page 168) for this part of our experimental analysis.

More specifically, we sought the set of hyperparameters for Graph2Vec and the SVM Regression that elicit the best possible learning results. For this, we carried out a random search optimization approach [25] to explore the parametrical space as much as possible. Random search is ideal because of its asynchronicity, preemption and predisposition to cover a larger span of the hyperparameter space with less trials.

We also assessed the effects of different training and test size ratios, aiming to investigate the differences in the asymptotical curves for different sizes of the training and test datasets with respect to the total number of graphs. Our target was to find the sweet spot ratio that provides very good learning results for the least training cost.

Table 5.2 shows the parameter grids randomly explored in our analysis. We computed a total of 6,202 trial tuples consisting of ($\mathcal{P}_{emb}$, $\mathcal{P}_{SVM}$, $\mathfrak{s}_{train}$, $\mathfrak{s}_{test}$), where $\mathcal{P}_{emb}$ and $\mathcal{P}_{SVM}$ denote the instances of parameter sampling for Graph2Vec and SVM Regression, and $\mathfrak{s}_{train}$ and $\mathfrak{s}_{test}$ the sizes of the dataset sampling for the learning and test phases —we fixed the random seed for this sampling in order to avoid a selection bias when comparing different trials.

We defined our loss function as the mean absolute error (MAE) of the predictions, i.e.:

$$MAE = \frac{\sum_{i=1}^{\mathfrak{s}_{test}} |\hat{\sigma} - \sigma|}{\mathfrak{s}_{test}} \qquad (5.5)$$

where $\sigma$ is the true value of the similarity function computed directly with the graphs (c.f.

<footnote n="15">15. Link: https://malpedia.caad.fkie.fraunhofer.de/details/win.nymaim</footnote>

<footnote n="16">16. More precisely, a total of 27, 127 files, which corresponds approximately to the same number of annotated and non-annotated files.</footnote>
<table><caption>Table 5.2 – Random Search: Parameter Grids for Graph2Vec and SVM Regression</caption><thead><tr><th></th><th>Parameter</th><th>Range</th><th>Step</th></tr></thead><tbody><tr><td rowspan="4">Graph2Vec</td><td>WL-iterations</td><td>[1, 128]</td><td>2<sup>x</sup>, 1 ≤ x ≤ 7</td></tr><tr><td>Dimensions</td><td>[128, 1192]</td><td>128</td></tr><tr><td>Epochs</td><td>[10, 100]</td><td>5</td></tr><tr><td>Learning Rates</td><td>[0.001, 0.250]</td><td>0.001</td></tr><tr><td rowspan="2">SVM Regression</td><td>Kernel: RBF</td><td>C<br>γ</td><td>10<sup>x</sup><br>{scale, auto}</td><td>x ∈ {-5, -4.9, ..., 4.9, 5}</td></tr><tr><td>Kernel: Poly</td><td>C<br>degree</td><td>10<sup>x</sup><br>[3, 20]</td><td>x ∈ {-5, -4.9, ..., 4.9, 5}<br>1</td></tr><tr><td>Dataset</td><td>Training<br>Test</td><td>size<br>size</td><td>[1000, 11000]<br>{1000, 5000, 10000}</td><td>500</td></tr></tbody></table>

section 4.4.2, page 164) and $\hat{\sigma}$ is the corresponding prediction provided by the SVM Regression model. Equation 5.5 implicitly assumes that the test dataset is composed of $\{(g_{1,1}, g_{1,2}), \dots, (g_{5_{test},1}, g_{5_{test},5})\}$ where both $\sigma_i$ and $\hat{\sigma}_i$ are computed with $(g_{i,1}, g_{i,2})$.

To help visualizing the training results for the 6,202 tuples, we elaborated a curve plot that we refer as *trend plot*; it takes the set of scores values for a given evaluation group, sorts them and presents them in a xy-plot where score values are projected on the y-axis and points are equally spaced on the x-axis (i.e. the distance between points on the x-axis is scaled with respect to the total number of points). This plot does not show the performance of individual data inputs due to the transformations in the x-axis, but it gives a quick coarse overview of overall performance.

Figure 5.15a shows the curve of MAE values corresponding to the entire set of trial tuples; the curve is composed of two segments: a steeply decreasing segment on the left side (from MAE= 1.88 up to MAE≈ 0.25), and a flat segment around MAE≈ 0.25.

Figure 5.15b shows the trend curve for MAE below 0.5 to identify the transition of the two aforementioned segments; we visually determined the threshold of MAE= 0.28 as the pivoting point of the transition —hence the splitting of the trend curve in the latter plot. The set of values with MAE≤ 0.28 comprises a total of 5,592 trial tuples, which correspond to 90.16% of the total number of tuples —the identification of this threshold is important to help setting a maximum cut-off value in the color scale of graphs whenever necessary.
<figure><img src="image_74.png" /><figcaption>Figure 5.15 – Trend plot of MAE for the total set of trial tuples.</figcaption></figure>

## Data cuts

The learning phase aims to reduce total prediction error according to MAE (equation 5.5), which is affected by each factor randomly sampled for the trial, i.e. ($\mathcal{P}_{emb}, \mathcal{P}_{SVM}, \mathfrak{s}_{train}, \mathfrak{s}_{train}$). Therefore, for this analysis we cut the sampled data into subsets to inspect their effects on the MAE obtained. The data cuts that we considered are:

- *Embedding Hyperparametrization:* the sets of hyperparameters used in the embedding learning.

- *SVM-Kernel:* kernels used in the SVM Regression.

- *Training-Test Ratio (TTR)*: ratios between the sizes of training and test datasets (computed as $\mathfrak{s}_{train}/\mathfrak{s}_{test}$).

The sizes for the *Embedding Hyperparametrization*, *SVM-Kernel* and *Training-Test Ratio* are respectively 122, 2 (i.e. either RBF or Polynomial) and 50. We emphasize that the subsets of a given data cut are mutually exclusive, but the intersection of subsets from different data cuts are not necessarily empty.

**Effects of Training-Test Ratio** Figure 5.16 shows the heatmap of average MAE value with respect to the *Embedding Hyperparametrization* and *TTR* data cuts; each row in the heatmap corresponds to a unique set of hyperparameters used during the embedding learning, while each column corresponds to a *TTR* value; the red lines delineate the frames that include all embedding hyperparameter sets for each *wl-iterations*.
It is not possible to identify any noticeable pattern —besides a darker region in the center, however the darker values are also present all over the entire range of *TTR* values —we therefore conclude that the *TTR* are not a dominant factor in the SVM Regression for the chosen ranges of *s*<sub>train</sub> and *s*<sub>test</sub> in our random search.

<figure><img src="image_75.png" /><figcaption>Figure 5.16 – Heatmap of average MAE with respect to the *Embedding Hyperparametrization* and *TTR* data cuts (MAE≤ 0.28).</figcaption></figure>

**Effects of SVM Kernel** Table 5.3 shows the results of MAE per SVM kernel. We note that the average MAE for the complete set of trials for each SVM kernel deviates considerably, and that the portion of trials removed by the threshold is considerably higher for the polynomial kernel. As for the MAE below the threshold, figure 5.17 shows that RBF kernel attain in general a lower MAE than the Polynomial kernel.

Figure 5.18 shows the average MAE obtained with the polynomial kernel for different degrees. No noticeable difference appear for the different degrees; the values with higher average MAE are due to outliers, hence the bigger standard deviations for these cases. Therefore, in the remainder of the analysis we disregard the possibility of the scores for the polynomial kernel being largely impacted by the setting of the degree parameter.
<table><caption>Table 5.3 – Random Search: SVM Kernels</caption><thead><tr><th rowspan="2">Kernel</th><th colspan="2">No Threshold</th><th colspan="3">Below threshold (MAE = 0.28)</th></tr><tr><th>#</th><th>Average MAE</th><th>#</th><th>% (No Threshold)</th><th>Average MAE</th></tr></thead><tbody><tr><td>RBF</td><td>3307</td><td>0.260 (±0.488)</td><td>3106</td><td>93.92%</td><td>0.252 (±0.021)</td></tr><tr><td>Poly</td><td>2895</td><td>0.287 (±0.136)</td><td>2486</td><td>85.87%</td><td>0.256 (±0.017)</td></tr></tbody></table>

<figure><img src="image_76.png" /><figcaption>Figure 5.17 – Trend plot of MAE (below threshold) for each SVM Kernel.</figcaption></figure>

To confirm the occurrence of a statistical difference between the results of both SVM kernels, we perform a two-sample independent T-Test considering the two scenarios (with and without threshold). Table 5.4 shows the value of the Percent Point Function (PPF) (for quantile equal 0.001 and freedom degree equal #RBF tuples + #Polynomial tuples - 2) and the values for t-statistic and p-value (with and without mean trimming<fnref n="17" />). In all scenarios the t-statistic values are considerably beyond the theoretical value from the t-distribution and the p-values are remarkably small, which confirms the statistical difference of MAE values obtained with each SVM Kernel in favor of RBF.

<table><caption>Table 5.4 – Random Search: Independent T-Tests of SVM Kernels</caption><thead><tr><th rowspan="2">Scenario</th><th rowspan="2">PPF<br>(quantile=0.001)</th><th colspan="2">t-statistic</th><th colspan="2">p-value</th></tr><tr><th>no trim</th><th>trim=0.15</th><th>no trim</th><th>trim=0.15</th></tr></thead><tbody><tr><td>No Threshold</td><td>-3.721</td><td>-10.075</td><td>-8.82</td><td>1.47 10<sup>-23</sup></td><td>1.58 10<sup>-18</sup></td></tr><tr><td>Threshold (MAE ≤ 0.28)</td><td>-3.092</td><td>-7.492</td><td>-5.642</td><td>7.824 10<sup>-14</sup></td><td>1.805 10<sup>-8</sup></td></tr></tbody></table>

<footnote n="17">17. The trimmed test corresponds to Yuen's t-test, which is recommended for long-tailed distributions or distributions contaminated with outliers.</footnote>
<figure><img src="image_77.png" /><figcaption>Figure 5.18 – Average MAE for polynomial kernel with different degrees.</figcaption></figure>

Figure 5.19 shows two violin plots that corroborate the statistical difference of both kernels in favor of RBF (when the threshold is considered or not). Figure 5.19a shows the distributions for the scenario without threshold; we notice that Polynomial kernel generates outliers resulting in a long-tailed distribution, whereas the BRF kernel does not suffer from the same deformity. Figure 5.19b shows the distributions for the scenario with threshold equal 0.28. We see that both kernels have a larger concentration of results around Average MAE≈ 0.27 and another around Average MAE≈ 0.24; however the RBF kernel one is larger around Average MAE≈ 0.24, whereas the Polynomial kernel one is larger around Average MAE≈ 0.27. Furthermore, the BRF median is far below the Polynomial mean and the lower quartile for the RBF is smaller than the upper quartile, while in the case of the Polynomial Kernel it is the opposite.

**Graph2Vec and SVM Regression with RBF Kernel** Since the RBF Kernel outperforms the Polynomial Kernel, we further inspected the hyperparametrization for this case. The hyper-parameters of the SVM Regression for the RBF Kernel are kernel coefficient ($\gamma$) and the regularization parameter $C$; the regularization parameter $C$ determines the trade-off between the flatness of the maximum-margin hyperplane and the total deviation tolerated (c.f. section 5.2.1, page 191). The kernel coefficient ($\gamma$) for the BRF kernel (i.e. $k(x_i, x_j) = e^{-\gamma||x_i-x_j||^2}$) defines the reach of influence of $x_i \in X$ for the training set $X = \{x_0, \dots, x_{\text{train}}\}$, which can be set to *scale* and *auto* as follows:
<figure><img src="image_78.png" /><figcaption>Figure 5.19 – Violin plots showing the distribution of average MAE for the Polynomial and RBF Kernels.</figcaption></figure>

$$ \gamma = \frac{1}{\mathfrak{s}_{\text{train}} \cdot \text{std}(X)} \qquad (\text{scale}) $$

$$ \gamma = \frac{1}{\mathfrak{s}_{\text{train}}} \qquad (\text{auto}) $$

Figure 5.21a shows the heatmap of the average MAE for the embedding learning versus the kernel coefficient: the rows correspond to each hyperparameter setting of the embedding learning and the columns to the setting of the kernel coefficient (i.e. *scale* of *auto*). Only the values below the MAE threshold (= 0.28) are shown in order not to disturb the color scale, and the red horizontal lines separate each set of *wl-iterations* in the embedding hyper-parameter. Although not being definitely conclusive, figure 5.21a shows that the best average MAE are obtained with the setting as *auto* for *γ*.

Figure 5.21b shows the heatmap of the average MAE for the embedding learning versus the regularization parameter *C*: the rows correspond to each hyperparameter setting of the embedding learning and the columns to the different values for *C*. We notice that the best average MAE are obtained with intermediary values for *C* (around 1 < *C* < 20).

Finally, we analyzed the impact of size for the training dataset. Figure 5.20 shows the trend plot for different dataset sizes —other sizes were not included in the plot to avoid clustering the graph. The best MAE scores are attained with the larger training dataset; in particular, there is a great difference in the minimum MAE scores obtained with training sizes equal 1000 and 10000, i.e. respectively 0.2192 and 0.2052.
<figure><img src="image_79.png" /><figcaption>Figure 5.20 – Trend plot of MAE curves of tuples trained with RBF kernel and different training dataset sizes.</figcaption></figure>
<figure><img src="image_80.png" /><figcaption>(a) SVM Setting for $\gamma$.</figcaption></figure>

<figure><img src="image_81.png" /><figcaption>(b) SVM Regularization Parameter C.</figcaption></figure>

Figure 5.21 – Heatmap of average MAE with respect to the hyperparameter settings for the embedding learning and the SVM Regression.
### 5.5.4 EMB-DUET

Evaluating EMB-DUET is challenging because of the potential combinatorial explosion of parameters when the underlying data views are taken into account (e.g. parameters for the symbolic execution as studied in chapter 3, page 111). Therefore, we focused mainly on the hyperparameters for the numerical embedding learning and clusterings to understand how they affect the final ensemble clustering.

Initially, we assessed the individual clusterings so as to identify the best strategy to select the set of hyperparameters [steps (i) and (ii) of figure 5.8], then we picked the most promising clusterings to evaluate the information gain attained with different ensemble clustering algorithms.

#### Generating the Data Views

Generating the data views involves the analysis of files for the extraction of “raw data”, which are subsequently used to compute the numerical embedding vectors. To avoid the parametric combinatorial explosion, we set the extraction parameters used to build the data views with basis on our background expertise.

In our experiments we considered three data views: e-ECDGs, strings and n-grams. For the generation of the e-ECDGs, we obtained the call traces of the experimental dataset (c.f. section 5.5.2, page 216) using *angr-extractor* (c.f. section 3.5.1, page 136) and our custom reporting module for Cuckoo (c.f. section 5.5.1, page 212). Since the Cuckoo report already includes the strings of the analyzed file, we use them to compute the numerical embedding vectors for the strings. We obtained the n-grams from our ngrams-extractor (c.f. section 5.5.1, page 212).

We decided not to use $\sigma^{ECDG}$-Learning with SVM Regression for this experiment to avoid any error propagation, because the MAE attained is still non negligible. Therefore, the remainder of this evaluation relies on the graph similarity obtained directly with Graph2Vec, instead of $\sigma^{ECDG}$ as studied in chapter 4 (pager 153).

Table 5.5 shows the set of the chosen parameters and the data extraction results. For different reasons some analyses could not provide any data after completion, therefore we do not consider them in the measurement of the average analysis runtime (for the e-ECDG generation). We also excluded the analyses that generated graphs without edges.
<table><caption>Table 5.5 – Data Views: Parameters and Results</caption><thead><tr><th colspan="2">Data View</th><th>Parameters</th><th># Results</th><th>Average Time</th></tr></thead><tbody><tr><td rowspan="2">e-ECDGs</td><td>angr-extractor</td><td>max active paths: 8<br>max mem: 10GB<br>step timeout: 8s<br>symb loop threshold: 4<br>global timeout: 1h</td><td>8577</td><td>62.54s(±309.26s)</td></tr><tr><td>Cuckoo</td><td>route: internet<br>timeout: 1h</td><td>5754</td><td>1104.75s(±159.23s)</td></tr><tr><td colspan="2">strings</td><td>*</td><td>27004</td><td>*</td></tr><tr><td colspan="2">n-grams</td><td>n ∈ {5, 8}</td><td>27127</td><td>-</td></tr></tbody></table>

Exploring hyperparameters for Numerical Embedding Learning

Since we did not have any *a priori* knowledge about how the hyperparameters interplay, we took once again the *random search optimization* approach [25] to explore our space of hyperparameters.

Tables 5.6 shows the grid of hyperparameters created for the experiment. Each trial randomly selected the hyperparameters within the defined ranges, wherein the values were dispersed in intervals of fixed sizes (annotated in field *step* in the table). In the case of e-ECDGs, we consider two different step sizes: big step and small step; the former intends to cover a large span of the hyperparameter space, whereas the latter aims to catch the magnitude of effects occurred due to small changes in the hyperparameters.

<table><caption>Table 5.6 – Random Search: Parameter Grids for Graph2Vec and Doc2Vec</caption><thead><tr><th></th><th></th><th>WL-iterations</th><th>Dimensions</th><th>Epochs</th><th>Learning Rates</th></tr></thead><tbody><tr><td rowspan="3">e-ECDGs</td><td>Range</td><td>2<sup>x</sup></td><td>[128, 1192]</td><td>[10, 100]</td><td>[0.01, 1.00]</td></tr><tr><td>Big Step</td><td>1 ≤ x ≤ 7</td><td>128</td><td>5</td><td>0.01</td></tr><tr><td>Small Step</td><td>1</td><td>5</td><td>1</td><td>0.005</td></tr><tr><td rowspan="2">strings</td><td>Range</td><td>-</td><td>[128, 1192]</td><td>[10, 50]</td><td>[0.001, 0.250]</td></tr><tr><td>Step</td><td>-</td><td>128</td><td>5</td><td>0.001</td></tr><tr><td rowspan="2">n-grams</td><td>Range</td><td>-</td><td>[128, 1192]</td><td>[10, 50]</td><td>[0.001, 0.250]</td></tr><tr><td>Step</td><td>-</td><td>128</td><td>5</td><td>0.001</td></tr></tbody></table>

We computed for several weeks the instances of the embedding vectors of the data views with different hyperparameters. Table 5.7 shows the summary information of these computations. For the string and ngrams data views, since the learning time was dominated by the
vocabulary building step (prior to the Skip-gram learning), we computed the vocabulary just once and kept it in cache.

<table><caption>Table 5.7 – Data Views: Numerical Embedding Learning</caption><thead><tr><th>Data View</th><th>Algorithm</th><th>Vocab. size</th><th># Trial</th><th>Average Time</th></tr></thead><tbody><tr><td>e-ECDGs</td><td>Graph2Vec</td><td>-</td><td>723</td><td>1683.13s (±2203.01s)</td></tr><tr><td>strings</td><td>Doc2Vec</td><td>260101</td><td>1621</td><td>573.34s (±246.82s)</td></tr><tr><td>n-grams-5</td><td rowspan="2">Doc2Vec</td><td>10263870</td><td>37</td><td>371096.38s (±130969.74s)</td></tr><tr><td>n-grams-8</td><td>10680473</td><td>150</td><td>414628.36s (±172604.98s)</td></tr></tbody></table>

To analyze the substantial standard deviation in the runtime as well as to deepen the understanding of the impact of each hyperparameter, we plotted a scatter graph of the hyperparameters and runtime for the embedding learning of all data views (c.f. figure 5.22). For this, we rescale the range of each hyperparameter on the y-axis and mark in different colors the coordinates corresponding to (runtime, hyperparameter); therefore, for a vertical cut in the x-axis (i.e. runtime), the plot includes *n* points, where *n* is the number of hyper-parameter.

The plots 5.22b and 5.22c clearly show a linear runtime growth with respect to the chosen number of *epochs* in the training, while the other factors are scattered all over the plot. We computed the linear regressions for strings and ngrams, getting the equations below (without the bias coefficient):

$$ \hat{t}_{learn} = 4.18 + 18.73 \cdot \varepsilon + 0.00 \cdot \delta + 15.62 \cdot \iota \approx 4.18 + 18.73 \cdot \varepsilon \quad (\text{strings}) $$

$$ \hat{t}_{learn} = 7400.47 + 12831.94 \cdot \varepsilon - 6.73 \cdot \delta - 45812.87 \cdot \iota \approx 7400.47 + 12831.94 \cdot \varepsilon \quad (\text{ngrams}) $$

where *ε*, *δ*, *ι* and *t̂*<sub>learn</sub> denotes respectively the number of *epochs*, the embedding dimension, the learning rate and the prediction of learning time. Note that the coefficients for the dimensions (i.e. *δ*) are negligible with respect to the others (i.e. *ε* and *ι*) and that *ε* ≈ 10<sup>4</sup>·*ι*, therefore the runtime is roughly linear with respect to the number of *epochs*.

The coefficient of determination (*R*<sup>2</sup>) for both curves are respectively 0.957 and 0.965, which means that the linear multivariate regressions fit the runtime measurements very closely. When the multivariate polynomial regression of degree 2 (or more) for the curves are computed the coefficient of determination *R*<sup>2</sup> deteriorates and leading coefficients<sup>18</sup> are

<footnote n="18">18. Coefficients of the highest monomial orders.</footnote>
<figure><img src="image_82.png" /><figcaption>Figure 5.22 – Scatter plots of hyperparameters and runtime for the embedding learning</figcaption></figure>

close to zero. For instance, for degree 2 the leading coefficients for the strings are {0.02, 0.00, -1.05, 0.00, -0.07, 49.03} and for the ngrams are {26.64, 0.07, -1403.36, 0.03, 104.82, 384687.62}, where the last coefficient multiplies the $\ell r^2$ which is itself a very small number.

As for the embedding learning of the e-ECDGs, plot 5.22a shows that there is a concurrent effect of *wl-iterations* and *epochs* —where both contribute to increasing total learning time. The multivariate linear regression of learning time of e-ECDGs with respect to the hyperparameters (whose $R^2 = 0.867$) is:

$$ \hat{t}_{learn} = -1059.62 + 14.56 \cdot e + 50.66 \cdot \text{wl} + 0.90 \cdot \delta - 588.58 \cdot \ell r \approx -1059.618 + 14.56 \cdot e + 50.66 \cdot \text{wl} $$
where $wI$ denotes the number of WL-iterations in the Graph2Vec algorithm. The multivariate polynomial regression of degree 2 improves $R^2$ to 0.980 —higher degrees do not provide much better $R^2$ —thanks to the coefficient that combines $\epsilon$ and $wI$, while the other coefficients are negligible. Therefore, the multivariate polynomial regression of degree 2 is roughly:

$$ \hat{t}_{learn} = -169.26 + 5.49 \cdot \epsilon + 26.69 \cdot wI + 0.44 \cdot \epsilon \cdot wI $$

which results in $R^2 = 0.911$.

## Clustering of the Embedded Data Views

To avoid tuning additional clustering parameters, we chose to evaluate the clustering of the embedded data views only with OPTICS and HDBSCAN, as they do not require any mandatory parameter. Nonetheless, they allow to set the minimum number of points in a cluster, which we arbitrarily set to 5.

Once again, we followed the random search optimization approach [25] to explore the space of possible clusterings spanned by the 2531 embedded vectors of the previously computed data views. Table 5.8 shows the clustering computations benchmark, grouping results with respect to dataset composition — annotated takes the intersection of elements of the data view with respect to files matching a Yara rule (c.f. section 5.5.2, page 216) —and the clustering algorithm.

We note that clusters computed with HDBSCAN take much less time than OPTICS to complete, attaining an overall speedup factor of 31, 1. Furthermore, clustering time is proportional to the number of vectors (equal to the number of samples) in the data views, which results in speedup factors (of HDBSCAN over OPTICS) of 12, 61, 52, 76 and 42, 64 for the annotated dataset and 9, 81, 28, 71 and 25, 78 for the full dataset.

## Internal and External Evaluation

To evaluate learning performance, we computed several scores of internal and external evaluation<fnref n="19" /> for all clusterings. The external evaluation scores include: *silhouette score*, *Calinski-Harabasz index* and *Davies-Bouldin index*. The internal evaluation takes the annotated dataset (i.e. samples matched with our set of selected Yara rules) as ground truth and computes the following scores: *rand score*, *AMI*, *NMI*, *homogeneity score*, *completeness score*, *v-measure score* and *Fowlkes-Mallows score*.

<footnote n="19">19. For a background on metrics for internal and external evaluation, refer to section 2.2.3 (page 82) and section 2.2.1 (page 60).</footnote>
<table><caption>Table 5.8 – Data Views: Clusterings Benchmark</caption><thead><tr><th></th><th>Algorithm</th><th># Clusterings</th><th>Dataset</th><th># Vectors</th><th>Average Time</th></tr></thead><tbody><tr><td rowspan="4">e-ECDGs</td><td rowspan="2">HDBSCAN</td><td>154</td><td>annotated</td><td>7660</td><td>81.83s (±43.30s)</td></tr><tr><td>66</td><td>full</td><td>11557</td><td>170.09s (±94.82s)</td></tr><tr><td rowspan="2">OPTICS</td><td>21</td><td>annotated</td><td>7660</td><td>1032.39s (±47.37s)</td></tr><tr><td>26</td><td>full</td><td>11557</td><td>1668.73s (±122.68s)</td></tr><tr><td rowspan="4">strings</td><td rowspan="2">HDBSCAN</td><td>304</td><td>annotated</td><td>13047</td><td>31.66s (±4.08s)</td></tr><tr><td>145</td><td>full</td><td>27004</td><td>134.03s (±11.94s)</td></tr><tr><td rowspan="2">OPTICS</td><td>36</td><td>annotated</td><td>13047</td><td>1670.40s (±17.92s)</td></tr><tr><td>51</td><td>full</td><td>27004</td><td>3848.42s (±52.80s)</td></tr><tr><td rowspan="4">n-grams</td><td rowspan="2">HDBSCAN</td><td>37</td><td>annotated</td><td>13062</td><td>39.42s (±3.75s)</td></tr><tr><td>17</td><td>full</td><td>27127</td><td>150.01s (±7.99s)</td></tr><tr><td rowspan="2">OPTICS</td><td>8</td><td>annotated</td><td>13062</td><td>1680.83s (±18.41s)</td></tr><tr><td>9</td><td>full</td><td>27127</td><td>3867.91s (±38.17s)</td></tr></tbody></table>

The score computations resulted in 8740 data points, which we grouped with respect to the dataset used in the clustering (i.e. *annotated* or *full*), the data view (i.e. *e-ECDGs*, *strings* and *n-grams*) and the clustering algorithm. To help visualizing scores we generated trend plots (c.f. section 5.5.3, page 218) using the data points corresponding to each evaluation group. Since the order of the points on the x-axis depends on the score value (projected on the y-axis), they vary from curve to curve. As before, these plots do not show the performance of each hyperparameter setting (due to the transformations for the x-axis), however they offer a quick and coarse overview of overall performance for each curve and compare the performances of clusterings for different data views and algorithms.

*Overall Internal Validation Scores:* Figure 5.23 contains all plots showing the score curves for all evaluation groups according to each internal validation metric. The plots show that:

- for any choice of dataset group (i.e. *annotated* or *full*) and data view, HDBSCAN *always* provides higher scores than OPTICS;

- the curves for *annotated-e-ECDGs* always contain the highest scoring data point;

- in general, *e-ECDGs* provides better performance than those for *strings* and *n-grams*, specially with respect to the Calinski-Harabasz and the Davies-Bouldin indexes and in the case of clustering of the full dataset;

*Overall External Validation Scores:* Figure 5.24 contains the main plots showing the score curves for all evaluation groups according to each external validation metric. The plots show that:
- for any choice of dataset group (i.e. *annotated* or *full*) and data view, HBDSCAN almost always provides higher scores than OPTICS —there are few exceptions, notably for lower scores of the e-ECDGs data view;

- generally, *strings*-based clusterings attain the best AMI score and *n*-grams-based clusterings perform considerably better than e-ECDGs-based clusterings;

- *n*-grams data views generate the clusterings with highest homogeneity scores;

- *strings* data views generate the clusterings with highest completeness scores;

- *strings* data views clustered with HDBSCAN attain Fowlkes-Mallows scores remarkably higher than any scoring group;
<figure><img src="image_83.png" /><figcaption>Figure 5.23 – Plots of score curves for the evaluation groups for each internal validation metric.</figcaption></figure>
<figure><img src="image_84.png" /><figcaption>Figure 5.24 – Plots of score curves for the evaluation groups for some external validation metrics.</figcaption></figure>
**Accuracy and Robustness (AnR)** We reuse the scores measured with the annotated part and the full dataset to perform a partial AnR analysis (c.f. section 4.4.3, page 165) —a detailed AnR would require to gradually insert noise in the evaluation dataset in order to gauge the impact of these changes on the chosen evaluation metrics.

For the accuracy phase, the most significant scores are the AMI and the homogeneity
scores. Figure 5.25 shows a scatter plot having the data points of the different HDBSCAN
clusterings with respect to their AMI and homogeneity score<sup>20</sup>. This figure shows that para-
metric settings that promote the improvement of AMI for the clusterings based on e-ECDGs
also improve their homogeneity score; a similar effect happens with the clusterings based
on strings, but the values for AMI and homogeneity scores are higher than those of the
previous case (and their variation range is much more compact); in the case of n-grams, the
measurements of AMI and homogeneity score are scattered within a compact region of high
values.

<figure><img src="image_85.png" /><figcaption>Figure 5.25 – Correlation plot (AMI x Homogeneity score)</figcaption></figure>

We note that as a rule clusterings based on *strings* provide a fairly high AMI (~ 0.80), while clusterings based on *n-grams* data views provide a fairly high homogeneity score (~ 0.78). Table 5.9 shows the hyperparameter settings that attained the best top 5 performances for AMI and homogeneity scores. As for clusterings based on e-ECDGs, they often provide lower AMI and homogeneity scores, but for the best settings these values can attain neat scores, i.e. ~ 0.7 and ~ 0.6, respectively (c.f. figures 5.26a and 5.26d).

<footnote n="20">20. The scatter plot of the OPTICS clustering displays a very different profile. For the sake of completeness, we include this plot in appendix figure 6.30.</footnote>
<table><caption>Table 5.9 – Top-5 Accuracy scores: Embedding & Clustering settings</caption><thead><tr><th rowspan="2">Score Name</th><th rowspan="2">Data View</th><th rowspan="2">Algorithm</th><th rowspan="2">Dataset</th><th colspan="3">Embedding</th><th rowspan="2">Score Value</th></tr><tr><th>dim</th><th>epochs</th><th>learn. rate</th></tr></thead><tbody><tr><td rowspan="5">AMI</td><td>strings</td><td>HDBSCAN</td><td>annotated</td><td>1152</td><td>10</td><td>0.074</td><td>0.806</td></tr><tr><td>strings</td><td>HDBSCAN</td><td>annotated</td><td>1280</td><td>10</td><td>0.024</td><td>0.806</td></tr><tr><td>strings</td><td>HDBSCAN</td><td>annotated</td><td>1152</td><td>10</td><td>0.023</td><td>0.804</td></tr><tr><td>strings</td><td>HDBSCAN</td><td>annotated</td><td>896</td><td>10</td><td>0.009</td><td>0.804</td></tr><tr><td>strings</td><td>HDBSCAN</td><td>annotated</td><td>1280</td><td>10</td><td>0.048</td><td>0.804</td></tr><tr><td rowspan="5">Homogeneity</td><td>ngrams-8</td><td>HDBSCAN</td><td>full</td><td>1280</td><td>35</td><td>0.158</td><td>0.789</td></tr><tr><td>n-grams-8</td><td>HDBSCAN</td><td>full</td><td>384</td><td>20</td><td>0.059</td><td>0.787</td></tr><tr><td>n-grams-5</td><td>HDBSCAN</td><td>full</td><td>1280</td><td>25</td><td>0.089</td><td>0.786</td></tr><tr><td>n-grams-5</td><td>HDBSCAN</td><td>full</td><td>640</td><td>25</td><td>0.17</td><td>0.786</td></tr><tr><td>n-grams-8</td><td>HDBSCAN</td><td>full</td><td>384</td><td>35</td><td>0.201</td><td>0.785</td></tr></tbody></table>

For the robustness phase, in addition to AMI and homogeneity score, we also consider the Fowlkes-Mallows index (as it is an index derived from recall and precision), completeness score and silhouette score. All curves (for all data views) are displayed in figure 5.26. We note that scores for the full dataset clusterings follow considerably closely the scores of the annotated dataset clusterings, in particular when HDBSCAN is used. Therefore we can claim that the set of embedded vectors for the chosen data views affords high robustness to the analysis.
<figure><img src="image_86.png" /><figcaption>(a) AMI (e-ECDG)</figcaption></figure>

<figure><img src="image_87.png" /><figcaption>(b) AMI (strings)</figcaption></figure>

<figure><img src="image_88.png" /><figcaption>(c) AMI (ngrams)</figcaption></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /><figcaption>(d) Homogeneity score (e-ECDG)</figcaption></figure>

<caption>(e) Homogeneity score (strings)</caption>

<figure><img src="image_92.png" /><figcaption>(f) Homogeneity score (ngrams)</figcaption></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /><figcaption>(g) Fowlkes-Mallows (e-ECDG)</figcaption></figure>

<caption>(h) Fowlkes-Mallows (strings)</caption>

<figure><img src="image_95.png" /><figcaption>(i) Fowlkes-Mallows (ngrams)</figcaption></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /><figcaption>(j) Completeness score (e-ECDG)</figcaption></figure>

<caption>(k) Completeness score (strings)</caption>

<figure><img src="image_98.png" /><figcaption>(l) Completeness score (ngrams)</figcaption></figure>

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /><figcaption>(m) Silhouette score (e-ECDG)</figcaption></figure>

<caption>(n) Silhouette score (strings)</caption>

<caption>(o) Silhouette score (ngrams)</caption>

Figure 5.26 – Plots of score curves for the AnR of evaluation groups wrt. some validation metrics.
**Score Correlations** We investigated the following question: *What can we infer about the external validation scores having access only to interval validation scores?* For this, we analyzed the correlations of the scores obtained with each clustering (taking into consideration only the annotated dataset). In particular, we considered the correlation between silhouette score (an internal validation score) and AMI (an external validation score).

Figure 5.27 shows the scatter plot having the data points with respect to their AMI and silhouette score in order to visualize the correlation of these scores. Figures 5.27b and 5.27c display relation between AMI and silhouette score showing an approximate (positive) linearity —the Pearson correlation coefficient of AMI and silhouette scores for the *strings* is 0.887 (being 0.921 for OPTICS clusterings and -0.291 for HDBSCAN clusterings, however the latter is restrained to a very compact range of values), while for *n-grams* the value of the Pearson correlation coefficient is 0.964.

<figure><img src="image_101.png" /><figcaption>Figure 5.27 – Correlation plots with AMI x Silhouette Score for all data views.</figcaption></figure>

As for *e-ECDGs*, at first glance there is no patent correlation between AMI and silhouette score (c.f. figure 5.27a). However, after scrutinizing the hyperparameter settings that provided the best AMI *and* silhouette score, we find out that the *learning rate* is an important factor to improve the positive correlation of AMI and silhouette score.

Figure 5.29 shows the correlation plot of the *e-ECDGs*-based clusterings split by different values of learning rate. We plotted in figure 5.28 the curves with the values of the Pearson correlation coefficient (for AMI and silhouette score) considering only clusterings with learning rate smaller than the threshold; in our experiment, the optimal overall threshold for the learning rate is around 0.23 (Pearson correlation coefficient = 0.928).
<figure><img src="image_102.png" /><figcaption>Figure 5.28 – Curves of Pearson correlation coefficients (AMI x Homogeneity score).</figcaption></figure>

<figure><img src="image_103.png" /><figcaption>Figure 5.29 – Correlation plots with AMI x Silhouette Score for the ECDG data view split by learning rate ($lr$).</figcaption></figure>
**Mutual Information of the Data Views**
To understand the information shared between data views, we computed AMI for all possible pairs of clusterings generated from different data views (only for the annotated dataset). However, in contrast to the AMI computations made heretofore, here we do not use the Yara labels of annotated dataset as ground truth; instead, we use the labels provided by one of the two ensembles in the pair as ground truth. This combination generated a total of 156,520 data points.

Figure 5.30 shows the score curves for all AMI scores computed with pairs of clusterings from different data views, grouped by data view. We note that clusterings based on strings and n-grams attain in average a higher AMI, meaning that strings and n-grams carry approximately the same information. The e-ECDGs data view attains roughly similar AMI when evaluated against strings and n-grams, but with a slightly higher AMI for the medium case.

<figure><img src="image_104.png" /><figcaption>Figure 5.30 – Score curves for AMI between data views.</figcaption></figure>

Figure 5.31 shows the heatmap of all the pairs of ensembles for each data view. The sorting order—which allows to visually recognize the importance order of the hyperparameters—for each data view are:

* e-ECDGs: learning rate

* strings: clustering algorithm, size of ngram, epochs, dimensions, learning rate

* n-grams: clustering algorithm, epochs, dimensions, learning rate

Figure 5.31a corroborates the brunt of higher values for the learning rate; they largely dominate the influence in the results, which precludes the sorting the remaining hyperparameters. Regarding the string and ngrams data views, the number of epochs proved to be an
important parameter —due to the relatively small range of the learning rate for the random search optimization of the embedding learning, we could not gauge the impact of learning rate values in the clustering.

<figure><img src="image_105.png" /><figcaption>Figure 5.31 – Heatmap of pairwise AMI for each data view.</figcaption></figure>

Table 5.10 presents the average AMI for the clustering of the data views considering different learning rate thresholds (first column). For each threshold value the AMI is computed considering only vectors of embedding learning instances trained with a learning rate smaller than the threshold. In the case of e-ECDGs this parameter has a major impact, not only in relation of e-ECDGs and the other data views, but also in the AMI obtained when solely e-ECDGs clusterings are considered.

<table><caption>Table 5.10 – AMI of Data Views (constrained by Learning Rate)</caption><thead><tr><th rowspan="2">Learning Rate</th><th colspan="3">e-ECDGs</th><th colspan="2">strings</th><th>n-grams</th></tr><tr><th>e-ECDGs</th><th>strings</th><th>n-grams</th><th>strings</th><th>n-grams</th><th>n-grams</th></tr></thead><tbody><tr><td>0.2</td><td>0.808(±0.111)</td><td>0.520(±0.080)</td><td>0.394(±0.102)</td><td>0.936(±0.051)</td><td>0.585(±0.112)</td><td>0.814(±0.107)</td></tr><tr><td>0.3</td><td>0.618(±0.242)</td><td>0.431(±0.164)</td><td>0.328(±0.137)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.4</td><td>0.556(±0.243)</td><td>0.397(±0.177)</td><td>0.302(±0.145)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.5</td><td>0.449(±0.270)</td><td>0.353(±0.194)</td><td>0.266(±0.158)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.6</td><td>0.346(±0.287)</td><td>0.308(±0.210)</td><td>0.231(±0.169)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.7</td><td>0.282(±0.287)</td><td>0.278(±0.217)</td><td>0.209(±0.173)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.8</td><td>0.220(±0.277)</td><td>0.245(±0.222)</td><td>0.184(±0.175)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr><tr><td>0.9</td><td>0.209(±0.273)</td><td>0.239(±0.222)</td><td>0.179(±0.175)</td><td>0.934(±0.053)</td><td>0.579(±0.113)</td><td>0.810(±0.106)</td></tr></tbody></table>

Figure 5.32 shows the heatmap of all the pairs of ensembles selected (sorting the entries by e-ECDGs, strings and n-grams). We see that the interval of AMI values for the e-ECDGs clusterings is very wide (due to the learning rate variation), however the e-ECDGs clusterings of smaller learning rates attain high AMI values for pairs with other e-ECDGs clusterings (also computed using small learning rates) and moderate values with strings and n-grams clusterings. strings clusterings obtain very high AMI scores when pairing with other clusterings
and fairly high scores with n-grams clusterings. n-grams clusterings reach increasingly better AMI scores when pairing with other n-grams clusterings as the number of epochs grows.

<figure><img src="image_106.png" /><figcaption>Figure 5.32 – Heatmap containing the AMI of all pairs of the selected data views.</figcaption></figure>

**Ensemble Clustering** To evaluate ensembles clustering, we selected the 10 ensembles per data view that attained the best silhouette scores, but we discarded those whose embedding learning used a learning rate higher than 0.3. Henceforth, we call *ensemble basis* this set of 30 ensembles, to avoid confusion with the clustering obtained from them, i.e. ensemble clustering (step (iii) in figure 5.8).

By targeting the best silhouette scores and small values for the learning rate, we intended to provide a criterion that is prone to maximize the mutual AMI —and maybe even fortuitously improve the AMI with respect to the ground truth —while being based solely on an internal validation score and a controllable parameter, hence independent from any ground
truth. Table 5.11 shows the complete list of the ensemble basis, including the hyperparameters used during the embedding learning, the clustering algorithm used and the silhouette score obtained.

We tested 5 ensemble clustering methods (ECM), including:

* CSPA

* HPGA

* MCLA

* HBGF

* NMF

Our evaluation was formulated in multiple rounds with increasing number of ensembles. More precisely, the first round included one ensemble per data view (i.e. those with the leading silhouette scores), the second included two ensembles per data view (i.e. those with the leading and second best silhouette scores), and so forth. Thus, our evaluation comprised 10 rounds testing all ECMs, such that the n-th round included the n best (silhouette) scoring ensembles for each data view.

Figure 5.33 shows the execution time of each ECM as function of the number of ensembles. We note that HBGF, HPGA and CSPA finished very quickly (i.e. respectively in ~ 5s, ~ 15s and ~ 30s), while NMF takes a higher (but rather constant) time to compute and MCLA has a considerably large and linearly growing runtime. The linear regressions of the execution times for NMF and MCLA (that respectively attained R² = 0.810 and R² = 0.903) are:

$$ \hat{t}_{\text{nmf}} = 1101.92 + 61.94 \cdot n_{\text{ens}} \qquad (\text{NMF}) $$

$$ \hat{t}_{\text{mcla}} = -3261.09 + 423.67 \cdot n_{\text{ens}} \qquad (\text{MCLA}) $$

these equations show that the computation time for NMF increased by roughly 1 minute per additional ensemble, while MCLA increased by roughly 7 minutes.

Figure 5.34a contains a heatmap table showing the AMI of all pairs of ensembles in the ensemble basis. We can see three lighter blocks in the main diagonal, showing that ensembles of the same data views attain higher AMI. Outside of the main diagonal, we also see that the regions having pairs of strings and n-grams data views are lighter than those having e-ECDGs. For these mixed pairs, e-ECDGs ensembles attain lower AMI with n-grams or strings (~ 0.6) than the mixed pairs having n-grams and strings, which attain moderate
<figure><img src="image_107.png" /><figcaption>Figure 5.33 – Benchmark of the ensembles clusterings per number of ensembles.</figcaption></figure>

AMI scores (~ 0.75). Although the ensemble basis elements were chosen only with basis on their silhouette scores and learning rates, these scores are in consonance with the presented previously larger analysis about the mutual information of data views.

The results of all AMI scores of the ensemble clusterings with respect to the instances of the ensemble basis are presented in figure 5.34b. We note that the best overall AMI scores were obtained with NMF (~ 0.72) and MCLA (~ 0.76) ECMs.

To measure the information gain attained with the ensemble clustering approach, we computed the average AMI considering all (ensemble) clusterings obtained with each ECM (i.e. one clustering per round). We also compute the average AMI between the ensembles of each data view; however in this case, to avoid a selection bias we did not count the AMI score of ensemble pairs of the same data view. The results of this measurements are presented in table 5.12. The heatmap table 5.35 shows the average AMI per category (i.e. data view or ECM) —including the AMI values for clusterings in the same category and across those generated with different ECMs, for the sake of completeness.
<table><caption>Table 5.11 – List of ensembles in the ensemble basis</caption><thead><tr><th rowspan="2"></th><th colspan="4">Hyperparameters</th><th rowspan="2">Algo</th><th rowspan="2">Silhouette Score</th></tr><tr><th>n or wl-iterations</th><th> dims </th><th> epochs </th><th> learning rate </th></tr></thead><tbody><tr><td rowspan="10">e-ECDGs</td><td>31</td><td>768</td><td>90</td><td>0.020</td><td>HDBSCAN</td><td>0.631</td></tr><tr><td>32</td><td>763</td><td>90</td><td>0.020</td><td>HDBSCAN</td><td>0.630</td></tr><tr><td>21</td><td>128</td><td>15</td><td>0.025</td><td>HDBSCAN</td><td>0.605</td></tr><tr><td>3</td><td>1029</td><td>10</td><td>0.025</td><td>HDBSCAN</td><td>0.594</td></tr><tr><td>2</td><td>128</td><td>11</td><td>0.025</td><td>HDBSCAN</td><td>0.579</td></tr><tr><td>2</td><td>896</td><td>10</td><td>0.025</td><td>HDBSCAN</td><td>0.573</td></tr><tr><td>2</td><td>1019</td><td>10</td><td>0.025</td><td>HDBSCAN</td><td>0.572</td></tr><tr><td>2</td><td>507</td><td>70</td><td>0.06</td><td>HDBSCAN</td><td>0.520</td></tr><tr><td>32</td><td>1280</td><td>10</td><td>0.055</td><td>HDBSCAN</td><td>0.503</td></tr><tr><td>32</td><td>1280</td><td>11</td><td>0.06</td><td>HDBSCAN</td><td>0.499</td></tr><tr><td>3</td><td>763</td><td>85</td><td>0.09</td><td>HDBSCAN</td><td>0.484</td></tr><tr><td rowspan="10">strings</td><td>-</td><td>256</td><td>40</td><td>0.127</td><td>HDBSCAN</td><td>0.590</td></tr><tr><td>-</td><td>896</td><td>40</td><td>0.188</td><td>HDBSCAN</td><td>0.589</td></tr><tr><td>-</td><td>512</td><td>45</td><td>0.116</td><td>HDBSCAN</td><td>0.588</td></tr><tr><td>-</td><td>768</td><td>45</td><td>0.031</td><td>HDBSCAN</td><td>0.588</td></tr><tr><td>-</td><td>1024</td><td>45</td><td>0.225</td><td>HDBSCAN</td><td>0.588</td></tr><tr><td>-</td><td>384</td><td>45</td><td>0.224</td><td>HDBSCAN</td><td>0.587</td></tr><tr><td>-</td><td>512</td><td>45</td><td>0.149</td><td>HDBSCAN</td><td>0.587</td></tr><tr><td>-</td><td>512</td><td>35</td><td>0.145</td><td>HDBSCAN</td><td>0.587</td></tr><tr><td>-</td><td>640</td><td>45</td><td>0.009</td><td>HDBSCAN</td><td>0.587</td></tr><tr><td>-</td><td>1152</td><td>45</td><td>0.234</td><td>HDBSCAN</td><td>0.586</td></tr><tr><td rowspan="10">n-grams</td><td>8</td><td>896</td><td>10</td><td>0.057</td><td>HDBSCAN</td><td>0.300</td></tr><tr><td>8</td><td>1024</td><td>10</td><td>0.157</td><td>HDBSCAN</td><td>0.300</td></tr><tr><td>8</td><td>640</td><td>10</td><td>0.161</td><td>HDBSCAN</td><td>0.300</td></tr><tr><td>8</td><td>384</td><td>10</td><td>0.01</td><td>HDBSCAN</td><td>0.300</td></tr><tr><td>5</td><td>1024</td><td>10</td><td>0.021</td><td>OPTICS</td><td>0.198</td></tr><tr><td>8</td><td>256</td><td>15</td><td>0.056</td><td>HDBSCAN</td><td>0.182</td></tr><tr><td>8</td><td>896</td><td>15</td><td>0.228</td><td>HDBSCAN</td><td>0.182</td></tr><tr><td>8</td><td>768</td><td>15</td><td>0.093</td><td>HDBSCAN</td><td>0.181</td></tr><tr><td>8</td><td>640</td><td>15</td><td>0.112</td><td>HDBSCAN</td><td>0.181</td></tr><tr><td>8</td><td>768</td><td>15</td><td>0.225</td><td>HDBSCAN</td><td>0.181</td></tr></tbody></table>
<figure><img src="image_108.png" /><figcaption>(a) Pairs of ensembles in the ensemble basis.</figcaption></figure>

<figure><img src="image_109.png" /><figcaption>(b) Pairs of ensembles from the ensemble basis and the set of ensemble clusterings (mutually exclusive).</figcaption></figure>

<caption>Figure 5.34 – Heatmap table containing AMI values for clustering pairs.</caption>
<table><caption>Table 5.12 – Ensemble Clusterings: AMI between groups of clusterings</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Mean AMI</th><th colspan="3">e-ECDGs</th><th colspan="3">strings</th><th colspan="3">n-grams</th></tr><tr><th>MEAN</th><th>MIN</th><th>MAX</th><th>MEAN</th><th>MIN</th><th>MAX</th><th>MEAN</th><th>MIN</th><th>MAX</th></tr></thead><tbody><tr><td>e-ECDGs</td><td>0.590 (±0.0346)</td><td>-</td><td>-</td><td>-</td><td>0.591 (±0.034)</td><td>0.548</td><td>0.647</td><td>0.589 (±0.035)</td><td>0.547</td><td>0.652</td></tr><tr><td>strings</td><td>0.675 (±0.0897)</td><td>0.591 (±0.034)</td><td>0.548</td><td>0.647</td><td>-</td><td>-</td><td>-</td><td>0.760 (±0.026)</td><td>0.548</td><td>0.800</td></tr><tr><td>n-grams</td><td>0.675 (±0.091)</td><td>0.589 (±0.035)</td><td>0.547</td><td>0.652</td><td>0.760 (±0.026)</td><td>0.548</td><td>0.800</td><td>-</td><td>-</td><td>-</td></tr><tr><td>mcla</td><td>0.760 (±0.0979)</td><td>0.652 (±0.034)</td><td>0.602</td><td>0.724</td><td>0.871 (±0.045)</td><td>0.753</td><td>0.915</td><td>0.758 (±0.039)</td><td>0.670</td><td>0.830</td></tr><tr><td>nmf</td><td>0.7189 (±0.0442)</td><td>0.690 (±0.019)</td><td>0.663</td><td>0.735</td><td>0.773 (±0.011)</td><td>0.753</td><td>0.795</td><td>0.694 (±0.030)</td><td>0.622</td><td>0.741</td></tr><tr><td>hgpa</td><td>0.645 (±0.0485)</td><td>0.644 (±0.023)</td><td>0.611</td><td>0.684</td><td>0.700 (±0.004)</td><td>0.692</td><td>0.709</td><td>0.591 (±0.024)</td><td>0.550</td><td>0.626</td></tr><tr><td>cspa</td><td>0.611 (±0.0390)</td><td>0.592 (±0.023)</td><td>0.548</td><td>0.637</td><td>0.657 (±0.008)</td><td>0.641</td><td>0.676</td><td>0.583 (±0.026)</td><td>0.531</td><td>0.617</td></tr><tr><td>hbgf</td><td>0.600 (±0.0513)</td><td>0.582 (±0.030)</td><td>0.509</td><td>0.636</td><td>0.661 (±0.022)</td><td>0.609</td><td>0.686</td><td>0.558 (±0.028)</td><td>0.482</td><td>0.599</td></tr></tbody></table>

<figure><img src="image_110.png" /><figcaption>Figure 5.35 – Heatmap table containing the average AMI scores across clusterings in different categories (i.e. data views/ECM).</figcaption></figure>
**Ensemble Clustering versus Ground Truth**

To analyze the outcome of ensemble clustering in the partitioning of the annotated dataset, we compared the clustering with best mean AMI$_{gt}$ against the Yara-based ground truth—we notate this AMI as AMI$_{gt}$ to avoid confusion with the AMI between pairs of ensembles as done previously.

The ensemble clustering with best AMI$_{gt}$ is *ensembles-9-mcla*, which corresponds to the instance that uses mcla as ECM and is computed using the 9 best silhouette-scoring ensembles for all three data views.

Furthermore, we also compared the scores of *ensembles-9-mcla* against the clusterings of the *ensemble basis* that have the highest AMI$_{gt}$ for each data view, to evaluate the effects of ensemble clustering.

Tables 5.13 shows the clusterings of the *ensemble basis* per data view that have the highest AMI$_{gt}$ scores—their ID are respectively *annotated-ecdg-2-1019-10-0.025-HDBSCAN*, *annotated-strings-256-40-0.127-HDBSCAN* and *annotated-ngrams-8-1024-10-0.157-HDBSCAN*.

<table><caption>Table 5.13 – List of ensembles in the *ensemble basis* with highest AMI$_{gt}$</caption><thead><tr><th rowspan="2">Data View</th><th colspan="4">Hyperparameters</th><th rowspan="2">Algo</th><th rowspan="2">AMI<sub>gt</sub></th></tr><tr><th>n or wl-iterations</th><th> dims</th><th>epochs</th><th>learning rate</th></tr></thead><tbody><tr><td>e-ECDGs</td><td>2</td><td>1019</td><td>10</td><td>0.025</td><td>HDBSCAN</td><td>0.6941</td></tr><tr><td>strings</td><td>-</td><td>256</td><td>40</td><td>0.127</td><td>HDBSCAN</td><td>0.7634</td></tr><tr><td>n-grams</td><td>8</td><td>1024</td><td>10</td><td>0.157</td><td>HDBSCAN</td><td>0.7575</td></tr></tbody></table>

Table 5.14 shows the scores for the selected clusterings. We see that *ensembles-9-mcla* improves on the clusterings directly computed from the individual data views for all supervised metrics. Furthermore, we observe that the level of noise is remarkably lower for *ensembles-9-mcla* than for the other counterparts.

Figure 5.36 shows the heatmap drawn from the contingency matrix of *ensembles-9-mcla* with respect to the Yara-based ground truth. Each row corresponds to one individual cluster in *ensembles-9-mcla*, each column corresponds to an individual Yara rule, and the cell at (i, j) corresponds to the number of instances that belong to the i-th clusters in *ensembles-9-mcla* and the j-th Yara rule. We see that (i) only two clusters in *ensembles-9-mcla* (around indexes 39 and 249) mix a large amount of samples matched with different Yara rules; (ii) several cases of vertically aligned points (in similar colors) appear, which indicates that *ensembles-9-mcla* is able to split files matched by a single Yara rule in multiple clusters, thus working as a “sub-family” refinement.
<table><caption>Table 5.14 – Supervised metrics of chosen clustering instances with highest AMI<sub>gt</sub></caption><thead><tr><th>Metric</th><th>ensembles-9-mcla</th><th>annotated-ecdg-2-1019-10-0.025-HDBSCAN</th><th>annotated-strings-256-40-0.127-HDBSCAN</th><th>annotated-ngrams-8-1024-10-0.157-HDBSCAN</th></tr></thead><tbody><tr><td>AMI<sub>gt</sub></td><td>0.7817</td><td>0.6941</td><td>0.7634</td><td>0.7575</td></tr><tr><td>NMI<sub>gt</sub></td><td>0.8079</td><td>0.7310</td><td>0.7925</td><td>0.7793</td></tr><tr><td>homogeneity score<sub>gt</sub></td><td>0.7113</td><td>0.6469</td><td>0.6992</td><td>0.7686</td></tr><tr><td>completeness score<sub>gt</sub></td><td>0.9348</td><td>0.8403</td><td>0.9145</td><td>0.7903</td></tr><tr><td>v-measure score<sub>gt</sub></td><td>0.8079</td><td>0.7310</td><td>0.7925</td><td>0.7793</td></tr><tr><td>rand score<sub>gt</sub></td><td>0.5411</td><td>0.4271</td><td>0.5270</td><td>0.4287</td></tr><tr><td>Fowlkes-Mallows score<sub>gt</sub></td><td>0.6115</td><td>0.5042</td><td>0.5905</td><td>0.4775</td></tr><tr><td># clusters</td><td>278</td><td>160</td><td>295</td><td>155</td></tr><tr><td># noise</td><td>11</td><td>486</td><td>774</td><td>3382</td></tr></tbody></table>
<figure><img src="image_111.png" /><figcaption>Figure 5.36 – Heatmap of the contingency matrix for *ensembles-9-mcla* (rows) versus the Yara-based ground truth (columns).</figcaption></figure>
## 5.6 Discussion

This section discusses the results obtained in the experimental section. Our experiments were carried out in two main segments: $\sigma^{\text{ECDG}}$-Learning with SVM Regression (section 5.5.3, page 218) and EMB-DUET (section 5.5.4, page 227). Both have numerical embedding learning as core, but the former targeted the transposition of $\sigma^{\text{ECDG}}$ from graphs to numerical vectors, while the latter uses numerical embedding as data view for ensemble clustering.

In the first experimental segment, as main contribution we assessed the primary influence of SVM parametrization ($\mathcal{P}_{\text{SVM}}$) on the quality of the $\sigma^{\text{ECDG}}$ transposition. In particular, the choice of the SVM Kernel has shown to be a key factor in overall learning performance. The other factors (i.e. parameters for embedding learning ($\mathcal{P}_{\text{emb}}$), size of the training dataset ($s_{\text{train}}$) and size of the test dataset ($s_{\text{test}}$)) were overshadowed by $\mathcal{P}_{\text{SVM}}$ for the ranges considered in our random search.

We tested the Polynomial and RBF kernels for the SVM Regression experiments. The MAE obtained with the Polynomial kernel were relatively steady with respect to the setup of the degree parameter; furthermore, the Polynomial kernel was greatly outperformed by the RBF kernel. Presumably, the Polynomial kernel does not inherently fit $\sigma^{\text{ECDG}}$-Learning data.

The RBF Kernel attained the best scores for MAE. These scores were however still relatively high—which deterred us from using $\sigma^{\text{ECDG}}$-Learning with SVM Regression in the EMB-DUET experiments. Yet, figure 5.20 (page 225) shows that MAE score can be improved by using larger training datasets in the learning phase. Thus, a possible alternative to improve regression results is to substantially increase the training data (i.e. $s_{\text{train}}$) size.

In the second experimental segment we analyzed EMB-DUET. We considered three data views (i.e. e-ECDGs, strings, n-grams), which were chosen due to their complementary characteristics. e-ECDGs provides a structural feature that aims to provide a behavioral description of the files, whereas strings and n-grams are both syntactical-oriented features, yet very distinct from each other.

One major motivation for EMB-DUET was the speedup of the data analysis time so as to render it scalable. In our experiment we assessed a linear time growth (with respect to the number of epochs) for the embedding computation when using Doc2Vec and a bilinear growth (with respect to the number of epochs and WL-iterations) when using Graph2Vec. This is an auspicious result for the scalability of EMB-DUET, notwithstanding the impact that vocabulary size may have on the time necessary to learn the numerical embedding. Nonetheless, this time is dominated by the construction of the vocabulary, which can be
reused and updated only every so often, and it can also be limited in size, discarding terms
that are less relevant for the learning —but we did not study this possible optimization
here.

Our experiments showed that computing clustering on embedded vectors is very effi-
cient; for instance, the average time to compute clusters for 27127 embedded vectors was
below 3 minutes using HDBSCAN (c.f. table 5.6, page 252). In our benchmark HDBSCAN
greatly outperformed OPTICS, as it attained an overall speedup factor of 31, 1 over OPTICS.
Furthermore, HDBSCAN generated the clusterings with the best (internal and external) scores
in our experiments. Therefore, HDBSCAN has shown to be an excellent choice for comput-
ing the ensembles of the data views.

Processing multiple data views in the analysis increases the computation cost. However, this computation is highly parallelizable, since the data views are independent from each other. Therefore, the critical path for the ensembles computation corresponds to the time taken by the slowest of them; in our experiments it was roughly the time taken to compute (without optimization) the numerical embedding for n-grams.

We conducted a lighter version of the AnR evaluation (c.f. section 4.4.3, page 165). The target metrics for accuracy were the AMI and the homogeneity score obtained by the ensembles, while the robustness phase also took into account Fowlkes-Mallows index, completeness score and silhouette score.

The accuracy analysis achieved fairly positive results; figure 5.25 (page 236) shows the
correlation of AMI and homogeneity scores for the HDBSCAN clusterings. We note that the
clusterings of the *strings* data view attained the highest AMI scores, which means that they
are able to (autonomously) capture the information contained in the Yara rules adopted as
ground truth. This is coherent with the fact that Yara rules in the overwhelming majority
of cases target *strings* to define matching criteria. We also note that the clusterings of the
*n-grams* data view attained the highest homogeneity scores, which is congruent with the
vocabulary size.

We emphasize that the external evaluation takes the selected Yara rules (from Malpedia and other sources) as ground truth. Despite our effort to winnow our set of Yara rules to the ones with highest quality only, they may have their own imperfections; for instance, we had 282 matches (out of 13,065) with files that belonged to our cleanware dataset. However, since we perform an unsupervised analysis, maximizing the AMI and homogeneity scores is not a central part of our objectives.

The robustness analysis shows minor differences in the trend plots of all metrics, notwith-
standing great differences in size between the full and annotated datasets (i.e. with ratios of 1,5087×, 2,0697× and 2,0768× for the e-ECDGs, strings and n-grams data views, respectively). It therefore demonstrate that the use of embedding learning does not degrade the inherent robustness afforded by data views.

We analyzed the correlation of internal and external scores, so as to make propitious selection of ensembles when no ground truth is available —which is the regular use case. It demonstrated a positive correlation between AMI and silhouette score, thus evincing the relationship between both metrics. However, this analysis also revealed the importance of the learning rate; in our experiments, the optimal learning rate was found to be around 0.23, presenting a negative impact for higher values (which generated clusterings with high silhouette score and low AMI).

The analysis of the mutual information of the data views assessed two distinct scenarios: the intrinsic analysis, which considers only the pairs of clusterings from the same data view, and the extrinsic analysis, which considers only the pairs of clusterings from the two different data views.

The syntactic-based data views (i.e. strings and n-grams) attained higher intrinsic AMI scores, which means that there is less variation across different clusterings of these data views. This corroborates the general rule of thumb that considers syntactic features more precise than semantic features. Furthermore, the syntactic-based data views also attained higher extrinsic AMI scores when evaluated against them (instead of e-ECDGs); this means that they share more inherent similarities, which is also coherent, as both data views are syntactic-based.

In the analysis of the ensemble clustering, we tested 5 ECMs. The best results were obtained with MCLA and NMF; both were able to greatly improve the mean extrinsic AMI of the data view. In other words, the ensemble clustering was able to learn from the entire set of data views, thus retaining the preeminent properties of each data view, that are the precision of the strings, homogeneity of the n-grams and the robustness of the e-ECDGs.

The execution time for both ECMs grew linearly with respect to the number of ensembles in our experiments. Nonetheless, the execution time to compute the ensemble clustering remained very adequate (even for $n_{ens} = 30$), thus showing to be suitable for the practical use case. Moreover, in an actual use, the number of ensembles to cluster may well be much less than the 30 chosen for benchmark in our analysis. This is an important result for the scalability of the overall framework, because the ensemble clustering is part of the whole computation critical path.
Finally, we analyzed the instance of ensemble clustering (*ensembles-9-mcla*) that obtained the highest extrinsic AMI, thus obtaining the maximum information across all data views in our experiments. Indeed, *ensembles-9-mcla* improved all external validation scores when compared with the ensembles of the individual data views with highest silhouette score.

We note that the use of ensemble clustering precludes the internal validation of clustering, because the metrics for this intent are based on the notion of distances between data points. In our case no natural notion of distance exist. In fact, the distances are computed in the context of each data view and thus comprised in the labels produced by the clusterings. When ECM acts on the ensembles, distances are disregarded and only labels are taken into account.

Another positive outcome of the ensemble clustering was the reduced amount of noise. It was able to reduce the number of noise samples from a few hundreds of data points to dozens (e.g. only 11 singletons in the case of *ensembles-9-mcla*). This demonstrated the ability of ensemble clusterings to handle missing values in datasets.

### 5.6.1 Threats to Validity

This section discusses the validity [244] of our study according to: construct, internal, external validity and reliability.

*Construct Validity:* we have performed controlled experiments that allowed us to assess the clustering properties (i.e. accuracy, robustness and scalability) targeted by our methodology. Our evaluation dataset was painstakingly built to allow appraising the performance of the proposed framework with respect to the targeted properties and the experiment followed closely the proposed methodology; in this regard, no issue should arise from our experiential study.

*Internal Validity:* our analysis is derived from the results obtained from a very large parametric space, which can not be fully explored. Thus, we have access to a limited range of data points, despite our effort to cover the parametric space as much as possible using the random search optimization. This may affect the completeness of our analysis, nonetheless it does not impact soundness.

*External Validity:* all techniques used here for embedding learning, SVM regression and ensemble clustering are use-case agnostic, being applied without any *ad-hoc* customization to our scenario. Therefore, there should be no issue with generalization of the methodologies proposed in this study. We strove to compose a proper evaluation dataset from real
world malware and cleanware instances and a curated list Yara rules; nonetheless, as the true profile of the malware and cleanware population is unknown, misrepresentation remain possible, which could engender a selection bias. This is withal an unmanageable issue, which is a common concern in malware research.

*Reliability*: our study is based on a set of methods and techniques that are of public knowledge and well established in the literature. Furthermore, we provided all details for proper parametrization, whenever it was the case. We constructed our evaluation dataset with publicly available Yara rules, which allows to replicate the generation process of a new dataset that should display the same properties as ours.

**5.7 Conclusion**

This chapter proposes and evaluates a new malware classification framework called EMB-DUET, which is able to handle different data views obtained from the dataset under analysis. EMB-DUET builds on DUET, which leverages ensemble clustering to consolidate multiple clusterings into a single one; however, EMB-DUET adds a numerical embedding layer to normalize the input data and boost overall framework performance. Ancillary to EMB-DUET, we also proposed and tested $\sigma^{\text{ECDG}}$-Learning with SVM Regression, which targets the transposition of $\sigma^{\text{ECDG}}$ to ECDGs in their vectorized form, instead of the original graph format.

Our experiments with EMB-DUET demonstrated that the use of embedding learning for feature vectorization does not encumber the original properties of the feature, as accuracy and robustness. However, this transformation is able to boost data analysis (e.g. clustering), as vectors are not only compact units of data, easy to handle in large numbers, but are also friendly to machine learning algorithms. We also demonstrated that ensemble clustering is able to coalesce the partitions from different ensembles into a single clustering, which attains a higher mutual information with each original ensemble than the mutual information between the original ensembles themselves.

The analysis revealed the importance and impact of different hyperparameters for EMB-DUET as well as $\sigma^{\text{ECDG}}$-Learning with SVM Regression. In the case of $\sigma^{\text{ECDG}}$-Learning with SVM Regression, the experiments evinced that the RBF kernel fits considerably better the regression than the Polynomial kernel. Kernel hyperparametrization was the leading performance factor for the range of values considered in our random search optimization, overshadowing hyperparametrization for the embedding learning and dataset sizes. For EMB-DUET, we ascertained the relationship between internal and external validation scores. Experiments
showed a (potential) positive correlation between silhouette score and AMI with respect to the Yara-based ground truth. However for this correlation to hold it was necessary to properly setup the learning rate of the Graph2Vec algorithm —for learning rate equal to 0.23, the Pearson coefficient measuring the correlation between AMI and silhouette score was 0.928.

The analysis of EMB-DUET also substantiated some empirical notions related to data views. Data views based on syntactical features (i.e. *strings* and *n-grams*) shared a higher AMI and attained higher precision and homogeneity with respect to the Yara-based ground truth. However, the semantic-based data view (i.e. *e-ECDGs*) obtained better silhouette scores and more malleability with respect to the hyperparametrization, which denotes its ability to handle highly diversified datasets (thus more likely to contain a higher rate of outliers). Thanks to ensemble clustering, all data views could be consolidated into a single cluster that improved the entire set of external validation scores while greatly reducing the amount of noise when compared with the original ensembles (c.f. table 5.14, page 250).

Besides accuracy and robustness (inherent to the choice of data views), *scalability* was another important characteristic sought for the proposed framework. The vocabulary size and chosen number of epochs for training were confirmed to be a burden on the execution time of the embedding learning with Doc2Vec and Graph2Vec. In case of Graph2Vec, the parametrization of the *WL-iteration* has also a substantial impact on the execution time. Yet, we were able to analyze a considerably large dataset (~ 27K samples) in few hours, considering only a single setting of embedding learning and clustering. The analysis showed a multi-linear growth rate (with manageable constant factors) to parallelize the execution time with respect to the set of hyperparameters.

We fortuitously verified the superiority of HDBSCAN over OPTICS. We used both clustering methods in our evaluation because of their autonomy in the process of cluster computation, which does not need any major parametrization to work —the only, optional, parameter that we set was the minimum number of data points to form a cluster. The results obtained with HDBSCAN were frequently better (for both for internal and external scores), and execution time was remarkably shorter, attaining an overall speedup factor of 31, 1.

As main contributions of this chapter:

(i) we proposed EMB-DUET as a data-driven framework that is able to handle heterogeneous features, which leverages numerical embedding to boost overall performance (c.f. section 5.4.1, page 208);

(ii) we verified the accuracy and robustness afforded by the aggregation of the different data views (i.e. *strings*, *n-grams* and *e-ECDGs*) (c.f. section 5.5.4, page 236);
(iii) we analyzed the impact and effects of different hyperparameters involved in EMB-DUET and $\sigma^{ECDG}$-Learning with SVM Regression, so as to provide a guideline for proper parameter setting (c.f. sections 5.5.4 and 5.5.3, page 227 and 218).

Additionally, we have contributions that derive from our experimental setup, i.e.:

(i) the deployment and evaluation of the Cuckoo framework for sandboxing, in particular the development of our custom report module that outputs the ECDG of the analyzed file and the related benchmarks obtained during the use of Cuckoo for the experiments;

(ii) the procedure developed to build the dataset, in particular the proposal of the *match-ing graph* as a means to refine the dataset labeling process.
CONCLUSION & FUTURE WORK

In this thesis we studied the problem of malware classification, in particular we endeavor
to develop methods and methodology that can respond to the modern scenario of multi-
tudinous and highly complex malware. This situation requires building a malware analysis
framework that is highly automated and autonomous —where the autonomy requirement
is particularly important to guarantee a true independence from human intervention (c.f.
section 1.3.2, page 50).

The bedrock of our malware classification framework are methods and methodologies of machine learning. Therefore, before tackling the technical matter of our research, we revisited and discussed the epistemological concerns related to malware research and machine learning (c.f. chapter 1, page 25).

Initially we confronted the matter of malware ontology (c.f. section 1.2, page 26) in order
to identify the basis of our approach to characterize malware instances. As a result of this
analysis, we formulated a realist malware ontology (c.f. section 1.2.2, page 42) that comprises
a dual facet (the *realization level* and the *teleological function*) which guided our axiology (c.f.
section 1.4, page 54).

Then, we examined the prevalent epistemological basis in malware research (c.f. sec-
tion 1.3.2, page 50), which allowed us to identify the preponderance of *Foundationalism* in
the wrought structure of knowledge in the field. We also brought to the light of malware re-
search the JTB definition of knowledge (c.f. section 1.3.1, page 45), which spurred the analy-
sis that pinpoints the pitfalls of external justification in the use of machine learning methods.
Finally, we took the argument that security is fundamentally a social discipline, which con-
ditions the intractability of entirely supplanting human acumen by machines, despite the
effort to make systems as autonomous as possible. These epistemological considerations
are also taken into account in our axiology.

We then took an accruing approach that followed the stages of our malware classifica-
tion workflow: enhancement of tooling for malware analysis using symbolic execution for
call tracing (chapter 3, page 111), study of ECDGs to represent the behavior of binary codes
(chapter 4, page 153), and the proposal of EMB-DUET as a polyvalent framework design
(chapter 5, page 189), which leverages the technique of numerical embedding to efficiently
handle big corpus of multifarious data and in addition uses methods of ensemble clustering, which are able to consolidate heterogeneous data in a single clustering.

The initial stage targeted the use of symbolic execution as an alternative method for binary analysis. Symbolic execution implements a very different technique than traditional static and dynamic methods, which are much more frequently evaded by malware. Our focus was to leverage symbolic execution to trace calls made by programs during their execution —which is essential to ECDGs, our graph-based behavioral representation of programs, studied in the chapter 4 (page 153)—therefore we investigated which are the effects of different heuristic parameterizations on SMT solvers and on the overall analysis. Our experiments (c.f. section 3.6, page 147) showed that:

(i) the number of active paths in the symbolic execution should be maximized, provided that this does not cause the analysis to exceed the available memory;

(ii) the SMT optimization (notably the chosen setting of tactics) is effective in enhancing the overall analysis, despite not being the main optimization factor;

(iii) limiting the number of loops in the execution improves the number of unique calls obtained in the analysis, although this resulted in a minor improvement in the quality of our ECDGs;

(iv) other factors (e.g. step timeout, the minimum size of traces) do not have a relevant impact on the analysis outcome.

The next stage analyzed the effectiveness of ECDGs in providing a birthmark representation for the structure of computer programs. Based on ECDGs, we defined a new similarity function ($\sigma^{ECDG}$) (c.f. section 4.4.2, page 164) that, in our efficiency benchmark (c.f. section 4.5.3, page 173), outperformed *radiff2*, a component of the well established open-source suite *radare2*. Our $\sigma^{ECDG}$ also achieved positive results in the evaluation of its accuracy and robustness. Striving for a stronger evaluation methodology, we formulated the *Accuracy and Robustness (AnR) paradigm* (c.f. section 4.4.3, page 165), which supports the use of mixtures of datasets, built according to different heuristic methodologies so as to test different aspects of the framework (i.e. its accuracy and robustness). To the best of our knowledge, no previous work in the literature adopted a similar holistic phased evaluation methodology. Our results (c.f. section 4.5.2, page 169) attained a high accuracy score (0.983), which shows that a $\sigma^{ECDG}$-based framework can autonomously describe malware families as accurately as our set of manually created signatures, and a strong resistance to outliers —hence robustness —when the test dataset intentionally contained a high rate of noise —the NMI score
between the initial and final (highly polluted) clusterings of this phase was 0.974, meaning
that the information contained in the initial clustering is greatly conserved after the insertion
of noise. On top of that, we showed that our framework is able to produce descriptive cluster
prototypes (c.f. section 4.5.6, page 180) which can be leveraged by other machine learning
methods (e.g. semi-supervised learning). Our cluster prototypes can assist the human anal-
ysis with a graphical representation of a common behavior in a set of programs and enhance
classification models used in malware detection due to their resemblance to signatures.

The last stage of our work focused on the data analysis aspect capacities of a malware
classification framework. For this, we proposed EMB-DUET as an improvement of the de-
sign architecture proposed in the DUET system [124]. Like the DUET system, EMB-DUET
takes advantage of ensemble clustering to consolidate multiple partitions of different fea-
tures in a single clustering. However, EMB-DUET features a numerical embedding layer
that vectorizes the different data views, thereby speeding up the whole analysis process (c.f.
chapter 5.4.1, page 208). We also studied the transposition of σ<sup>ECDG</sup> on embedded vectors
computed from ECDGs using Graph2Vec —experimental segment referred as σ<sup>ECDG</sup>-Learning
with SVM Regression (c.f. chapter 5.4.2, page 210). Our experiments showed that the em-
bedding layer does not degrades the properties related to the accuracy and robustness of
the transformed features (c.f. chapter 5.5.4, page 227). We also measured the impact and
effects of different hyperparameters involved in EMB-DUET and σ<sup>ECDG</sup> -Learning with SVM
Regression, to provide a guideline for optimal parameter setting.

We consider that guidelines stipulated by our axiology were followed along our work,
because:

- **Axiology-1**<fnref n="21" />: in this work we did not pursue the hypothesis that existed any *natural* upper classes for software, like malware/cleaware or any innate type of software (e.g. virus, worm, etc). We do rely on constituted malware families for analysis (in chapter 3, page 111) and evaluation (in chapters 4 and 5, pages 111 and 189), but in the perspective of *software birthmarks* as assented by our axiology.

- **Axiology-2**<fnref n="22" />: as we pursued a *data driven* approach to classify malware, we turned to unsupervised machine learning methods (with special regard to internal validation metrics). We still deem valuable the use of ground truth for external validation, but for this we carefully built our evaluation dataset avoiding any outsourced labeling —the

<footnote n="21">21. Axiology-1: pursuing no ontological difference between malware and software at the “realization level”.</footnote>

<footnote n="22">22. Axiology-2: being more coherent driven.</footnote>
only exception is in chapter 3, but the focus there is to improve of binary analysis and
not unsupervised learning for malware classification.

— Axiology-3<sup>23</sup>: as a consequence of our careful preparation of our evaluation dataset for unsupervised learning, we consider that we attained a satisfactory level of doxastic justification.

By and large, we built our malware classification that managed to fulfill our set of ob-
jectives. This included the improvement of tool, methods and methodologies for malware
analysis as well as the creation of a data-driven approach to construct a malware classifica-
tion framework that is accurate, robust and scalable.

Furthermore, our incremental strategy allowed us to take a greedy approach to optimize
each stage of the end-to-end analysis, otherwise we would be confronted with a gigantic
number of parameters to handle. As a consequence, part of the early stages of our workflow
could be re-evaluated as we progressed, which reinforced our analysis and provided more
results to back up our prior conclusions —which was precisely the case with the call tracing
with symbolic execution, and with the use of ECDG as a representation for the behavior of
computer programs.

**5.7.1 Future Work**

Our work comprehends a diversified scope of themes related to malware classification.
As each of these topics can undertake its own direction without interfering with the others,
we split their presentation in separated sections.

Call Tracing with Symbolic Execution

We have several leads to improve the effectiveness of our call tracer. A patent idea is to evolve our call tracer to run a full-fledged *concolic* execution, which carries out concrete and symbolic analysis simultaneously [17]. The goal is to obtain a *deep and wide* exploration of the program state space, relying on the respective strengths of concrete and symbolic execution.

Past works comparing the different analysis methods presented a noticeable improve-
ment in code coverage of concolic execution over (pure) symbolic [175, 271]. Toward this
optimization, we tested the integration of angr with unicorn [227]; our preliminary experi-
ments with full concrete execution obtained a minor speedup with at the expense of lower

<footnote n="23">23. Axiology-3: being more doxastically justified.</footnote>
code coverage. This trial evinced that the strategy to duly interleave both execution methods
is a complex topic, which deserves a full study dedicated to this task.

To improve our symbolic call tracer for the generation ECDGs, another noteworthy en-
hancement is to implement taint analysis targeting the symbolic variables in order to track
the data flow during the execution. In our work, we rely on the def-use dependence as de-
fined by Christodorescu et al [47] to build the edges of the ECDGs. This def-use strategy is
based on the *data type* and the *data value* of the common arguments. However, with the
taint analysis —facilitated by the symbolic execution —we should be able to improve this
definition to attain a finer strategy to connect calls, based on a finer analysis of data depen-
dancies.

Testing other granularity levels is also an axis for improvement. Our current granularity considers the whole binary code, but with symbolic execution it is possible to explore smaller chunks of the program (e.g. a single function). By breaking down the scope of analysis, it is possible to largely parallelize the analysis and to keep the burden on systems (e.g. memory consumption) under a controlled level.

As for SMT optimization, there are several re-assessments that can be explored in other
contexts to polish the interpretation of our results. It would be interesting to implement
the proposed optimizations into other tools exploiting similar combinations of formal and
heuristic approaches for software analysis [10, 18, 29, 62, 134] with the purpose of un-
derstanding the presence of domain-specific results and to evaluate the possibility of gener-
alizing these results.

Structural-Based Binary Code Similarity

Working with graph-based similarities of binary codes is in general costly due to the
graph matching step involved, as it frequently requires solving instances of NP-complete or
NP-hard problems. Our choice of gSpan has proven to be rewarding, but it would be interest-
ing to test alternative approaches for frequent subgraph mining—the survey of T. Ramraja
and R. Prabhakarb lists alternative algorithms [229].

Other strategies such as Deep Graph Learning can also be considered to address the
problem of frequent subgraph mining. Although partially explored in our last chapter (c.f.
section 5, page 189), further exploring how to use numerical embedding and other learning
techniques to learn about inner structures of graphs, thus working around the problem of
graph matching, is also worth investigating [173].

Different methods to measure the graph similarity (or distance) can also be adopted,
such as the (widely used) Graph Edit Distance (GED), Markov Chain Monte Carlo sampling
technique [161], adjacency matrix rewriting to represent graph as a least-square problem [322],
or Graph Convolutional Networks [208].

Furthermore, exploring other graph mining techniques [232] such as graph clustering
and graph classification (e.g. [144]) to analyze ECDGs can also be considered, withal this
would require changing the $\sigma^{ECDG}$ measure of similarity. This leads to manifold directions that
can leverage ECDGs to explore the similarity of binary codes beyond $\sigma^{ECDG}$, which is a coarse-
granularity similarity function (i.e. it takes into account the entire graph and produces a
unique similarity measure as outcome).

Alternatively, it is possible to explore the similarity of ECDGs differently, taking into ac-
count multiple and spread matches in the graphs, which could reveal multiple behaviors
subsumed in EDCG. This approach contrasts with ours, as our approach targets the largest
common subgraph —roughly the “largest common behavior”, which is not necessarily the
most relevant one (despite making for a good software birthmark in practice)—whereas an
approach that looks for (multiple and) decentralized similarities in (sub-)graphs could ob-
tain very different results.

Enhanced versions for ECDGs is also a possible topic for further research. We partially
explored such an idea with the fusion of ECDGs obtained from symbolic and (pure) dynamic
analysis (c.f. section 5, page 189), but the closer investigation of the effects and optimization
of this (and other) composition(s) remains open. It is also possible to study the association of
calls (i.e. the nodes of the graph) to a set of categories (similarly to some previous works [85,
112]), which is an auspicious alternative to get more compact graphs and even to progress
towards an agnostic description that is able to represent behaviors of different types of pro-
grams (e.g. Windows PE, ELF Linux, APKs, scripts, etc).

## EMB-DUET

Our work on EMB-DUET can be improved for thorough completion. Due to the com-
putation cost of the random search exploration and the lack of initial knowledge about the
impact magnitude of the hyperparameter settings in the final results, in our study we fol-
lowed a fixed design process [244]. Thus, new exploratory settings can still provide a deeper
understanding about the effects of different hyperparametrizations of the learning layers (i.e.
embedding learning + partition clustering) on the final ensemble clustering.

A major milestone yet to be achieved is the full integration of EMB-DUET with $\sigma^{ECDG}$ -
learning. However, a successful integration presumes the improvement of $\sigma^{ECDG}$ -learning to
very low MAE scores in order to avoid propagating regression errors to the clustering of the
data view. This requires at first place to further improve the SVM hyperparametrization and
to enlarge the training dataset. Further improvements on the embedding learning hyper-
parametrization may also attain an important impact on performance of σ<sup>ECDG</sup> -learning, but
their effects were outweighed by the other components of the workflow. Alternatively, new
regression approaches can be used such as the ensemble model for SVM Regression [43] and
ensemble of deep learning belief networks (DBN) [226], to list but a few possibilities.

Further improvements and enhancements for EMB-DUET are also possible. We consid-
ered only three data views in our analysis, however there is no theoretical upper limit to the
number of features that can be associated, therefore new analysis can explore other combi-
nations of feature sets. There is considerable room for improvement in the clustering step
that computes the ensemble basis, since we restricted the analysis only to HDBSCAN and
OPTICS with a single setting (i.e. minimum number of points to form a cluster set to 5). The
embedding learning step can be also optimized to work with smaller vocabulary sizes and
other hyperparameter settings, which can potentially speed up computation and eventually
improve the quality of the computed vectors.

EMB-DUET also opens new perspectives for future research. The vector representation
of can be exploited for other usages, such as fast malware search, benefiting from this short
vector encoding that also enables efficient lookup. Another perspective would be to evolve
EMB-DUET in order to furnish it with the ability to perform clustering with constraints so
that standard clusterings, which are computed solely with basis on the affinity of the data
points, could adapt to supplementary information available. In practice, this would allow
to integrate the autonomous approach of EMB-DUET with external information (e.g. labels
from signature matching or from expert analysis), thus consolidating every facet of machine
and expert analysis in a final clustering.

Future Perspectives

The field of malware analysis faces the challenging task of assessing the malice of pro-
grams. On the one hand, human expertise is needed to fulfill this complex task, as machine
fail to interpret the nuances of contexts and stratagems behind a cyberattack; on the other
hand, human effort alone cannot keep pace with the exponential growth of cyber threats.
Therefore, the future of malware analysis inevitably goes through the synergy of human and
computer work. The challenge is to find the optimal balance between both and develop
means of cooperation, so that machines are used to facilitate human work as much as pos-
sible.

The leading trend in this direction is *machine learning* —as approached in this thesis.
However, the scenario of malware analysis encompasses epistemological issues that are not
often present in traditional uses of machine learning:

— establishing a universal malware detector is an undecidable problem, which impairs
the composition of a unbiased ground-truth;

— since it depends on the use context (e.g. malware detection), the outcome of the analysis is only tentative at best, because of the possibility of software to malfunction;

— adversaries are by definition malicious, thus leading to an adaptive adversarial model
against the machine learning techniques used in the analysis;

As a consequence, malware research has to deal with these peculiarities if it is to succeed
in a real scenario. This calls for the development of methodologies that can take these issues
into account.

In particular, the adaptive adversarial model<sup>24</sup> has been for a long time an important
issue in malware analysis. To tackle this scenario, the focus in malware research has been
greatly put on dealing with anti-reverse engineering techniques developed by malware au-
thors and on creating signatures as countermeasure. However, in a scenario where machine
learning is a central piece of the security workflow, this traditional focus overly concentrates
on just part of the issue, namely the feature extraction phase of a machine learning-based
analysis. So far, little has been proposed to cope with adversarial machine learning in the
context of malware analysis (e.g. [155]).

Another analysis perspective, complementary to the one we developed in this thesis, is to
consider the grouping of malware instances by cyberattack, i.e. associating instances that do
not necessarily display an inherent kinship (e.g. similar behavior)—as we did here—but that
were seen together in the context of a cyberattack. This approach targets malware instances
that are likely to participate together in a cyberattack, thus working as indicators of the actors
behind the cyberattack; this allows to gain information about the *Tactics, Techniques and
Procedures* of attackers, which can be availed to infer *attribution*.

Convergence is yet another important topic for the future of malware analysis. With the
emergence of new technologies that are being connected to the Internet, cyberattackers have
a larger attack surface to exploit. This allows for a much more complex composition of attack

<footnote n="24">24. Commonly referred as the “cat and mouse game”.</footnote>
vectors, which may well include different technologies (e.g. desktop, mobile, wearable, IoT devices, etc).

Therefore, it will become increasingly necessary to treat a diversified ensemble of tech-
nologies in the analysis. To handle that, agnostic descriptions of software birthmarks will
have to be developed and analysis methodologies will have to evolve to contemplate this
prospective scenario.
<figure><img src="image_112.png" /></figure>
BIBLIOGRAPHY

[1] Adel Abusitta, Miles Q. Li, and Benjamin C.M. Fung, « Malware classification and composition analysis: A survey of recent developments », in: *Journal of Information Security and Applications* 59 (2021), p. 102828, ISSN: 2214-2126, DOI: https://doi.org/10.1016/j.jisa.2021.102828, URL: https://www.sciencedirect.com/science/article/pii/S2214212621000648.

[2] Hira Agrawal et al., « Detection of global, metamorphic malware variants using control and data flow analysis », in: *31st IEEE Military Communications Conference, MILCOM 2012*, Orlando, FL, USA, October 29 - November 1, 2012, pp. 1-6, DOI: 10.1109/MILCOM.2012.6415581, URL: https://doi.org/10.1109/MILCOM.2012.6415581.

[3] Mansour Ahmadi et al., « Novel feature extraction, selection and fusion for effective malware family classification », in: *Proceedings of the sixth ACM conference on data and application security and privacy*, 2016, pp. 183-194.

[4] Frances E. Allen, « Control Flow Analysis », in: *Proceedings of a Symposium on Compiler Optimization*, Urbana-Champaign, Illinois: Association for Computing Machinery, 1970, 1-19, ISBN: 9781450373869, DOI: 10.1145/800028.808479, URL: https://doi.org/10.1145/800028.808479.

[5] Mauricio Almeida et al., « Basic Formal Ontology 2.0 », in: (2015).

[6] Blake Anderson, Curtis Storlie, and Terran Lane, « Improving malware classification: bridging the static/dynamic gap », in: *Proceedings of the 5th ACM workshop on Security and artificial intelligence*, 2012, pp. 3-14.

[7] Nicola Angius, Giuseppe Primiero, and Raymond Turner, « The Philosophy of Computer Science », in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Spring 2021, Metaphysics Research Lab, Stanford University, 2021.

[8] Quynh Nguyen Anh, « Capstone: next generation disassembly framework », in: USA: *BlackHat* (2014).
[9] Mihael Ankerst et al., «OPTICS: Ordering Points to Identify the Clustering Structure», in: *Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data*, SIGMOD '99, Philadelphia, Pennsylvania, USA: Association for Computing Machinery, 1999, 49–60, ISBN: 1581130848, DOI: 10.1145/304182.304187, URL: https://doi.org/10.1145/304182.304187.

[10] Mihael Ankerst et al., «OPTICS: Ordering Points to Identify the Clustering Structure», in: *SIGMOD Rec*. 28.2 (June 1999), 49–60, ISSN: 0163-5808, DOI: 10.1145/304181.304187, URL: https://doi.org/10.1145/304181.304187.

[11] Andrea Aquino, Giovanni Denaro, and Pasquale Salza, «Worst-Case Execution Time Testing via Evolutionary Symbolic Execution», in: *2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)*, 2018, pp. 76–87, DOI: 10.1109/ISSRE.2018.00019.

[12] Robert Arp, Barry Smith, and Andrew D. Spear, *Building Ontologies with Basic Formal Ontology*, The MIT Press, 2015, ISBN: 0262527812.

[13] David Arthur and Sergei Vassilvitskii, *k-means++: The advantages of careful seeding*, tech. rep., Stanford, 2006.

[14] Yara Awad, Mohamed Nassar, and Haidar Safa, «Modeling malware as a language», in: *2018 IEEE International Conference on Communications (ICC)*, IEEE, 2018, pp. 1–6.

[15] Jinrong Bai and Junfeng Wang, «Improving malware detection using multi-view ensemble learning», in: *Security and Communication Networks* 9.17 (2016), pp. 4227–4241, DOI: https://doi.org/10.1002/sec.1600, eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/sec.1600, URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/sec.1600.

[16] Michael Bailey et al., «Automated classification and analysis of internet malware», in: *International Workshop on Recent Advances in Intrusion Detection*, Springer, 2007, pp. 178–197.

[17] Roberto Baldoni et al., «A Survey of Symbolic Execution Techniques», in: *ACM Comput. Surv.* 51.3 (2018).

[18] Sébastien Bardin, Robin David, and Jean-Yves Marion, «Backward-Bounded DSE: Targeting Infeasibility Questions on Obfuscated Codes», in: *2017 IEEE Symposium on Security and Privacy (SP)*, 2017, pp. 633–651, DOI: 10.1109/SP.2017.36.
[19] Clark Barrett, Leonardo De Moura, and Pascal Fontaine, « Proofs in satisfiability modulo theories », in: *All about proofs, Proofs for all* 55.1 (2015), pp. 23–44.

[20] Clark Barrett, Pascal Fontaine, and Cesare Tinelli, *The SMT-LIB Standard - Version 2.6*, website, May 12, 2021, URL: http://smtlib.cs.uiowa.edu/papers/smt-lib-reference-v2.6-r2021-05-12.pdf.

[21] Clark Barrett, Daniel Kroening, and Thomas Melham, *Problem solving for the 21st century: Efficient solver for satisfiability modulo theories*, English (US), Knowledge Transfer Report, Technical Report 3, London Mathematical Society, Smith Institute for Industrial Mathematics, and System Engineering, June 2014.

[22] Ulrich Bayer et al., « Scalable, Behavior-Based Malware Clustering », in: *Proceedings of the Network and Distributed System Security Symposium, NDSS 2009, San Diego, California, USA, 8th February - 11th February 2009*, The Internet Society, 2009, URL: https://www.ndss-symposium.org/ndss2009/scalable-behavior-based-malware-clustering/.

[23] Ulrich Bayer et al., « Scalable, behavior-based malware clustering. », in: *NDSS*, vol. 9, Citeseer, 2009, pp. 8–11.

[24] Najah Ben Said et al., « Detection of Mirai by Syntactic and Behavioral Analysis », in: *2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE)*, 2018, pp. 224–235, DOI: 10.1109/ISSRE.2018.00032.

[25] James Bergstra and Yoshua Bengio, « Random search for hyper-parameter optimization. », in: *Journal of machine learning research* 13.2 (2012).

[26] GNU Binutils, *objdump*, 2021, URL: https://linux.die.net/man/1/objdump.

[27] Nikolaj Bjørner et al., *Programming Z3*, website, URL: https://theory.stanford.edu/~nikolaj/programmingz3.html.

[28] Peter Boonstoppel, Cristian Cadar, and Dawson Engler, « RWset: Attacking path explosion in constraint-based test generation », in: *International conference on Tools and Algorithms for the Construction and Analysis of Systems*, Springer, 2008, pp. 351–366.

[29] Marcus Botacin et al., « Challenges and pitfalls in malware research », in: *Computers & Security* 106 (2021), p. 102287.
[30] Marcus Botacin et al., «Challenges and pitfalls in malware research», in: *Computers & Security* 106 (2021), p. 102287, ISSN: 0167-4048, DOI: https://doi.org/10.1016/j.cose.2021.102287, URL: https://www.sciencedirect.com/science/article/pii/S0167404821001115.

[31] Mark Bradley et al., « GoannaSMT-A Static Analyzer with SMT-based Refinement », in: *Tools for Automatic Program Analysis (TAPAS 2012)*, ENTCS Deauville, France, 2012.

[32] Alexandre Braga et al., « Practical Evaluation of Static Analysis Tools for Cryptography: Benchmarking Method and Case Study », in: *2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE)*, 2017, pp. 170–181, DOI: 10.1109/ISSRE.2017.27.

[33] Rodrigo Rubira Branco, « Ltrace internals », in: *Proceedings of the Linux Symposium*, vol. 1, 2007, pp. 41–52.

[34] Rory Bray, Daniel Cid, and Andrew Hay, *OSSEC host-based intrusion detection guide*, Syngress, 2008.

[35] Thomas Brewster, « An NSA Cyber Weapon Might Be Behind A Massive Global Ransomware Outbreak », in: *Forbes* (May 12, 2017), URL: https://www.forbes.com/sites/thomasbrewster/2017/05/12/rsa-exploit-used-by-wannacry-ransomware-in-global-explosion/?sh=9b5d58be599b (visited on 05/24/2022).

[36] « Burma hit by massive net attack ahead of election », in: *BBC News* (Nov. 4, 2010), URL: https://www.bbc.com/news/technology-11693214 (visited on 05/24/2022).

[37] Cristian Cadar and Koushik Sen, « Symbolic execution for software testing: three decades later », in: *Communications of the ACM* 56.2 (2013), pp. 82–90.

[38] Cristiano Calcagno and Dino Distefano, « Infer: An Automatic Program Verifier for Memory Safety of C Programs », in: *NASA Formal Methods*, ed. by Mihaela Bobaru et al., Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 459–465, ISBN: 978-3-642-20398-5.

[39] Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander, « Density-based clustering based on hierarchical density estimates », in: *Pacific-Asia conference on knowledge discovery and data mining*, Springer, 2013, pp. 160–172.

[40] Davide Canali et al., « A quantitative study of accuracy in system call-based malware detection », in: *Proceedings of the 2012 International Symposium on Software Testing and Analysis*, 2012, pp. 122–132.
[41] John Capps, « The Pragmatic Theory of Truth », in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Summer 2019, Metaphysics Research Lab, Stanford University, 2019.

[42] Sang Kil Cha et al., « Unleashing mayhem on binary code », in: *2012 IEEE Symposium on Security and Privacy*, IEEE, 2012, pp. 380–394.

[43] Snehamoy Chatterjee, Ansuman Dash, and Sukumar Bandopadhyay, « Ensemble support vector machine algorithm for reliability estimation of a mining machine », in: *Quality and Reliability Engineering International* 31.8 (2015), pp. 1503–1516.

[44] Thomas Chen and Jean-Marc Robert, « The Evolution of Viruses and Worms », in: (Dec. 2004), DOI: 10.1201/9781420030884.ch16.

[45] Julia Yu-Chin Cheng, Tzung-Shian Tsai, and Chu-Sing Yang, « An information retrieval approach for malware classification based on Windows API calls », in: *2013 International conference on machine learning and cybernetics*, vol. 4, IEEE, 2013, pp. 1678–1683.

[46] Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea, « S2E: A platform for in-vivo multi-path analysis of software systems », in: *Acm Sigplan Notices* 46.3 (2011), pp. 265–278.

[47] Mihai Christodorescu, Somesh Jha, and Christopher Kruegel, « Mining Specifications of Malicious Behavior », in: *Proceedings of the 6th Joint Meeting of the ESEC and the ACM SIGSOFT SFE*, ESEC-FSE ’07, Dubrovnik, Croatia: Association for Computing Machinery, 2007, 5–14, ISBN: 9781595938114, DOI: 10.1145/1287624.1287628.

[48] Mihai Christodorescu et al., « Semantics-Aware Malware Detection », in: *Proceedings of the 2005 IEEE Symposium on Security and Privacy*, SP ’05, USA: IEEE Computer Society, 2005, 32–46, ISBN: 0769523390, DOI: 10.1109/SP.2005.20, URL: https://doi.org/10.1109/SP.2005.20.

[49] Michael Chui et al., *IoT value set to accelerate through 2030: Where and how to capture it*, Accessed: 2022-05-24, 2021, URL: https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/iot-value-set-to-accelerate-through-2030-where-and-how-to-capture-it.

[50] Emily Chung, « PlayStation data breach deemed in 'top 5 ever' », in: *CBC News* (Apr. 27, 2011), URL: https://www.cbc.ca/news/science/playstation-data-breach-deemed-in-top-5-ever-1.1059548 (visited on 05/24/2022).
[51] Cisco, *Cisco Annual Internet Report*, Accessed: 2022-05-24, URL: https://www.cisco.com/c/en/us/solutions/executive-perspectives/annual-internet-report/air-highlights.html.

[52] ClamAV, *ClamAV 0.99b Meets YARA!* ClamAV blog, Accessed: 2022-02-03, June 3, 2015, URL: https://blog.clamav.net/2015/06/clamav-099b-meets-yara.html.

[53] CLANG, Accessed: 2021-10-26, URL: http://clang.org/.

[54] Fred Cohen, « Computer viruses: Theory and experiments », in: *Computers & Security* 6.1 (1987), pp. 22–35, ISSN: 0167-4048, DOI: https://doi.org/10.1016/0167-4048(87)90122-2, URL: https://www.sciencedirect.com/science/article/pii/0167404887901222.

[55] Julian Cohen, *Contemporary Automatic Program Analysis*, Slideshow presented at Blackhat USA 2014, 2014, URL: https://www.blackhat.com/docs/us-14/materials/us-14-Cohen-Contemporary-Automatic-Program-Analysis.pdf.

[56] Christian Collberg et al., *The Tigress diversifying C virtualizer*, 2015.

[57] Gerald Combs, *Wireshark*, 2022, URL: https://www.wireshark.org.

[58] Juan Comesaña and Peter Klein, « Skepticism », in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Winter 2019, Metaphysics Research Lab, Stanford University, 2019.

[59] EUROPEAN COMMISSION, *Shaping Europe's digital future*, Accessed: 2022-05-24, 2020, URL: https://eur-lex.europa.eu/legal-content/en/TXT/?uri=CELEX:52020DC0067.

[60] Federal Trade Commission, *Monitoring Software on Your PC: Spyware, Adware, and Other Software*, Mar. 2005, URL: https://www.ftc.gov/sites/default/files/documents/reports/spyware-workshop-monitoring-software-your-personal-computer-spyware-adware-and-other-software-report/050307spywarerpt.pdf.

[61] Keith D. Cooper et al., « ACME: Adaptive Compilation Made Efficient », in: *Proceedings of the 2005 ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems*, LCTES '05, Chicago, Illinois, USA: Association for Computing Machinery, 2005, 69–77, ISBN: 1595930183, DOI: 10.1145/1065910.1065921, URL: https://doi.org/10.1145/1065910.1065921.
[62] Target Corporation, *Target Confirms Unauthorized Access to Payment Card Data in U.S. Stores*, 2013, URL: https://corporate.target.com/press/releases/2013/12/target-confirms-unauthorized-access-to-payment-car (visited on 05/24/2022).

[63] Patrick COUSOT et al., «Varieties of Static Analyzers: A Comparison with ASTREE», in: *First Joint IEEE/IFIP Symposium on Theoretical Aspects of Software Engineering (TASE '07)*, 2007, pp. 3–20, DOI: 10.1109/TASE.2007.55.

[64] CULT OF THE DEAD COW, *RUNNING A MICROSOFT OPERATING SYSTEM ON A NETWORK? OUR CONDOLENCES*, 1998, URL: https://media.defcon.org/DEF%20CON%206/DEF%20CON%206%20articles/DEF%20CON%206%20-%20Cult%20of%20the%20Dead%20Cow%20-%20Back%20Rifice%20Announcement.txt.

[65] George E. Dahl et al., «Large-scale malware classification using random projections and neural networks», in: *2013 IEEE International Conference on Acoustics, Speech and Signal Processing* (2013), pp. 3422–3426.

[66] Ben Dai, Xiaotong Shen, and Junhui Wang, «Embedding learning», in: *Journal of the American Statistical Association* (2020), pp. 1–13.

[67] Khanh Huu The Dam and Tayssir Touili, «Precise extraction of malicious behaviors», in: *2018 IEEE 42nd Annual Computer Software and Applications Conference (COMP-SAC)*, vol. 1, IEEE, 2018, pp. 229–234.

[68] Anusha Damodaran et al., «A comparison of static, dynamic, and hybrid analysis for malware detection», in: *Journal of Computer Virology and Hacking Techniques* 13.1 (2017), pp. 1–12.

[69] Leonardo De La Rosa et al., «Efficient characterization and classification of malware using deep learning», in: *2018 Resilience Week (RWS)*, IEEE, 2018, pp. 77–83.

[70] Department of Defense (DoD), *Department of Defense Strategy for Operating in Cyberspace*, July 2011, URL: https://csrc.nist.gov/CSRC/media/Projects/ISPAB/documents/DOD-Strategy-for-Operating-in-Cyberspace.pdf (visited on 05/24/2022).

[71] Richard C Detmer, *Introduction to 80x86 Assembly Language and Computer Architecture*, Jones & Bartlett Learning, 2006.
[72] Marko Dimjašević et al., «Evaluation of android malware detection based on system calls», in: *Proceedings of the 2016 ACM on International Workshop on Security And Privacy Analytics*, 2016, pp. 1–8.

[73] Yuxin Ding et al., «A malware detection method based on family behavior graph», in: *Computers & Security* 73 (2018), pp. 73–86, ISSN: 0167-4048, DOI: https://doi.org/10.1016/j.cose.2017.10.007, URL: https://www.sciencedirect.com/science/article/pii/S0167404817302146.

[74] Brendan L Douglas, «The weisfeiler-lehman method and graph isomorphism testing», in: *arXiv preprint arXiv:1101.5211* (2011).

[75] William Duncan, *Making Ontological Sense of Hardware and Software*, Accessed: 2021-12-08, 2009, URL: https://cse.buffalo.edu/~rapaport/584/S10/duncan09-HWSWont.pdf.

[76] William Duncan, «Ontological distinctions between hardware and software», in: *Applied Ontology* 12 (Feb. 2017), pp. 1–28, DOI: 10.3233/AO-170175.

[77] Joseph C Dunn, «Well-separated clusters and optimal fuzzy partitions», in: *Journal of cybernetics* 4.1 (1974), pp. 95–104.

[78] Editors, «A Pathology of Computer Viruses», in: *Virus Bulletin* (Mar. 1992), book review, ISSN: 0956-9979, URL: https://www.virusbulletin.com/uploads/pdf/magazine/1992/199203.pdf.

[79] Editors, «Towards a Common Language», in: *Virus Bulletin* (Dec. 1989), editorial, ISSN: 0956-9979, URL: https://www.virusbulletin.com/uploads/pdf/magazine/1989/198912.pdf.

[80] Ammar Elhadi et al., «Enhancing the Detection of Metamorphic Malware using Call Graphs», in: *Computers & Security* 46 (Oct. 2014), pp. 62–78.

[81] C.C. Elisan, *Malware, Rootkits & Botnets A Beginner's Guide*, Beginner's Guide, McGraw-Hill Education, 2012, ISBN: 978-0-07-179205-9.

[82] Adam Entous, Ellen Nakashima, and Greg Miller, «Secret CIA assessment says Russia was trying to help Trump win White House», in: *The Washington Post* (Dec. 9, 2016), URL: https://www.washingtonpost.com/world/national-security/obama-orders-review-of-russian-hacking-during-presidential-campaign/2016/12/09/31d6b300-be2a-11e6-94ac-3d324840106c_story.html (visited on 05/24/2022).
[83] Ikpeme Erete and Alessandro Orso, «Optimizing Constraint Solving to Better Support Symbolic Execution», in: *2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops*, 2011, pp. 310–315, DOI: 10.1109/ICSTW.2011.98.

[84] Mojtaba Eskandari, Zeinab Khorshidpour, and Sattar Hashemi, «HDM-Analyser: a hybrid analysis approach based on data mining techniques for malware detection», in: *Journal of Computer Virology and Hacking Techniques* 9.2 (2013), pp. 77–93.

[85] Mojtaba Eskandari and Hooman Raesi, «Frequent sub-graph mining for intelligent malware detection», in: *Security and Communication Networks* 7.11 (2014), pp. 1872–1886.

[86] Yong Fang et al., «Semi-supervised malware clustering based on the weight of byte-code and API», in: *IEEE Access* 8 (2019), pp. 2313–2326.

[87] Jonathan Fildes, «Stuxnet virus targets and spread revealed», in: *BBC News* (Feb. 15, 2011), URL: https://www.bbc.com/news/technology-12465688 (visited on 05/24/2022).

[88] Jim Finkle, «U.S. firm blames Russian 'Sandworm' hackers for Ukraine outage», in: *Reuters* (Jan. 8, 2016), URL: https://www.reuters.com/article/us-ukraine-cybersecurity-sandworm-idUSKBN0UM00N20160108 (visited on 05/24/2022).

[89] Andrea Fioraldi, «Symbolic Execution and Debugging Synchronization», Bachelor's thesis, Sapienza University of Rome, Oct. 2018.

[90] Halvar Flake, «Structural Comparison of Executable Objects», in: *In Proceedings of the IEEE Conference on Detection of Intrusions and Malware & Vulnerability Assessment* (DIMVA, 2004), pp. 161–173.

[91] Luciano Floridi, «The Method of Levels of Abstraction», in: *Minds and Machines* 18.3 (Sept. 2008), 303–329, ISSN: 1572-8641, DOI: 10.1007/s11023-008-9113-7.

[92] Luciano Floridi, Nir Fresco, and Giuseppe Primiero, «On malfunctioning software», in: *Synthese* 192.4 (Apr. 2015), 1199–1220, ISSN: 1573-0964, DOI: 10.1007/s11229-014-0610-3.

[93] Pascal Fontaine, «Satisfiability Modulo Theories: state-of-the-art, contributions, project», Habilitation à diriger des recherches, Université de Lorraine, Oct. 2018, URL: https://tel.archives-ouvertes.fr/tel-01968404.
[94] Matt Fredrikson et al., «Synthesizing near-optimal malware specifications from suspicious behaviors», in: *2010 IEEE Symposium on Security and Privacy*, IEEE, 2010, pp. 45–60.

[95] Lucas Galante et al., «Machine learning for malware detection: Beyond accuracy rates», in: *Anais Estendidos do XIX Simpósio Brasileiro de Segurança da Informação e de Sistemas Computacionais*, SBC, 2019, pp. 47–56.

[96] Tianchong Gao et al., «Android Malware Detection via Graphlet Sampling», in: *IEEE Transactions on Mobile Computing* 18.12 (2019), pp. 2754–2767, DOI: 10.1109/TMC.2018.2880731.

[97] Daniel B Garrie, Alan F Blakley, and Matthew J Amstrong, «Legal Status of Spyware», in: *Fed. Comm. LJ* 59 (2006), p. 157.

[98] Ilir Gashi et al., «A study of the relationship between antivirus regressions and label changes», in: *2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE)*, 2013, pp. 441–450, DOI: 10.1109/ISSRE.2013.6698897.

[99] Xiuting Ge et al., «AMDroid: Android Malware Detection Using Function Call Graphs», in: *2019 IEEE 19th International Conference on Software Quality, Reliability and Security Companion (QRS-C)*, 2019, pp. 71–77, DOI: 10.1109/QRS-C.2019.00027.

[100] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins, «Learning to forget: Continual prediction with LSTM», in: *Neural computation* 12.10 (2000), pp. 2451–2471.

[101] Ibrahim Ghafir, Vaclav Prenosil, et al., «Advanced persistent threat attack detection: an overview», in: *Int J Adv Comput Netw Secur* 4.4 (2014), p. 5054.

[102] Babu Nath Giri, Nitin Jyoti, and M Avert, «The emergence of ransomware», in: *AVAR, Auckland* (2006).

[103] Michael Glanzberg, «Truth», in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Summer 2021, Metaphysics Research Lab, Stanford University, 2021.

[104] Patrice Godefroid, Nils Klarlund, and Koushik Sen, «DART: Directed Automated Random Testing», in: *SIGPLAN Not.* 40.6 (June 2005), 213–223, ISSN: 0362-1340, DOI: 10.1145/1064978.1065036, URL: https://doi.org/10.1145/1064978.1065036.

[105] Allen T Goldberg, *On the complexity of the satisfiability problem*, New York University, 1979.
[106] Dan Goodin, « Hackers expose 453,000 credentials allegedly taken from Yahoo service », in: Ars Technica (July 12, 2012), URL: https://arstechnica.com/information-technology/2012/07/yahoo-service-hacked/ (visited on 05/24/2022).

[107] Ronald L Graham and Pavol Hell, « On the history of the minimum spanning tree problem », in: Annals of the History of Computing 7.1 (1985), pp. 43–57.

[108] Kent Griffin et al., « Automatic Generation of String Signatures for Malware Detection », in: Recent Advances in Intrusion Detection, ed. by Engin Kirda, Somesh Jha, and Davide Balzarotti, Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 101–120, ISBN: 978-3-642-04342-0.

[109] Aditya Grover and Jure Leskovec, « node2vec: Scalable feature learning for networks », in: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, 2016, pp. 855–864.

[110] André Grégio et al., « Ontology for malware behavior: A core model proposal », in: 2014 IEEE 23rd International WETICE Conference, 2014, pp. 453–458, DOI: 10.1109/WETICE.2014.72.

[111] Deepak Gupta and Rinkle Rani, « Improving malware detection using big data and ensemble learning », in: Computers & Electrical Engineering 86 (2020), p. 106729.

[112] Sanchit Gupta, Harshit Sharma, and Sarvjeet Kaur, « Malware characterization using windows API call sequences », in: International Conference on Security, Privacy, and Applied Cryptography Engineering, Springer, 2016, pp. 271–280.

[113] Ibai Gurrutxaga et al., « Evaluation of Malware clustering based on its dynamic behaviour », in: Proceedings of the Seventh Australasian Data Mining Conference (AusDM 2008), ed. by John F. Roddick et al., vol. 87, CRPIT, Australian Computer Society, 2008, pp. 163–170, URL: http://crpit.scem.westernsydney.edu.au/abstracts/CRPITV87Gurrutxaga.html.

[114] Nicolás Gálvez Ramírez et al., « Towards automated strategies in satisfiability modulo theory », in: European Conference on Genetic Programming, Springer, 2016, pp. 230–245.

[115] Gavin Hackeling, Mastering Machine Learning with scikit-learn, Packt Publishing Ltd, 2017.

[116] Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis, « Cluster validity methods: part I », in: ACM Sigmod Record 31.2 (2002), pp. 40–45.
[117] Greg Hamerly and Charles Elkan, «Alternatives to the k-means algorithm that find better clusterings», in: *Proceedings of the eleventh international conference on Information and knowledge management*, 2002, pp. 600–607.

[118] Steven Strandlund Hansen et al., «An approach for detection and family classification of malware based on behavioral analysis», in: *2016 International conference on computing, networking and communications (ICNC)*, IEEE, 2016, pp. 1–5.

[119] Irfan Ul Haq and Juan Caballero, *A Survey of Binary Code Similarity*, 2019, arXiv: 1909.11424 [cs.CR].

[120] Ali Hasan and Richard Fumerton, «Foundationalist Theories of Epistemic Justification», in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Fall 2018, Metaphysics Research Lab, Stanford University, 2018.

[121] Mehadi Hassen and Philip K Chan, «Scalable function call graph-based malware classification», in: *Proceedings of the Seventh ACM on Conference on Data and Application Security and Privacy*, 2017, pp. 239–248.

[122] The White House, *International Strategy for Cyberspace*, May 2011, URL: https://obamawhitehouse.archives.gov/sites/default/files/rss viewer/international strategy for cyberspace.pdf (visited on 05/24/2022).

[123] Dr Jan Hruska, *The Story of Sophos: From 0 to 1200+ employees*, YouTube, URL: https://www.youtube.com/watch?v=n5SeafKv_M0&t=208s (visited on 05/24/2022).

[124] Xin Hu and Kang G Shin, «DUET: integration of dynamic and static analyses for malware clustering with cluster ensembles», in: *Proceedings of the 29th annual computer security applications conference*, 2013, pp. 79–88.

[125] Xin Hu et al., «{MutantX-S}: Scalable Malware Clustering Based on Static Features», in: *2013 USENIX Annual Technical Conference (USENIX ATC 13)*, 2013, pp. 187–198.

[126] Na Huang et al., «Deep Android Malware Classification with API-Based Feature Graph», in: *2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE)*, 2019, pp. 296–303, DOI: 10.1109/TrustCom/BigDataSE.2019.00047.
[127] Intelligence Community Assessment (ICA), Background to "Assessing Russian Activities and Intentions in Recent US Elections": The Analytic Process and Cyber Incident Attribution, Jan. 6, 2017, URL: https://www.dni.gov/files/documents/ICA_2017_01.pdf (visited on 05/24/2022).

[128] Jonathan Jenkins Ichikawa and Matthias Steup, «The Analysis of Knowledge», in: The Stanford Encyclopedia of Philosophy, ed. by Edward N. Zalta, Summer 2018, Meta-physics Research Lab, Stanford University, 2018.

[129] Nwokedi C. Idika and Aditya P. Mathur, «A Survey of Malware Detection Techniques», in: 2007.

[130] Michael Isikof, «Chinese hacked Obama, McCain campaigns, took internal documents, officials say», in: NBC News (June 7, 2013), URL: https://www.nbcnews.com/id/wbna52133016 (visited on 05/24/2022).

[131] Rafiqul Islam et al., «Classification of malware based on integrated static and dynamic features», in: Journal of Network and Computer Applications 36.2 (2013), pp. 646–656, ISSN: 1084-8045, DOI: https://doi.org/10.1016/j.jnca.2012.10.004, URL: https://www.sciencedirect.com/science/article/pii/S1084804512002214.

[132] International Telecommunication Union (ITU), 2.9 billion people still offline, Accessed: 2022-05-24, 2021, URL: https://www.itu.int/en/mediacentre/Pages/PR-2021-11-29-FactsFigures.aspx.

[133] Chuntao Jiang, Frans Coenen, and Michele Zito, A Survey of Frequent Subgraph Mining Algorithms, 2004.

[134] Haodi Jiang, Turki Turki, and Jason TL Wang, «Dlgraph: Malware detection using deep learning and graph embedding», in: 2018 17th IEEE international conference on machine learning and applications (ICMLA), IEEE, 2018, pp. 1029–1033.

[135] Andreas Johnsen et al., «Experience Report: Evaluating Fault Detection Effectiveness and Resource Efficiency of the Architecture Quality Assurance Framework and Tool», in: 2017 IEEE 28th International Symposium on Software Reliability Engineering (IS-SRE), 2017, pp. 271–281, DOI: 10.1109/ISSRE.2017.31.

[136] Maximilian Junker et al., «SMT-Based False Positive Elimination in Static Program Analysis», in: Formal Methods and Software Engineering, ed. by Toshiaki Aoki and Kenji Taguchi, Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 316–331, ISBN: 978-3-642-34281-3.
[137] Martin Jurecek, Olha Jurecková, and Róbert Lórencz, «Improving Classification of Malware Families using Learning a Distance Metric.», in: *ICISSP*, 2021, pp. 643–652.

[138] Martin Jureček and Róbert Lórencz, «Malware detection using a heterogeneous distance function», in: *Computing and Informatics* 37.3 (2018), pp. 759–780.

[139] Aparna Sunil Kale, Fabio Di Troia, and Mark Stamp, «Malware Classification with Word Embedding Features», in: *arXiv preprint arXiv:2103.02711* (2021).

[140] Alex Kantchelian et al., «Better Malware Ground Truth: Techniques for Weighting Anti-Virus Vendor Labels», in: *Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security*, AISec '15, Denver, Colorado, USA: Association for Computing Machinery, 2015, 45–56, ISBN: 9781450338264, DOI: 10.1145/2808769.2808780, URL: https://doi.org/10.1145/2808769.2808780.

[141] Youngjoon Ki, Eunjin Kim, and Huy Kang Kim, «A novel approach to detect malware based on API call sequence analysis», in: *International Journal of Distributed Sensor Networks* 11.6 (2015), p. 659101.

[142] Eric Kim, *Everything you wanted to know about the kernel trick*, 2017, URL: https://www.eric-kim.net/eric-kim-net/posts/1/kernel_trick_blog_ekim_12_20_2017.pdf (visited on 10/17/2022).

[143] Sungkwan Kim et al., «A Brief Survey on Rootkit Techniques in Malicious Codes.», in: *J. Internet Serv. Inf. Secur.* 2.3/4 (2012), pp. 134–147.

[144] Joris Kinable and Orestis Kostakis, «Malware classification based on call graph clustering», in: *Journal in Computer Virology* 7.4 (Nov. 1, 2011), pp. 233–245, DOI: 10.1007/s11416-011-0151-y, URL: https://doi.org/10.1007/s11416-011-0151-y.

[145] James C. King, «A New Approach to Program Testing», in: *SIGPLAN Not.* 10.6 (Apr. 1975), 228–233, ISSN: 0362-1340, DOI: 10.1145/390016.808444, URL: https://doi.org/10.1145/390016.808444.

[146] Kevin Kirchner and Stefan Rosenthaler, «bin2llvm: analysis of binary programs using LLVM intermediate representation», in: *Proceedings of the 12th International Conference on Availability, Reliability and Security*, 2017, pp. 1–7.

[147] Thomas Kluyver et al., *Jupyter Notebooks-a publishing format for reproducible computational workflows*. Vol. 2016, 2016.
[148] Clemens Kolbitsch et al., «Effective and Efficient Malware Detection at the End Host», in: *Proceedings of the 18th Conference on USENIX Security Symposium*, SSYM’09, Montreal, Canada: USENIX Association, 2009, 351–366.

[149] Deguang Kong and Guanhua Yan, «Discriminant malware distance learning on structural information for automated malware classification», in: *Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining*, 2013, pp. 1357–1365.

[150] Keith Allen Korcz, «The Epistemic Basing Relation», in: *The Stanford Encyclopedia of Philosophy*, ed. by Edward N. Zalta, Spring 2021, Metaphysics Research Lab, Stanford University, 2021.

[151] Orestis Kostakis, «Classy: Fast Clustering Streams of Call-Graphs», in: *Data Mining and Knowledge Discovery* 28.5–6 (Sept. 2014), 1554–1585, ISSN: 1384-5810, DOI: 10.1007/s10618-014-0367-9, URL: https://doi.org/10.1007/s10618-014-0367-9.

[152] Christopher Kruegel et al., «Polymorphic Worm Detection Using Structural Information of Executables», in: *Recent Advances in Intrusion Detection*, ed. by Alfonso Valdes and Diego Zamboni, Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 207–226, ISBN: 978-3-540-31779-1.

[153] Tarald O Kvalseth, «Entropy and correlation: Some comments», in: *IEEE Transactions on Systems, Man, and Cybernetics* 17.3 (1987), pp. 517–519.

[154] Kaspersky Lab, *Machine Learning and Human Expertise*, tech. rep., White Paper, 2020, URL: https://media.kaspersky.com/en/business-security/machine-learning-human-expertise.pdf.

[155] Raphael Labaca-Castro et al., «Universal adversarial perturbations for malware», in: *arXiv preprint arXiv:2102.06747* (2021).

[156] Bell Labs, *Unix Programmer's Manual (First Edition)*, URL: https://www.bell-labs.com/usr/dmr/www/man21.pdf (visited on 05/30/2022).

[157] Arun Lakhotia, Mila Dalla Preda, and Roberto Giacobazzi, «Fast Location of Similar Code Fragments Using Semantic 'Juice'», in: *Proceedings of the 2nd ACM SIGPLAN Program Protection and Reverse Engineering Workshop*, PPREW '13, Rome, Italy: Association for Computing Machinery, 2013, ISBN: 9781450318570, DOI: 10.1145/2430553.2430558, URL: https://doi.org/10.1145/2430553.2430558.
[158] Erik Lambrechts et al., « Round-robin tournaments generated by the circle method have maximum carry-over », in: *Mathematical Programming* 172.1 (2018), pp. 277–302.

[159] Chris Lattner, « LLVM and Clang: Next generation compiler technology », in: *The BSD conference*, vol. 5, 2008.

[160] Quoc Le and Tomas Mikolov, « Distributed representations of sentences and documents », in: *International conference on machine learning*, PMLR, 2014, pp. 1188–1196.

[161] Jusuk Lee, Kyoochang Jeong, and Heejo Lee, « Detecting Metamorphic Malwares Using Code Graphs », in: *Proceedings of the 2010 ACM Symposium on Applied Computing*, SAC ’10, Sierre, Switzerland: Association for Computing Machinery, 2010, 1970–1977, ISBN: 9781605586397, DOI: 10.1145/1774088.1774505, URL: https://doi.org/10.1145/1774088.1774505.

[162] Wan-Jui Lee, Sergey Verzakov, and Robert PW Duin, « Kernel combination versus classifier combination », in: *International Workshop on Multiple Classifier Systems*, Springer, 2007, pp. 22–31.

[163] John Healy Leland McInnes and Steve Astels, *The hdbscan Clustering Library*, Accessed: 2022-03-02, 2016, URL: https://hdbscan.readthedocs.io/en/latest/index.html.

[164] Robert Lemos, « Home Depot estimates data on 56 million cards stolen by cybercriminals », in: *Ars Technica* (Sept. 19, 2014), URL: https://arstechnica.com/information-technology/2014/09/home-depot-estimates-data-on-56-million-cards-stolen-by-cybercriminals/ (visited on 05/24/2022).

[165] Peng Li et al., « On Challenges in Evaluating Malware Clustering », in: *RAID 2010, Ottawa, Ontario, Canada, September 15-17, 2010. Proceedings*, ed. by Somesh Jha, Robin Sommer, and Christian Kreibich, vol. 6307, Lecture Notes in Computer Science, Springer, 2010, pp. 238–255, DOI: 10.1007/978-3-642-15512-3\_13, URL: https://doi.org/10.1007/978-3-642-15512-3\_13.

[166] Yi Li et al., « Symbolic Optimization with SMT Solvers », in: *SIGPLAN Not.* 49.1 (Jan. 2014), 607–618, ISSN: 0362-1340, DOI: 10.1145/2578855.2535857, URL: https://doi.org/10.1145/2578855.2535857.
[167] You Li et al., « Steering symbolic execution to less traveled paths », in: *ACM SigPlan Notices* 48.10 (2013), pp. 19–32.

[168] Andrew Liptak, « The WannaCry ransomware attack has spread to 150 countries », in: *The Verge* (May 14, 2017), URL: https://www.theverge.com/2017/5/14/15637888/authorities-wannacry-ransomware-attack-spread-150-countries (visited on 05/24/2022).

[169] Bo Long, Philip S Yu, and Zhongfei Zhang, « A general model for multiple view unsupervised learning », in: *Proceedings of the 2008 SIAM international conference on data mining*, SIAM, 2008, pp. 822–833.

[170] Renjie Lu, « Malware detection with lstm using opcode language », in: *arXiv preprint arXiv:1906.04593* (2019).

[171] Chi-Keung Luk et al., « Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation », in: *Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation*, PLDI ’05, Chicago, IL, USA: Association for Computing Machinery, 2005, 190–200, ISBN: 1595930566, DOI: 10.1145/1065010.1065034, URL: https://doi.org/10.1145/1065010.1065034.

[172] Weilin Luo and Ou Wei, « WAP: SAT-Based Computation of Minimal Cut Sets », in: *2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE)*, 2017, pp. 146–151, DOI: 10.1109/ISSRE.2017.13.

[173] Guixiang Ma et al., « Deep graph similarity learning: A survey », in: *Data Mining and Knowledge Discovery* 35.3 (2021), pp. 688–725.

[174] Hugo Daniel Macedo and Tayssir Touili, « Mining malware specifications through static reachability analysis », in: *European Symposium on Research in Computer Security*, Springer, 2013, pp. 517–535.

[175] Rupak Majumdar and Koushik Sen, « Hybrid concolic testing », in: *29th International Conference on Software Engineering (ICSE’07)*, IEEE, 2007, pp. 416–426.

[176] *Malware Attribute Enumeration and Characterization (MAEC)*, Accessed: 2021-12-01, URL: https://maecproject.github.io.

[177] William B March, Parikshit Ram, and Alexander G Gray, « Fast euclidean minimum spanning tree: algorithm, analysis, and applications », in: *Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining*, 2010, pp. 603–612.
[178] John Markoff, « Before the Gunfire, Cyberattacks », in: *The New York Times* (Aug. 18, 2008), URL: https://www.nytimes.com/2008/08/13/technology/13cyber.html (visited on 05/24/2022).

[179] Mohammad M Masud, Latifur Khan, and Bhavani Thuraisingham, « A scalable multi-level feature extraction technique to detect malicious executables », in: *Information Systems Frontiers* 10.1 (2008), pp. 33–45.

[180] Mohammad M Masud, Latifur Khan, and Bhavani Thuraisingham, « Feature based techniques for auto-detection of novel email worms », in: *Pacific-Asia Conference on Knowledge Discovery and Data Mining*, Springer, 2007, pp. 205–216.

[181] Deborah L McGuinness, Frank Van Harmelen, et al., « OWL web ontology language overview », in: *W3C recommendation 10.10* (2004), p. 2004.

[182] Leland McInnes and John Healy, « Accelerated Hierarchical Density Based Clustering », in: *Data Mining Workshops (ICDMW), 2017 IEEE International Conference on*, IEEE, 2017, pp. 33–42.

[183] Leland McInnes, John Healy, and Steve Astels, « hdbscan: Hierarchical density based clustering. », in: *J. Open Source Softw.* 2.11 (2017), p. 205.

[184] Leland McInnes, John Healy, and Steve Astels, « hdbscan: Hierarchical density based clustering », in: *The Journal of Open Source Software* 2.11 (Mar. 2017), DOI: 10.21105/joss.00205, URL: https://doi.org/10.21105/2Fjoss.00205.

[185] Dirk Merkel, « Docker: Lightweight Linux Containers for Consistent Development and Deployment », in: *Linux J.* 2014.239 (Mar. 2014), ISSN: 1075-3583.

[186] Jason Merrill, « Generic and gimple: A new tree representation for entire functions », in: *Proceedings of the 2003 GCC Developers' Summit*, Citeseer, 2003, pp. 171–179.

[187] Microsoft, *Microsoft Malware Classification Challenge (BIG 2015)*, Website, URL: https://www.kaggle.com/c/malware-classification (visited on 05/24/2022).

[188] Microsoft, *Process Monitor (ProcMon)*, 2022, URL: https://docs.microsoft.com/en-us/sysinternals/downloads/procmon.

[189] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig, « Linguistic regularities in continuous space word representations », in: *Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies*, 2013, pp. 746–751.
[190] Tomas Mikolov et al., «Distributed Representations of Words and Phrases and their Compositionality», in: *Advances in Neural Information Processing Systems*, ed. by C. J. C. Burges et al., vol. 26, Curran Associates, Inc., 2013, URL: https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf.

[191] Tomas Mikolov et al., «Efficient estimation of word representations in vector space», in: *arXiv preprint arXiv:1301.3781* (2013).

[192] Jiang Ming et al., «{TaintPipe}: Pipelined Symbolic Taint Analysis», in: *24th USENIX Security Symposium (USENIX Security 15)*, 2015, pp. 65–80.

[193] Volodymyr Mnih et al., «Playing atari with deep reinforcement learning», in: *arXiv preprint arXiv:1312.5602* (2013).

[194] Aziz Mohaisen, Omar Alrawi, and Manar Mohaisen, «Amal: High-fidelity, behavior-based automated malware analysis and classification», in: *computers & security* 52 (2015), pp. 251–266.

[195] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, *Foundations of machine learning*, MIT press, 2018.

[196] James H. Moor, «Three Myths of Computer Science», in: *The British Journal for the Philosophy of Science* 29.3 (1978), pp. 213–222, ISSN: 00070882, 14643537, URL: http://www.jstor.org/stable/687009.

[197] A. Moser, C. Kruegel, and E. Kirda, «Limits of Static Analysis for Malware Detection», in: *Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007)*, Dec. 2007, pp. 421–430, DOI: 10.1109/ACSAC.2007.21.

[198] Leonardo de Moura and Grant Olney Passmore, «The Strategy Challenge in SMT Solving», in: *Automated Reasoning and Mathematics: Essays in Memory of William W. McCune*, ed. by Maria Paola Bonacina and Mark E. Stickel, Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 15–44, ISBN: 978-3-642-36675-8, DOI: 10.1007/978-3-642-36675-8_2, URL: https://doi.org/10.1007/978-3-642-36675-8_2.

[199] David A Mundie and David M Mcintire, «An ontology for malware analysis», in: *2013 International Conference on Availability, Reliability and Security*, IEEE, 2013, pp. 556–558.
[200] Annamalai Narayanan et al., « graph2vec: Learning distributed representations of graphs », in: arXiv preprint arXiv:1707.05005 (2017).

[201] NATO, *Wales Summit Declaration*, Sept. 5, 2014, URL: https://www.nato.int/cps/en/natohq/official texts_112964.htm (visited on 05/24/2022).

[202] NATO, *Warsaw Summit Key Decisions*, Feb. 6, 2017, URL: https://www.nato.int/nato_static_f12014/assets/pdf/pdf_2017_02/20170206_1702-factsheet-warsaw-summit-key-en.pdf (visited on 05/24/2022).

[203] Shah Nazir, Sara Shahzad, and Neelam Mukhtar, « Software Birthmark Design and Estimation: A Systematic Literature Review », in: *Arabian Journal for Science and Engineering* 44.4 (Apr. 1, 2019), pp. 3905–3927, DOI: 10.1007/s13369-019-03718-9, URL: https://doi.org/10.1007/s13369-019-03718-9.

[204] Nicholas Nethercote and Julian Seward, « Valgrind: a framework for heavyweight dynamic binary instrumentation », in: *ACM Sigplan notices* 42.6 (2007), pp. 89–100.

[205] Stavros D. Nikolopoulos and Iosif Polenakis, « A graph-based model for malware detection and classification using system-call groups », in: *Journal of Computer Virology and Hacking Techniques* 13.1 (Feb. 2017), 29–46, ISSN: 2263-8733, DOI: 10.1007/s11416-016-0267-1.

[206] Lamine Noureddine et al., « SE-PAC: A Self-Evolving PAcker Classifier against rapid packers evolution », in: *Proceedings of the Eleventh ACM Conference on Data and Application Security and Privacy*, 2021, pp. 281–292.

[207] Leo Obrst, Penny Chase, and Richard Markeloff, « Developing an Ontology of the Cyber Security Domain. », in: STIDS, Citeseer, 2012, pp. 49–56.

[208] Angelo Oliveira and R Sassi, « Behavioral malware detection using deep graph convolutional neural networks », in: *TechRxiv* (2019).

[209] Paul Oppenheim and Hilary Putnam, « Unity of Science as a Working Hypothesis », in: *Minnesota Studies in the Philosophy of Science* 2 (1958), pp. 3–36.

[210] UN Secretary-General's High level Panel, *The Age of Digital Interdependence*, Accessed: 2022-05-24, 2019, URL: https://www.un.org/en/pdfs/DigitalCooperation-report-for%20web.pdf.

[211] Younghee Park, D.s Reeves, and Mark Stamp, « Deriving common malware behavior through graph clustering », in: *Computers & Security* 39 (Nov. 2013), pp. 419–430, DOI: 10.1016/j.cose.2013.09.006.
[212] Younghee Park et al., « Fast Malware Classification by Automated Behavioral Graph Matching », in: *Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research*, CSIIRW '10, Oak Ridge, Tennessee, USA: Association for Computing Machinery, 2010, ISBN: 9781450300179, DOI: 10.1145/1852666.1852716, URL: https://doi.org/10.1145/1852666.1852716.

[213] Sunhera Paul and Mark Stamp, « Word embedding techniques for malware evolution detection », in: *Malware Analysis Using Artificial Intelligence and Deep Learning*, Springer, 2021, pp. 321–343.

[214] Fabian Pedregosa et al., « Scikit-learn: Machine learning in Python », in: *the Journal of machine Learning research* 12 (2011), pp. 2825–2830.

[215] Jeffrey Pennington, Richard Socher, and Christopher D Manning, « Glove: Global vectors for word representation », in: *Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)*, 2014, pp. 1532–1543.

[216] Harold Emanuel Petersen and Rein Turn, « System implications of information privacy », in: *Proceedings of the April 18-20, 1967, spring joint computer conference*, 1967, pp. 291–300.

[217] Rob Pike, *The go programming language*, 2009, URL: https://9p.io/sources/contrib/ericvh/go-plan9/doc/go_talk-20091030.pdf (visited on 10/17/2022).

[218] Daniel Plohmann et al., « Malpedia: a collaborative effort to inventorize the malware landscape », in: *Proceedings of the Botconf*, 2017.

[219] Pavel Polityuk and Steve Holland, « Cyberattack hits Ukraine as U.S. warns Russia could be prepping for war », in: *Reuters* (Jan. 14, 2022), URL: https://www.reuters.com/world/europe/expect-worst-ukraine-hit-by-cyberattack-russia-moves-more-troops-2022-01-14/ (visited on 05/24/2022).

[220] Mila Dalla Preda et al., « A Semantics-based Approach to Malware Detection », in: *SIGPLAN Not.* 42.1 (Jan. 2007), pp. 377–388, ISSN: 0362-1340, DOI: 10.1145/1190215.1190270, URL: http://doi.acm.org/10.1145/1190215.1190270.

[221] Robert Clay Prim, « Shortest connection networks and some generalizations », in: *The Bell System Technical Journal* 36.6 (1957), pp. 1389–1401.

[222] Giuseppe Primiero, « Information in the philosophy of computer science », in: *The Routledge handbook of philosophy of information* (2016), pp. 90–106.