<figure><img src="image_11.png" /><figcaption>Figure 5.2: 3 types of open triads.</figcaption></figure>

ists. Therefore, we dedicate this section to the review of the literature on signed *vertex* embedding.

Historically, the first type of such methods relies on random walks, e.g. **Signed Network Embedding (SNE)** [158] or **Signed Directed Embeddings (SIDE)** [159]. The general idea is to sample the graph using random walks, and feed them to a standard neural network, which learns a representation that preserves simultaneously graph structure and edge signs. Put differently, the representations of 2 well-connected vertices tend to be close in the embedding space, whereas those of 2 vertices connected by a negative edge tend to be distant.

**Signed Network Embedding (SiNE)** is a deep learning framework for vertex embedding in signed graphs [195]. Following [196], it is based on the assumption that the representation of a vertex should be similar to its positive neighbors, and dissimilar to its negative ones. To model signed networks based on this principle, SiNE proceeds at a local level by focusing on a very specific type of subgraph. It extracts the set of all open triads (i.e. 3 vertices connected by 2 edges) present in the graph. Figure 5.2 shows the 3 types of possible open triads. SiNE focuses on those containing one positive and one negative edges (Figure 5.2.b). This set of triads represents the graph, and is fed to a deep learning framework composed of 2 neural networks sharing the same weights. In accordance with the principle mentioned earlier, this model is trained in order to minimize the similarity between the representation of the vertex located at the center of the triad and its negative neighbors, while maximizing its similarity with its positive ones. One neural network within the framework is dedicated to the positive neighbors, while the other network handles the negative ones.

A limitation of this approach is that it ignores vertices whose neighbors are all positive or all negative. To handle this case, SiNE introduces a dummy vertex that is connected through a negative edge to each vertex with an all-positive neighborhood. This allows creating as many dummy triads containing one positive and one negative edges, which can be processed by the framework. The principle is the same for all-negative neighborhoods, except with positive dummy edges.

Most GCNs are designed to produce *vertex*-level representations. The general principle is as follows. Each vertex is initially represented by a vector, that can be generated randomly or
based on some vertex features. In a convolution layer, the representation of a given vertex is combined with those of its neighbors. The result is then fed to a generally non-linear function (e.g. multilayer perceptron) in order to get the updated vertex representation. The information is propagated through multiple layers to incorporate information from multi-hop neighbors. The maximal number of hops corresponds to the number of convolution layers in the network. GCNs achieve state-of-the-art performances on many tasks [160] such as vertex classification, edge prediction, and community detection.

Standard GCNs are only able to handle unsigned networks, though. Leveraging the information conveyed by edge signs mainly requires adapting the message passing rules used when computing the graph-based convolution. Signed Graph Convolutional Networks (SGCN) [160] allow doing so. This method relies on a dual hidden representation of a vertex, corresponding to its positive vs. negative reach sets, respectively noted $h^+$ and $h^-$. It uses balance theory to aggregate and propagate vertex representations across layers. Formally, the hidden representations $h$ at layer $t$ are updated as follows:

$$h_t^+(u) = \sigma \left( W_t^+ \left[ \sum_{v \in N^+} \frac{h_{t-1}^+(v)}{k^+(u)}, \sum_{v \in N^-} \frac{h_{t-1}^-(v)}{k^-(u)}, h_{t-1}^+(u) \right] \right), \quad (5.1)$$

$$h_t^-(u) = \sigma \left( W_t^- \left[ \sum_{v \in N^+} \frac{h_{t-1}^-(v)}{k^+(u)}, \sum_{v \in N^-} \frac{h_{t-1}^-(v)}{k^-(u)}, h_{t-1}^-(u) \right] \right), \quad (5.2)$$

where $\sigma$ is a non-linear activation function, [] denotes the concatenation, and the $W$ matrices are learnable weights. Ultimately, the dual hidden representations are concatenated to obtain a single vertex representation.

In addition, methods such as Signed Graph Attention Networks (SiGAT) [197] or Signed Network Embedding via Graph Attention (SNEA) [198] introduce attention to GNNs, in order to give more importance to relevant neighbors during the message passing step.

## 5.3 Datasets

A bottleneck in the development of whole-graph oriented methods for signed graphs is the lack of data. Indeed, these methods require collections of signed graphs, with each graph representing an instance in the treated task. However, the literature is mostly built around single, very large graphs, appropriate to test vertex representation learning techniques. The abuse detection task that we treat throughout this manuscript does not provide enough diversity to allow a correct assessment of our proposed methods. Therefore, we propose to extend it with 2 additional collections of signed networks annotated for diverse classification tasks. They come from various sources and are different in their structures and sizes. The benchmark that we constitute is thus composed of 3 collections of graphs. The first one is the SpaceOrigin collection extracted in Chapter 4.1 from
the raw dataset described in Section 2.3. It contains exactly the same graphs as before, so we do
not further describe it in this section. The second one (Section 5.3.1) is an existing collection of
artificially generated graphs, and the third one (Section 5.3.2) is an existing collection of real-world
vote networks. We leverage the metadata associated to these collections, in order to repurpose
them and define proper classification tasks. Global statistics describing the 3 constituted datasets
are available in Table 5.1.

5.3.1 Correlation Clustering Instances

This dataset is proposed in [199], originally as a means to study the space of optimal solutions
of the *Correlation Clustering* (CC) problem [166], described in Section 5.1. This graph partition-
ing problem consists in detecting internally collaborative but mutually hostile communities, which
amounts to identifying the vertex partition that minimizes the total number of frustrated edges.
The authors want to study the multiplicity and the diversity of the optimal solutions to CC. For this
purpose, they define a random model and generate a collection of artificial graphs with planted par-
tition, applying various levels of noise to control the problem difficulty. They use an exact method to
identify all possible optimal solutions for each graph in this collection, and study how certain graph
characteristics relate to the number of solutions.

Due to the NP-hard nature of CC, they focus on relatively small graphs, with a maximal order (number of vertices) of $n = 50$. They produce a total of 24,660 unweighted signed graphs, including 22,560 completely connected graphs (i.e. every pair of vertices is connected), while the remaining 2,100 graphs are not completely connected, with a density ranging from 0.25 to 0.75.

In order to use this dataset in the present work, we define a classification problem by associ-
ating each graph of the collection to a label. This problem, noted *Correlation Clustering Solutions*
(CCS), consists in predicting whether there is a single vs. several optimal CC solutions for the
graph of interest. It is thus a binary classification task.

5.3.2 European Parliament Roll-Calls

The third and last dataset is based on a collection of signed graphs extracted in [200] from a
description of the voting activity at the European Parliament (EP). The raw data corresponds to roll-
call votes cast individually by Members of the EP (MEPs) during plenary sessions, in the course
of the 7th term (2009–2014). Such votes can take one out of 3 values: FOR (MEP supporting
the proposition), AGAINST (MEP opposing the proposition) or ABSTENTION (MEP not taking a
stand despite being present). MEPs can also be absent, and consequently not take part at all to
a roll-call. Some MEPs resigned from their position in the course of their term of office, and some
MEPs were appointed after the beginning of the term. In such cases, we consider them absent for
all the roll-call that they missed.
Each network extracted by the authors corresponds to a specific roll-call, using vertices to model MEPs and edges to represent agreement between them: a positive sign models an identical vote, and a negative one a disagreement. Note that ABSTENTION is an expressed opinion, so a FOR vs ABSTENTION or AGAINST vs ABSTENTION is considered as a disagreement. Contrariwise, an absent MEP is not considered as expressing an opinion and no edge is associated with him. The goal of the authors is to study the polarization of the EP, and more specifically its voting patterns, and how these are affected by various criteria such as the topic of the voted proposition. For this purpose, they first identify factions of similarly voting MEPs in each roll-call network. Next, they compare the resulting vertex partitions to identify types of situations that result in a comparable voting pattern.

The raw data contains 6,595 roll-calls, from which the authors extract much more networks by leveraging the metadata associated to the voting activity. Since each MEP belongs to a member state, and to a European political group, they extract overall networks containing all MEPs, but also state- and group-specific networks, containing only the MEPs from a given member state or political group, respectively. In total, they produce 244,015 networks. Some of them are too small or too sparse (almost empty) to be interesting in a classification context, though. We constitute our own dataset by first filtering out these unusable instances. We consider state- and group-specific networks with 20 or more MEPs. Then, we randomly sample 6,000 networks among all the networks matching the previous criteria.

Like for the previous dataset, the initial collection of networks considered here is not originally used to perform any prediction task. We leverage the clusters of networks exhibiting similar voting patterns identified by the authors, and define a classification task consisting in predicting the num- ber of factions identified in a network. There are 3 classes (i.e. 1 faction, 2 factions, 3 factions) in this task. We call the resulting dataset *European Parliament Factions* (EPF). We present the statistics of the 3 datasets in the next section.

### 5.3.3 Brief Comparison

Table 5.1 provides a few descriptive statistics for all 3 datasets, in order to help to interpret the classification results in Section 5.5.1. CCS is by far the largest dataset in number of graphs (# Graphs), however these are smaller. Moreover, the size of graphs in terms of vertices is not as uniform in both SO and EPF, covering 2 orders of magnitude. The number of edges is quite variable in all 3 dataset. In EPF, a few graphs have no positive or no negative edges at all. Overall, the graphs tend to be rather dense compared to unsigned real-world networks.

The Frustration, which we compute for both types of considered balance (SB vs. GB) is expressed as a proportion over the total number of edges in the graph, in order to have comparable values. The top right columns contain the number of classes in each dataset, and the Gini index showing how imbalanced they are.
<table><caption>Table 5.1: Statistics describing the 3 datasets. Notations ± and [] denote the standard deviation, and minimum & maximum.</caption><thead><tr><th>Dataset</th><th># Graphs</th><th>Avg. # Vertices</th><th># Classes</th><th>Gini Index</th></tr></thead><tbody><tr><td>SO</td><td>2,545</td><td>47.74 ±20.34 [2;214]</td><td>2</td><td>0.74</td></tr><tr><td>CCS</td><td>24,660</td><td>27.31 ±7.44 [16;50]</td><td>2</td><td>0.66</td></tr><tr><td>EPF</td><td>6,000</td><td>67.34 ±59.21 [20;274]</td><td>3</td><td>0.44</td></tr></tbody></table>

<table><thead><tr><th>Dataset</th><th>Avg. # Negative Edges</th><th>Avg. # Positive Edges</th><th>% Positive Edges</th></tr></thead><tbody><tr><td>SO</td><td>166.1 ±22.96 [1;1,692]</td><td>245.9 ±30.41 [1;2,323]</td><td>59.61 ±24.73 [0.0;100.0]</td></tr><tr><td>CCS</td><td>220.6 ±130.45 [25;833]</td><td>131.0 ±80.11 [24;392]</td><td>37.54 ±11.99 [20.1;79.0]</td></tr><tr><td>EPF</td><td>333.9 ±877.20 [0;15,933]</td><td>2,552.2 ±5,761.46 [0;33,153]</td><td>78.30 ±22.22 [0.0;100.0]</td></tr></tbody></table>

<table><thead><tr><th>Dataset</th><th>Avg. SB Frustration</th><th>Avg. GB Frustration</th><th>Avg. density</th></tr></thead><tbody><tr><td>SO</td><td>0.30 ±0.04 [0.01;0.48]</td><td>0.25 ±0.04 [0.01;0.46]</td><td>0.48 ±0.16 [0.10;1.00]</td></tr><tr><td>CCS</td><td>0.37 ±0.05 [0.03;0.51]</td><td>0.33 ±0.04 [0.01;0.49]</td><td>0.95 ±0.17 [0.19;1.00]</td></tr><tr><td>EPF</td><td>0.28 ±0.04 [0.01;0.46]</td><td>0.22 ±0.03 [0.00;0.45]</td><td>0.70 ±0.19 [0.07;1.00]</td></tr></tbody></table>

## 5.4 Proposed Methods

In this section, we describe the 3 families of methods which we propose to handle signed whole-graph representation learning. The first can be considered as a baseline, and relies on the aggregation of signed vertex embeddings (Section 5.4.1). The second is an adaptation of an unsigned whole-graph embedding method to signed graphs (Section 5.4.2). The third is based on a Graph Convolutional Network able to learn signed vertex representations, which we adapt to handle whole signed graphs (Section 5.4.3).

### 5.4.1 Signed Network Embedding

As described in Section 5.2.3, SiNE learns a compact representation for each vertex in the input graph, based only on its direct neighborhood. Since our tasks consist in classifying graphs and not vertices, our proposition is to aggregate the representations of all the vertices in the graph in order to get a graph-level representation. This approach is quite standard in the literature [201], [202]. It relies on the assumption that individual vertex representations each contain some part of information that describes the whole graph. Thus, aggregating all the vertex representations allows obtaining a graph-level representation. For the sake of completeness, we consider 2 approaches for the aggregation process: averaging the vertex representations, and summing them.

SiNE exhibits some limitations, though. First, it does not really handle vertices possessing only negative edges. The authors argue that this case is very uncommon in real-world data, but
such vertices exist in our datasets. Second, SiNE is based only on local information, while we can suppose that global information might be important to represent the graph in its entirety. Therefore, this method can be considered as a baseline to assess whether it is beneficial to use whole-graph embedding methods to learn representations for the classification of signed networks, compared to more standard vertex embedding methods.

### 5.4.2 Signed Graph2vec

Graph2vec [161] enumerates rooted subgraphs to represent the neighborhood of a vertex. These rooted subgraphs are named using labels obtained with the relabeling procedure of the Weisfeiler-Lehman isomorphism test [203] (WL for short). Starting with the degree as the initial vertex label, this procedure goes through 2 phases to iteratively update these labels. First, each vertex is described by a tuple constituted of its previous label, and a sorted multiset containing those of its neighbors. Second, each unique tuple is replaced by a new label, to be used in the next iteration. 2 identical tuples are replaced by the same label, but 2 different one get distinct labels. This phase allows keeping a compact representation of the vertices. At the end of the process, each rooted subgraph is represented by its root's label. More formally, the label update rule is:

$$ \ell_t(u) = f(\ell_{t-1}(u), \{\ell_{t-1}(v) : v \in N(u)\}), \quad (5.3) $$

where $\ell_t(u)$ is the label of the subgraph rooted in $u$ at iteration $t$, $N(u)$ is the neighborhood of $u$, and $f$ is an injective function used to replace the tuples by new labels. The number of iterations corresponds to the desired order of the neighborhood covered by the rooted subgraph. A higher number of iterations leads to more distant vertices being considered during the relabeling phase.

At the end of this process, 2 isomorphic rooted subgraphs should get the same label. The obtained labels are then used to train the standard Doc2vec SkipGram model. Graph2vec has proven its effectiveness in many tasks involving the classification of unsigned graphs [204]–[206].

This method is not able to take advantage of the additional information present in signed graphs (i.e. edge signs), though. For this purpose, we define Signed Graph2vec, an adaptation of Graph2vec that relies on 2 variants of the WL relabeling procedure able to handle edge signs. The first one, noted SG2Vn (n for neutral), is straightforward and does not assume that the network has any form of structural balance. Regarding label initialization, instead of using the degree, we use both positive and negative degrees. For each vertex $u$, we first define a tuple $(k^+(u); k^-(u))$, and then replace each unique pair by a unique label using $f$. For the rest of the iterations, we proceed as in Graph2vec, except that we append the sign of the concerned edge in front of each neighbor when building the labels:

$$ \ell_t(u) = f(\ell_{t-1}(u), \{[s(u, v), \ell_{t-1}(v)] : v \in N(u)\}), \quad (5.4) $$
where $s(u, v)$ is the sign of edge $(u, v)$, and $[]$ denotes string concatenation. This allows distinguishing vertices holding the same label, but connected to the root with edges of opposed signs.

The second relabeling method, noted SG2Vsb, is the one proposed (for a different purpose) by Zhang et al. [207], and it assumes that the network is structurally balanced. Each vertex is represented by 2 labels, based on its positive and negative reachable sets, respectively (see Section 5.1). The authors do not explain how they perform their initialization, so we use the positive (resp. negative) degree for the positive (resp. negative) label. The method requires one update rule for each label:

$$ \ell_t^+(u) = f(\ell_{t-1}^+(u), \{\ell_{t-1}^+(v) : v \in N^+(u)\}, \{\ell_{t-1}^-(v) : v \in N^-(u)\}) \quad (5.5) $$

$$ \ell_t^-(u) = f(\ell_{t-1}^-(u), \{\ell_{t-1}^-(v) : v \in N^+(u)\}, \{\ell_{t-1}^+(v) : v \in N^-(u)\}), \quad (5.6) $$

At the end of the process, $f$ is applied to tuples formed by the positive and negative labels of each vertex, resulting in the final rooted subgraph labels.

### 5.4.3 Signed Graph Convolutional Networks

This method is based on SGCN, whose implementation is conveniently available online<fnref n="2" />, and can be modified to fit our needs. Similarly to SG2Vsb (Section 5.4.2), this method uses the balance theory to aggregate and propagate vertex representations across layers.

SGCN effectively learns representations of vertices in signed graphs. However, our objective is to handle whole graphs. A solution proposed for unsigned graphs in the literature [193] consists in adding a master node (also called virtual or super node), which is connected to all other vertices in the graph. One then uses the representation of this master node as the representation of the whole graph. The intuition is that, as the master node is connected to all parts of the graph, its representation is able to aggregate all its information.

In the context of signed graphs, connecting a master node to the rest of the graph is not trivial, though, as there are 2 different types of edges. We propose 5 interconnection schemes. Figure 5.3 shows an example for each one of them. The first 3 ones do not respect structural balance, and can be considered as baselines: SGCN+ (Figure 5.3.a) and SGCN- (Figure 5.3.b) consist in connecting a single master node to the rest of the graph using only positive and only negative edges, respectively. With SGCN± (Figure 5.3.c), we use 2 distinct master nodes (one positive and one negative), which allows us to combine both previous schemes at once. The whole-graph representation is obtained by summing both master node representations. The fourth scheme, SGCNsb (Figure 5.3.d), is based on the (strict) structural balance. Using the signnet library<fnref n="3" />, we detect the optimal graph bisection. Then, we connect one distinct master node to each cluster

<footnote n="2">²https://github.com/benedekrozemberczki/SGCN</footnote>

<footnote n="3">³https://github.com/schochastics/signnet</footnote>
<figure><img src="image_12.png" /><figcaption>Figure 5.3: Examples of the 5 proposed interconnection schemes: SGCN+ (a), SGCN- (b), SGCN±(c), SGCNsb (d) and SGCNgb (e). *MN* stands for Master Node. Green and red edges represent positive and negative connections, respectively.</figcaption></figure>

while respecting structural balance, i.e. positive edges within the cluster and negative ones with the
other cluster. Like before, the whole-graph embedding is obtained by summing the representations
of both master nodes. Finally, the fifth scheme, SGCNgb (Figure 5.3.e), relies on the *generalized*
balance. Like before, we find the optimal graph partition, but this time there may be more than
2 clusters. We add one master node for each cluster, and sum their representations to get the
whole-graph embedding.

5.5 Experiments

Our experimental protocol consists in assessing all the methods described in Section 5.4 on
the datasets and tasks presented in Section 5.3. As a reminder, there are 3 tasks each based
on a distinct dataset of signed graphs. The SO and CCS tasks contain 2 classes while EPF has
3. For all the tasks and methods, we first produce the representations and then train an SVM to
perform the classification, using a 10-fold cross-validation. For the sake of consistency, this is the
<table><caption>Table 5.2: Results in terms of macro F-measure obtained with SiNE. Each line focuses on a task while columns focus on the type of aggregation functions for vertex representations.</caption><thead><tr><td>Task</td><td>Sum</td><td>Average</td></tr></thead><tbody><tr><td>SpaceOrigin (SO)</td><td>55.42</td><td>50.21</td></tr><tr><td>Correlation Clustering Solutions (CCS)</td><td>50.48</td><td>48.62</td></tr><tr><td>European Parliament Factions (EPF)</td><td>69.81</td><td>67.58</td></tr></tbody></table>

same classifier and the same procedure as in Section 4. For all tasks, we experiment with 1 to 5 iterations or layers. We fix the limit to 5 because it is the maximum average diameter among our 3 datasets. Furthermore, increasing this parameter requires a lot of computational resources for some methods. We conduct our experiments on an Nvidia RTX 2080 Ti GPU. We first present and discuss our detailed results for each method in Section 5.5.1. Then, we compare the 3 families of methods in Section 5.5.2.

### 5.5.1 Results

In this section, we discuss each family of methods separately: SiNE (Section 5.5.1.1), Signed Graph2vec (Section 5.5.1.2), and Signed Graph Convolutional Networks (Section 5.5.1.3). All the performance values are expressed in terms of macro F-measure, i.e. by computing the F-measure for each class separately, then averaging them to get the overall performance. This allows giving the same importance to all classes, even in imbalanced datasets as explained in Section 2.4.2.2.

#### 5.5.1.1 Signed Network Embedding

We first present the results obtained with SiNE, the signed vertex embedding method that we consider as our baseline. The performance scores are shown in Table 5.2 for all 3 datasets.

On SO and CCS, the performances are close to a random classifier, which would get an expected 50% macro F-measure. The method is completely unable to learn correct representations of signed graphs for these 2 datasets. SiNE performs better on EPF with the best result reaching 69.81%. We assume that for this task, the local information available is often sufficiently discriminative. Moreover, vertex-oriented methods such as SiNE are usually designed to operate on very large graphs. This is not the case for the first 2 datasets. The graphs in EPF tend to be larger both in terms of vertices and edges. Hence, SiNE can extract a greater number of training triads, contributing to the improved model performance. Furthermore, the choice of the aggregation function has a notable impact on performance, with the sum operation yielding better results across all tasks. Averaging the representations has a tendency to smooth them which ultimately result in a loss of information.
5.5.1.2 Signed Graph2vec

Second, we present the results obtained with our 2 proposed variants of the *Signed Graph2vec*
method: SG2Vn (which does not enforce SB) and SG2Vsb (which does). As a reference, we also
include the original Graph2vec method (noted G2V) in the study, which simply ignores all edge
signs. The performances are shown in Table 5.3 for various numbers of iterations. As a reminder,
this parameter controls the number of iterations performed with the WL algorithm. In other terms, it
controls the order of the rooted subgraphs extracted to describe the graph, and therefore the range
taken into account when characterizing vertex neighborhoods. In the following, we discuss each
dataset separately.

5.5.1.2.1 SpaceOrigin Conversations

The top part of Table 5.3 presents the performances of the 3 Graph2vec variants on the *Signed SpaceOrigin* dataset. It appears that they are affected diversely by the number of iterations. On the one hand, G2V and SG2Vn get their best score with a single iteration, and increasing iterations tend to reduce the performance. This is a tendency that we already observed with this method in a previous work [19]. On the other hand, SG2Vsb starts low but increases consistently with the number of iterations. In the end, it reaches an *F*-measure of 77.44% and outperforms both other variants. The average diameter is 5.47 in this dataset, which means 5 iterations correspond to an almost complete graph coverage.

G2V and SG2Vn obtain very similar performances. From this observation, we can assume that a lot of information is already conveyed by the unsigned graph structures, in this dataset. This is consistent with the results obtained in the article that published the original data [6], and with our experiments in Section 4, where we had some success performing a similar task with the unsigned version of this dataset. Nevertheless, SG2Vsb obtain better performances than the other 2 variants which indicates that the signs bring some additional discriminative power, which SG2Vsb is able to leverage. By comparison, this is not the case of SG2Vn. We infer that the relatively low level of SB Frustration in this dataset (0.30) favors methods relying on SB.

5.5.1.2.2 Correlation Clustering Solutions

The results obtained for this dataset are shown in the middle part of Table 5.3. The performance
is clearly lower, for all 3 variants, and even similar to the score expected from a random classifier
(50%), with a maximum F-measure of 52.57% obtained by SG2Vn. As explained previously, the
dataset contains 2 subsets: some graphs are completely connected, and the rest are not. Assum-
ing that one type of graph might be more difficult to handle than the other, we try training separately
on these subsets. However, we do not get any significant difference between the obtained results,
<table><caption>Table 5.3: Results in terms of macro F-measure obtained with Graph2vec and our 2 proposed signed adaptations. Each column focuses on a specific number of iterations used when extracting rooted subgraphs.</caption><thead><tr><th>Task</th><th>Method</th><th>1 it.</th><th>2 it.</th><th>3 it.</th><th>4 it.</th><th>5 it.</th></tr></thead><tbody><tr><td rowspan="3">SpaceOrigin (SO)</td><td>G2V</td><td>75.09</td><td>71.32</td><td>72.62</td><td>73.96</td><td>73.77</td></tr><tr><td>SG2Vn</td><td>74.85</td><td>71.44</td><td>72.15</td><td>72.88</td><td>72.37</td></tr><tr><td>SG2Vsb</td><td>67.29</td><td>72.01</td><td>74.88</td><td>76.98</td><td><strong>77.44</strong></td></tr><tr><td rowspan="3">Correlation Clustering Solutions (CCS)</td><td>G2V</td><td>48.43</td><td>48.12</td><td>45.97</td><td>51.85</td><td>52.07</td></tr><tr><td>SG2Vn</td><td>49.70</td><td>49.25</td><td>48.37</td><td><strong>52.57</strong></td><td>52.12</td></tr><tr><td>SG2Vsb</td><td>49.84</td><td>51.81</td><td>49.70</td><td>51.37</td><td>51.62</td></tr><tr><td rowspan="3">European Parliament Factions (EPF)</td><td>G2V</td><td>45.63</td><td>49.76</td><td>52.31</td><td>60.43</td><td>63.14</td></tr><tr><td>SG2Vn</td><td>75.63</td><td>81.44</td><td>84.18</td><td>86.16</td><td>88.68</td></tr><tr><td>SG2Vsb</td><td>80.83</td><td>82.90</td><td>86.31</td><td>87.99</td><td><strong>89.98</strong></td></tr></tbody></table>

which are similar to those of Table 5.3.

We assume that either this classification task is too hard, in the sense that the information available in the graphs is not sufficient to perform the prediction, or that none of the 3 Graph2vec variants are able to capture the relevant information. The value predicted in this task is directly related to the distribution of edge signs (see Section 5.3.1), so we know with certainty that the information conveyed by signs is essential. The fact that G2V has similar performance to its signed counterparts hints at the second assumption (methods unable to capture relevant information). In addition, the higher level of SB Frustration (0.37) may hinder the performance of SG2Vsb, compared to SG2Vn.

Increasing the number of iterations eventually improves the performance, but the effect is not as strong and as stable as for the previous dataset. The graphs are more compact in this dataset, with an average diameter of only 3.60, which may partially explain this observation. Indeed, a few iterations are enough to retrieve all available information, and increasing their number does not bring any new neighbors.

### 5.5.1.2.3 European Parliament Factions

The results for this dataset are shown in the bottom part of Table 5.3. It appears that both our signed adaptations perform drastically better than the original unsigned method. The difference in F-measure is the largest of the 3 datasets: 63.14 (G2V vs. 88.68 (SG2Vn) and 89.98 (SG2Vsb). Edge signs thus appear to be even more important for this task than it was for the Signed SpaceOrigin dataset. The SB-based variant SG2Vsb is slightly above SG2Vn, which seems to indicate that this
type of structure may be relevant to this classification task. The average SB Frustration is 0.28 for this dataset.

This dataset contains larger graphs than the others, with an average of 67.34 vertices (vs. 47.74 and 27.31 previously) and a mean diameter of 4.12. This may explain the very strong effect of the number of iterations on the performance, even for G2V, the unsigned variant. It seems that even a small increase (proportionally to the network size) of the part of the graph covered when extracting rooted subgraphs, is enough to greatly improve the quality of the classification.

### 5.5.1.3 Signed Network Embedding

In this section, we present the results obtained with the 5 variants that we proposed for the SGCN method. Each one relies on the addition of one or several master nodes, through different interconnection schemes. The first 3 ($SGCN^+$, $SGCN^-$, $SGCN^{\pm}$) ignore any type of structural balance, whereas the others enforce respectively SB ($SGCN^{sb}$) and GB ($SGCN^{gb}$). We consider 2 methods to extract a graph representation: using only the last layer vs. the sum of all layers. Preliminary experiments show that the former performs better, so we only focus on this approach in the following discussion.

We also include SGCN as a reference in our study, i.e. the original method without any master node. To get a graph-level representation, we proceed like for SiNE, and sum the representations of all vertices. We alternatively experimented with averaging them, but got lower performance, which explains why we focus only on sum, here. The results are shown in Table 5.4, for various numbers of convolution layers. Each additional layer allows considering neighbors that are located 1-hop farther away from the vertex. In the following, we discuss each dataset separately.

#### 5.5.1.3.1 Signed Space Origin

The top part of Table 5.4 shows the results on the *Signed SpaceOrigin* dataset. Increasing the number of layers in the convolutional network results in better performances for all variants, but to different extents. For instance, when going from 1 to 5 layers, the performance gain is +7,1 *F*-measure points for $SGCN^{gb}$ but only +0.11 for $SGCN^{\pm}$. With an average diameter of 5.47 in this dataset, using 5 layers allows an almost complete graph coverage.

All 3 variants that ignore any form of balance ($SGCN^+$, $SGCN^-$, $SGCN^{\pm}$) are below the unsigned baseline (SGCN), which indicates that using signs improperly is counterproductive as it decreases the classification performance. $SGCN^+$ largely outperform both other variants, probably because there are many more positive than negative edges in this dataset.

Among the variants that take balance into account, $SGCN^{gb}$ consistently outperforms $SGCN^{sb}$: 55.67 vs. 73.69 with 5 layers. This shows that the type of balance selected when learning the representation must match the structural properties of the considered graphs. Interestingly, SGCN
<table><caption>Table 5.4: Results in terms of macro F-measure obtained with SGCN and our 5 proposed interconnection schemes. Each column focuses on a specific number of convolution layers.</caption><thead><tr><th>Task</th><th>Method</th><th>1 lay.</th><th>2 lay.</th><th>3 lay.</th><th>4 lay.</th><th>5 lay.</th></tr></thead><tbody><tr><td rowspan="6">SpaceOrigin (SO)</td><td>SGCN</td><td>66.48</td><td>67.21</td><td>68.06</td><td>68.87</td><td>69.54</td></tr><tr><td>SGCN+</td><td>65.12</td><td>66.78</td><td>68.42</td><td>68.98</td><td>69.29</td></tr><tr><td>SGCN-</td><td>54.19</td><td>54.89</td><td>55.56</td><td>55.59</td><td>55.89</td></tr><tr><td>SGCN±</td><td>49.28</td><td>48.95</td><td>49.01</td><td>49.25</td><td>49.39</td></tr><tr><td>SGCNsb</td><td>52.69</td><td>54.99</td><td>55.49</td><td>55.08</td><td>55.67</td></tr><tr><td>SGCNgb</td><td>66.59</td><td>68.85</td><td>71.21</td><td>72.28</td><td><strong>73.69</strong></td></tr><tr><td rowspan="6">Correlation Clustering Solutions (CCS)</td><td>SGCN</td><td>70.29</td><td>70.65</td><td>71.27</td><td>71.43</td><td>71.89</td></tr><tr><td>SGCN+</td><td>70.50</td><td>70.86</td><td>71.13</td><td>71.27</td><td>71.35</td></tr><tr><td>SGCN-</td><td>70.14</td><td>70.55</td><td>70.84</td><td>71.02</td><td>71.18</td></tr><tr><td>SGCN±</td><td>70.76</td><td>71.02</td><td>71.00</td><td>71.20</td><td>71.37</td></tr><tr><td>SGCNsb</td><td>69.59</td><td>70.12</td><td>70.86</td><td>71.21</td><td>71.46</td></tr><tr><td>SGCNgb</td><td>71.75</td><td>72.20</td><td>72.98</td><td>73.24</td><td><strong>73.49</strong></td></tr><tr><td rowspan="6">European Parliament Factions (EPF)</td><td>SGCN</td><td>90.16</td><td>90.87</td><td>91.63</td><td>92.04</td><td>92.65</td></tr><tr><td>SGCN+</td><td>88.56</td><td>89.30</td><td>90.49</td><td>91.11</td><td>91.87</td></tr><tr><td>SGCN-</td><td>88.61</td><td>89.30</td><td>90.49</td><td>91.09</td><td>91.80</td></tr><tr><td>SGCN±</td><td>86.32</td><td>88.45</td><td>90.01</td><td>90.78</td><td>91.56</td></tr><tr><td>SGCNsb</td><td>91.11</td><td>92.09</td><td>93.31</td><td>94.17</td><td>94.99</td></tr><tr><td>SGCNgb</td><td>92.36</td><td>93.65</td><td>95.29</td><td>96.04</td><td><strong>96.43</strong></td></tr></tbody></table>

is the second-best method, which illustrates the methodological importance of the interconnection scheme when using a master node approach. It is on par with SGCNgb when using a single layer, because they are equivalent for this specific parameter value, however the difference quickly grows with the number of layers.

### 5.5.1.3.2 Correlation Clustering Solutions

The middle part of Table 5.4 shows the F-measure scores for the Correlation Clustering Solutions. On this dataset, all methods yield quite similar results, except for SGCNgb that once again obtains the best performances. In particular, this method is able to capture more information at the whole-graph level than SGCN at the vertex level. Increasing the number of layers still improves the results, but the effect is much weaker than for the previous dataset. The graphs are smaller there, which may explain this, as more layers do not bring more information after a certain point. These results also show that this task is not as challenging as assumed when discussing Graph2vec re-
results, since it is possible to get scores much higher than the expected performance of a random classifier.

### 5.5.1.3.3 European Parliament Factions

The bottom part of Table 5.4 shows the results obtained for the EPF graphs. The behavior on this dataset is similar enough to that on *Signed SpaceOrigin*: the 3 variants that ignore balance (SGCN+, SGCN-, SGCN±) are below the original SGCN method (the latter being the worst, again), whereas SGCNgb gets the best results, peaking at 96.43% when using 5 layers. There are 3 differences, though: first, the overall performance is much better, with a minimal *F*-measure of 86.32. This is in line with the behavior exhibited by Graph2vec on the same datasets, and could be explained by the low level of Frustration (0.20). Second, SGCN+ and SGCN- perform similarly. Third, SGCNsb is above SGCN, which could mean that many graphs have a 2-cluster structure in this dataset.

## 5.5.2 Comparison

We now compare and analyze the results of the 3 families of methods. Our baseline based on SiNE is consistently the least efficient method on all tasks. The gap of *F*-measure reaches up to 22.02 points with the best method, on the *Signed SpaceOrigin* dataset. This could be explained in part by the size of the graphs that constitute our benchmark: these are relatively small, whereas SiNE was designed to handle large graphs, with hundreds or thousands of vertices. Nevertheless, it remains that our results indicate that applying a method designed to handle whole graphs directly leads to much better classification results than simply aggregating multiple vertex representations.

Signed Graph2vec, through its SG2Vsb variant, obtains the best performance on the *Signed SpaceOrigin* dataset, while SGCN, through its SGCNgb variant, dominates on both other datasets. Overall, the variants that respect some form of structural balance obtain better results than the ones that do not. This confirms the interest of including this notion in the design of representation learning methods for signed graphs, in order to preserve this property in the representation space. However, it is worth noticing that datasets do not equally favor the 2 types of balance considered here: *Signed Space Origin* leads to better results with SB, whereas both others favor GB.

Increasing the number of iterations in Graph2vec, or that of layers in SGCN, has a positive impact on performances for all datasets, and for almost all variants, including the unsigned ones. This effect is generally stronger when the appropriate type of balance is leveraged to aggregate the representation of direct and indirect neighbors, though, which confirms the importance of this concept when dealing with signed graphs. For the SGCN variants, it also shows that the method does not suffer from oversmoothing on the considered datasets.

Signed methods consistently obtain better performances than the unsigned Graph2vec method
and all datasets. This proves that signs are indeed relevant information for the considered graph classification tasks. Including them directly in the representation learning process improves the quality of the learned representations for all tasks.

Graph Convolutional Networks are the best performing method, overall. However, this comes at a cost: they also are the most expensive in terms of computational runtime. Learning the representation of a graph with SGCN variants takes more than 25 seconds in average. As a comparison, Graph2vec variants take an average of 0.15 second per graph, and SiNE 0.65 second.

## 5.6 Conclusion

In this chapter, we have presented 2 methods for signed whole-graph embedding. The first is an adaptation to signed whole graphs of Graph2vec, an *unsigned* whole-graph embedding method. We generalize the original Weisfeiler-Lehman relabeling procedure in order to handle edge *signs*. Our second proposed method is based on SGCN, a Graph Convolutional Network method designed to learn signed *vertex* representations, which we adapt to handle signed *whole graphs*. Our proposition is to integrate one or several master nodes connected to all parts of the graph, and to use their representation as the representation of the whole graph. The intuition is that, as the master node is connected to all parts of the graph, this representation is able to aggregate all its information. This idea was first proposed for unsigned graphs in the literature, and we propose solutions to adapt it to signed graphs. The challenge is to determine how to connect the master node to the rest of the graph, since there are 2 different types of edges. We propose 5 interconnection schemes.

We constituted a benchmark for signed graphs classification containing artificially generated and real-world graphs. By applying our methods to 3 graph classification tasks included in this benchmark, we could show that both of them are more efficient on all tasks than our baseline based on the aggregation of vertex representations. In particular, methods respecting the structural balance obtain better results than the ones that do not. Therefore, this is a property that must be considered when developing an embedding method for signed graphs. To return to the 2 questions that we wanted to answer throughout this chapter, our experimental results show that methods specifically designed to handle whole graphs are indeed able to learn better representations of such objects compared to vertex oriented methods, especially for our dataset. This is an important result, since this is a part of the literature that is currently rather neglected. The second question was about the interest of using signed representations of graphs. We demonstrated that the signed methods were more effective than unsigned ones on all the datasets, confirming the benefit of considering signs.

We believe that this work opens new directions for future research. First, our proposed benchmark of 3 graph classification tasks allows evaluating and comparing methods designed for signed
whole-graph representation learning. This benchmark could be expanded with new collections of
graphs, especially larger ones. Second, the master node approach could be applied to other types
of signed GNNs such as Graph Attention Networks. Third, it would be interesting to generalize
our methods so that they can use edge weights, as we showed in Section 4 that this brings some
additional discriminative power when available: we propose such an extension in Chapter 6.1.3.
Finally, another interesting future direction is to experiment with other master node interconnection
schemes, based on other variants of Structural Balance. The concept of *Relaxed Balance* [208],
[209], in particular, is very interesting, as it allows inter-cluster (resp. intra-cluster) edges to be
positive (resp. negative). With these signed whole-graph embedding methods, we now have all
the tools available to apply representation learning to abuse detection. This is the subject of our
next chapter.
REPRESENTATION LEARNING FOR ABUSE
DETECTION

<table><tbody><tr><td><strong>6.1</strong></td><td><strong>Embedding Methods</strong></td><td><strong>98</strong></td></tr><tr><td>6.1.1</td><td>Lexical Embedding Methods</td><td>98</td></tr><tr><td>6.1.2</td><td>Selected Graph Embedding Methods</td><td>100</td></tr><tr><td>6.1.3</td><td>Proposed Whole-Graph Embedding Methods</td><td>103</td></tr><tr><td><strong>6.2</strong></td><td><strong>Experiments</strong></td><td><strong>106</strong></td></tr><tr><td>6.2.1</td><td>Experimental Protocol</td><td>107</td></tr><tr><td>6.2.2</td><td>Classification Results</td><td>107</td></tr><tr><td>6.2.3</td><td>Fusion of Embeddings</td><td>113</td></tr><tr><td>6.2.4</td><td>Results Summary</td><td>115</td></tr><tr><td><strong>6.3</strong></td><td><strong>Feature Study</strong></td><td><strong>115</strong></td></tr><tr><td>6.3.1</td><td>Text Features</td><td>116</td></tr><tr><td>6.3.2</td><td>Graph Features</td><td>116</td></tr><tr><td><strong>6.4</strong></td><td><strong>Conclusion</strong></td><td><strong>118</strong></td></tr></tbody></table>

In Chapter 4, we noted 2 main limitations of feature-based approaches. On the one hand, one has to manually search the existing measures and select the most appropriate ones to constitute the feature set. To be exhaustive, it often has to include a few hundred features. On the other hand, the computation of these feature sets can be resource-intensive, in particular that of graph-based methods. Representation learning can solve this problem by automatically learning appropriate embedding representations of text and graphs. These methods can rely on edge attributes such as signs, weights, and directions to improve the representations. However, no signed whole-graph embedding method existed in the literature. In Chapter 5, we filled this gap by proposing 2 such approaches and showed that they performed better than their unsigned counterparts. In this chapter, we leverage these approaches, together with other existing representation learning methods, to tackle the same task as in Chapter 4, and thus study how automatically learned representations compare with feature engineering.
<figure><img src="image_13.png" /><figcaption>Figure 6.1: Illustration of the differences between the feature-based method presented in Section 4.2.1 (in orange) and text embedding methods used throughout this chapter (in red).</figcaption></figure>

Lexical embeddings transform words into vectors that preserve their semantic and syntactic information. These methods offer robust representations that can help overcome the limitations of standard NLP techniques. Different methods can learn representations of text at different levels of granularity. Some of them provide representations of words, while others provide representations of sentences or documents. Textual embeddings have been used in a wide range of applications, including the detection of sexist and racist comments [49] and the offensive or hate speech detection [83], [210].

Graph embedding methods learn representations of various parts of graphs (i.e. vertices, edges, subgraphs, whole graphs, etc.). By construction, different methods are assumed to capture different aspects of the graph structure or properties, but it is difficult to compare them directly, for the same reason. One way to assess the appropriateness of a graph embedding method for a task is to do so empirically [170].

In this chapter, we apply embedding techniques to the abuse detection task. We follow the same procedure as in Chapter 4 with the exception that representations of text messages and conversational graphs are automatically learned through embedding methods instead of features. The differences between these 2 methodologies are illustrated by Figure 6.1 (text) and Figure 6.2 (graphs). We perform extensive experiments to compare multiple textual and graph embedding methods, including the signed whole-graph embedding approaches proposed in Chapter 5, to assess their appropriateness for this specific task. Our objective here is to determine how embedding methods perform compared to feature-based ones. Since the learned representations are not directly interpretable, it is not straightforward to understand exactly which information is captured by the embeddings. Therefore, we also perform an in-depth analysis of the methods to detect which properties are captured by which methods.

The following chapter is based on our works published in the international journal *SN Computer*
<figure><img src="image_14.png" /><figcaption>Figure 6.2: Illustration of the differences between the feature-based method presented in Section 4.2.2 (in orange) and graph embedding methods used throughout this chapter (in green).</figcaption></figure>

*Science* (2021) [19], the French journal *Traitement Automatique des Langues* (2021) [21], and the *MARAMI* 2020 conference [22]. Some of the results on edge attributes and directions have not yet been published. This chapter makes the following contributions:

1. We propose extensions of the SG2V and SGCN models from Chapter 5 that can use edge weights and directions in addition to the signs

2. We propose 3 variants of Graph2vec, a graph embedding method, that integrate information relative to our task.

3. We experiment and compare 6 lexical embedding methods and 11 graph embedding methods designed to operate at different scales of the graph (vertex and whole graph). We assess these methods on the abuse detection task on the SpaceOrigin dataset and estimate their complementarities by combining them following the 3 fusion strategies presented in Section 4.2.3.

The rest of this chapter is organized as follows. In Section 6.1, we present all the lexical embedding methods that we use, the parameters of graph embedding approaches, the weighted directed extensions of the 2 signed whole-graph embedding methods developed in the previous chapter and the variants of Graph2vec that we propose to incorporate task-specific information. Next, we put all these methods into practice on our abuse detection task and discuss their results in Section 6.2. Then, we put a focus on determining which information is captured by every method in Section 6.3. Finally, we review our main findings in Section 6.4, and identify some perspectives for this work.
## 6.1 Embedding Methods

In this section, we present the lexical models from the literature, which we use in our experiments (Section 6.1.1). We then summarize the existing methods from the literature that we selected to produce graph embeddings (Section 6.1.2). Finally, we present some extensions of graph embedding approaches designed to fit our needs (Section 6.1.3.3).

### 6.1.1 Lexical Embedding Methods

Representation learning for text data is the task that consists of transforming textual input into vector representations which can then be used by machine learning algorithms for various tasks. Word embedding methods learn dense vectors of fixed size that represent a set of words, one vector per token in the vocabulary. They efficiently encode the semantic and syntactic information of words. Those that are used in similar ways obtain very close representations in the vector space, naturally capturing their meaning. However, representing polysemy and homonymy is difficult. We can distinguish 2 categories of word embedding approaches: the ones that are context-invariant and output a *fixed* representation for each word, and the *contextualized* ones that can generate different embeddings for a given word depending on the context in which it is used. Here, we experiment with 5 methods, the first 2 are context-invariant while the other 3 are contextualized. One constraint of our dataset is that it is constituted of French messages. Therefore, we selected models for which a pre-trained French version is publicly available.

**Word2vec** [174] generates fixed representations of words. It is one of the most popular word embedding approaches in recent years. In particular, it proposes the SkipGram model architecture which has been reused in numerous methods since. The model learns representations while trying to keep the semantic and/or syntactic similarity of words. Word2vec is based on a distributional hypothesis and learns the meaning of words from a large corpus of texts. Technically, it uses a neural network that has an input layer, an output layer, and a projection layer. The latter constitutes the word embedding. The output layer is used to perform a classification task allowing to train the model. There are 2 variants of Word2vec with symmetrical architectures: the Continuous Bag of Words (CBOW) model that predicts a word based on its context fed as input, and the SkipGram model that aims to predict the surrounding context words given the center word. Word2vec models are trained with a sliding window process, i.e. at each iteration, the model targets a word before moving one word forward. The size of that window determines how many words before and after a given word are included in its context. In this application, we set it to 10 words. We use a model<fnref n="1" /> trained on French texts extracted from Wikipedia. Each word is associated with a representation of dimension 300.

<footnote n="1">¹https://github.com/Kyubyong/wordvectors</footnote>
Since the representation process of Word2vec is based on the words observed in the training corpus, this method is unable to deal with out-of-vocabulary words. For the same reason, uncommon words that appear only a few times in the training phase often obtain poor-quality representations. **fastText** [211] is an extension of Word2vec that was developed to mitigate the aforementioned limitation. It breaks words down to *N*-grams of characters instead of treating full words as Word2vec does. In this way, each word can be represented as a bag-of-characters *N*-grams (with $3 \le N \le 6$, generally). For instance, with $N = 3$, the word *mode* would be represented by `<mo,mod,ode,de>` where the `" < "` and `" > "` symbols correspond to the beginning and the end of the word. Once the character *N*-grams are extracted, a SkipGram model learns their representations. The representation of a word is obtained by summing all its *N*-grams. With this approach, rare and out-of-vocabulary words can still have quality representations since it is very likely that at least a portion of their *N*-grams appear in other words. We use a model<fnref n="2" /> [211] trained on French texts from Wikipedia. All the representations are of dimension 300.

**CamemBERT** [212] and **FlauBERT** [213] are 2 French word embedding models directly adapted from the RoBERTa [214] architecture. Therefore, they share a lot of similarities. The main difference between them lies in the way that they mask tokens in the masked language model. CamemBERT also uses more training data. They both provide contextualized representations of words.

**CamemBERT** [212] is trained on a masked language modeling task. The authors follow the original architecture configurations of BERT. They propose a CamemBERT<sub>BASE</sub> model with 12 layers of encoders, 768 hidden dimensions, 12 attention heads and 110M parameters, and a CamemBERT<sub>LARGE</sub> model with 24 layers of encoders, 1,024 hidden dimensions, 16 attention heads and 335M parameters. According to the authors' study [212], the latter gives better results on named entity recognition but similar results on part-of-speech and dependency parsing tasks. Thus, we use the CamemBERT<sub>LARGE</sub> model in the remainder of this chapter. This model³ is pre-trained on the CCNet [215] corpus containing 135GB of French text extracted from various websites. All the representations are of size 1,024.

Similar to CamemBERT, **FlauBERT** [213] proposes a FlauBERT<sub>BASE</sub> model composed of 12 layers of encoders, 768 hidden dimensions and 12 attention heads, and a **LARGE** model with 24 layers of encoders, 1,024 hidden dimensions and 16 attention heads. These models are trained on a corpus of 71GB of French text aggregated from multiple online sources. We use the **LARGE** model⁴ that provides embeddings of dimension 1,024.

Unlike the 4 previous approaches, **Flair** [216] learns representations that are built at the character level and not the word level. This feature allows Flair to be particularly efficient when dealing with rare or misspelled words and to better model language-related concepts such as prefixes

<footnote n="2">²https://github.com/flairNLP/fasttext
³https://camembert-model.fr/
⁴https://github.com/getalp/Flaubert</footnote>
and suffixes. This architecture uses a bi-directional LSTM operating on characters. The model is trained to predict the next character for each element in the sequence to process. Therefore, it learns 2 hidden representations for each character of the sequence: one for the forward network and one for the backward network. The embedding of a word is obtained by combining the forward representations of the characters located before the end of the word and the backward representations of the characters located after the beginning of the word. We use the forward and backward models<sup>5</sup> trained on French Wikipedia articles. Each model generates representations of size 1,024 that we concatenate to obtain the final embedding of a word, with 2,048 dimensions.

6.1.2 Selected Graph Embedding Methods

In this chapter, we compare several graph embedding techniques applied to abuse detection. This task consists of classifying conversational graphs. However, our approach to represent conversations through conversational graphs is particular. Indeed, vertices represent authors and the objective is to classify one message posted by a specific author, given the context represented by the whole graph. Therefore, one could assume that the vertex corresponding to this specific author is more important than other vertices and that using the representation of that single vertex could be enough to characterize the whole conversation. Furthermore, it is possible to construct a representation of the graph by combining vertex-level features. For this reason, we include both vertex and whole-graph embedding methods in this study. The methods from the literature that we selected use different strategies and focus on preserving various aspects of the graph, to include as much diversity as possible. All implementations are from the *Karate club toolkit* [201] except Node2vec, which was implemented by E. Cohen<sup>6</sup>.

In the remainder of this section, we list the methods selected for our experiments, half of them treating the graph at the vertex scale, and the other half treating the whole graph. We selected them to represent a diversity of techniques. For each, we indicate their parameter settings, which we experimentally determined in a previous study [19]. In the following description, the names of the parameters correspond to those used in these toolboxes. Table 6.1 summarizes the exact parameter settings for all the graph embedding methods, which are described in more detail in the previous chapter (Section 5.2). Table 6.2 shows a summary of the edge attributes (weights, directions, signs) that the graph embedding methods are able to handle.

6.1.2.1 Vertex Embedding Methods

In the conversational graph extracted from the context of the targeted message, all vertices
are not equal. One of them represents the author of the targeted message (represented in red

<footnote n="5">⁵https://github.com/flairNLP/flair</footnote>

<footnote n="6">⁶https://github.com/eliorc/node2vec</footnote>
<table><caption>Table 6.1: Parameters of the graph embedding methods.</caption><thead><tr><th>Parameter</th><th>FGSD</th><th>SF</th><th>NGNN</th><th>G2V</th><th>SG2V</th><th>SGCN</th><th>DW</th><th>WL</th><th>N2V</th><th>BNE</th><th>GW</th><th>KH</th></tr></thead><tbody><tr><td>dimensions</td><td>200</td><td>128</td><td>64</td><td>128</td><td>128</td><td>128</td><td>128</td><td>32</td><td>128</td><td>8</td><td>100</td><td>64</td></tr><tr><td>hist_range</td><td>10</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>wlIterations</td><td>-</td><td>-</td><td>-</td><td>1</td><td>5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>layers</td><td>-</td><td>-</td><td>8</td><td>-</td><td>-</td><td>5</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>down_sampling</td><td>-</td><td>-</td><td>-</td><td>10<sup>-4</sup></td><td>10<sup>-4</sup></td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>learning_rate</td><td>-</td><td>-</td><td>10<sup>-3</sup></td><td>0.06</td><td>0.06</td><td>-</td><td>0.05</td><td>0.05</td><td>-</td><td>-</td><td>-</td><td>10<sup>-3</sup></td></tr><tr><td>epochs</td><td>-</td><td>-</td><td>300</td><td>12</td><td>12</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>200</td></tr><tr><td>min_count</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-</td><td>-</td><td>1</td><td>1</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>window_size</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>10</td><td>4</td><td>10</td><td>-</td><td>-</td><td>-</td></tr><tr><td>walk_number</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>5</td><td>5</td><td>10</td><td>-</td><td>-</td><td>-</td></tr><tr><td>walk_length</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>80</td><td>80</td><td>20</td><td>-</td><td>-</td><td>-</td></tr><tr><td>p</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.95</td><td>-</td><td>-</td><td>-</td></tr><tr><td>q</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1.0</td><td>-</td><td>-</td><td>-</td></tr><tr><td>iterations</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>16</td><td>-</td><td>-</td></tr><tr><td>order</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1</td><td>-</td><td>-</td></tr><tr><td>alpha</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.01</td><td>-</td><td>-</td></tr><tr><td>step_size</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.2</td><td>-</td></tr><tr><td>heatCoefficient</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.5</td><td>-</td></tr><tr><td>approximation</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>100</td><td>-</td></tr><tr><td>switch</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>1,000</td><td>-</td></tr></tbody></table>

in Figure 4.4), which we assume plays a particular role if an abuse is occurring at this moment of the conversation. We can suppose that this vertex is the most important in the graph, and that characterizing it individually could be enough to represent the full conversation. The vertex embedding methods presented in this section allow learning the representation of this vertex.

**DeepWalk** [175] (DW) samples a certain number of uniform random walks starting from each vertex to represent the graph structure. There are multiple parameters to tweak the generation of random walks. We extract 5 random walks starting at each vertex of the graph with a maximum length of 80. We use a neighborhood of 10 vertices and consider all vertices appearing at least once in the graph. The representations are of dimension 128.

**Node2vec** [85] (N2V) also uses random walks to characterize the graph structure. When drawing these random walks, it introduces a bias controlled by 2 parameters. The return parameter *p* controls the likelihood of immediately revisiting a vertex during the walk, and the in-out parameter *q* controls the balance between the breadth-first and depth-first strategies. We use values 0.95 and 1.00 respectively in order to obtain representations that capture the global structure of the graph and the structural equivalence between vertices. We extract 10 random walks of length 20 starting from each vertex with a *window_size* of 10. The representations are of size 128.

**Walklets** [179] (WL) is an extension of DeepWalk which aims at explicitly modeling multi-scale
relationships, i.e. combining distinct views of vertex relationships at different granularity levels. The key change is in the random walk sampling algorithm, as the walks can now skip some vertices to reach farther parts of the network. It creates a representation for each size of *skip*) and the output representation is the result of their concatenation. We use the same parameters as DeepWalk, except for the *window_size* that denotes the size of *skips* in random walks. We use a value of 4 and 32 dimensions for each representation. The final representation is thus of size 128, the product of the values of these 2 parameters.

**BoostNE** [173] (BNE) also learns multiple graph representations at different granularity levels, but unlike Walklets, it relies on matrix factorization. It successively factorizes a vertex connectiv-ity matrix to obtain representations corresponding to an increasingly finer granularity. The final embedding is obtained by concatenating these representations. We consider 16 iterations that each generates a representation of size 8, for a final embedding of size 128.

**GraphWave** [217] (GW) mimics a physical process consisting in propagating some energy through the graph structure, starting from the vertex of interest. The way this energy is diffused over the graph is assumed to characterize the vertex and its neighborhood. The *heatCoefficient*, which corresponds in terms of graph structure to the radius of the considered vertex neighborhood, is set to 0.5. We set the size of the embeddings to 100.

**k-hop Graph Neural Networks** [218] (KH) aggregates the direct neighbors of a vertex and its k-hop neighborhood through a graph neural network. We use a learning rate of 0.001 with 200 epochs to obtain representations of dimension 64.

### 6.1.2.2 Whole-Graph Embedding Methods

Using a description of the whole graph amounts to consider the entire conversation at once when performing the classification. It can help better capturing the global dynamics of the ex-changes between users in the graph.

**Family of Graph Spectral Distances** [185] (FGSD) discretizes the distribution of the vertex pairwise distances through a histogram. We can control the way this histogram is computed. We set the range covered by our histogram to 10 and generate representations of dimension 200.

**Spectral Features** [183] (SF) computes the spectrum of the normalized graph Laplacian, keeping only the *k* smallest positive Eigenvalues. These Eigenvalues, in ascending order, form the representation of the graph. Parameter *k*, called *dimensions* in the implementation, therefore directly controls the size of the representation. We set it to 128.

**Nested Graph Neural Networks** [219] (NGNN) extracts rooted subgraphs and pool their representations into a single one to represent the whole graph. We use a learning rate of 0.001, with 300 epochs and 8 GNN layers to learn representations of dimension 64.

**Graph2vec** [161] (G2V) considers a graph as a collection of subgraphs, which are then used to train a SkipGram model. More precisely, it looks for so-called *rooted* subgraph, i.e. vertex
neighborhoods of a certain order. We extract rooted subgraphs of degree 1 (wlIterations), with an embedding size of 128.

**Signed Graph2vec (SG2V)** is a method that we proposed in Chapter 5.4.2. It is an adaptation of the original Graph2vec approach that is able to learn representation of signed graphs. Following the results in Chapter 5.5.1, we extract rooted subgraphs of degree 5 (wlIterations) with a representation size of 128. We use the variants which enforce structural balance (SG2Vsb).

**Signed Graph Convolutional Networks (SGCN)** is the other signed whole-graph embedding method that we developed (Chapter 5.4.3). It includes master vertices to connect some parts of the graph and uses the representations of such global vertices to represent the whole graph. We use 5 layers in our graph convolutional network and generate representations of size 128. We use the best performing variant (SGCNgb), which enforces the generalized balance.

<table><caption>Table 6.2: Summary of the edge attributes handled by the selected graph embedding methods.</caption><thead><tr><th>Scale</th><th>Method</th><th>Reference</th><th>Weighted</th><th>Directed</th><th>Signed</th></tr></thead><tbody><tr><td rowspan="6">Vertices</td><td>DeepWalk</td><td>[175]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Node2vec</td><td>[85]</td><td>✓</td><td>✓</td><td>-</td></tr><tr><td>Walklets</td><td>[179]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>BoostNE</td><td>[173]</td><td>✓</td><td>-</td><td>-</td></tr><tr><td>GraphWave</td><td>[217]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>KH</td><td>[218]</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan="11">Whole graph</td><td>FGSD</td><td>[185]</td><td>✓</td><td>-</td><td>-</td></tr><tr><td>Spectral Features</td><td>[183]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>NGNN</td><td>[219]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SG2V</td><td>Section 5.4.2</td><td>-</td><td>-</td><td>✓</td></tr><tr><td>WD-SG2V</td><td>Section 6.1.3.1</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>SGCN</td><td>Section 5.4.3</td><td>-</td><td>-</td><td>✓</td></tr><tr><td>WD-SGCN</td><td>Section 6.1.3.2</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Graph2vec</td><td>[161]</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Graph2vec-author</td><td>Section 6.1.3.3</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Graph2vec-target</td><td>Section 6.1.3.3</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Graph2vec-distance</td><td>Section 6.1.3.3</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

### 6.1.3 Proposed Whole-Graph Embedding Methods

In Chapter 5, we proposed SG2V and SGCN as whole-graph embedding methods able to operate on undirected unweighted signed graphs. However, in Chapter 4.4.2 we found that consid-
ering edge attributes and directions has a positive impact on the detection performance of abusive messages. Consequently, we think that including edge weights and directions directly in the representation learning process of SG2V and SGCN could improve the quality of their learned representations of whole graphs. We first introduce a Weighted Directed variant of SG2V (WD-SG2V) in Section 6.1.3.1 that generalizes the Weisfeiler-Lehman (WL) relabeling procedure in order to handle these supplementary edge attributes. Then, we propose a Weighted Directed variant of SGCN (WD-SGCN) in Section 6.1.3.2. Finally, we propose an extension of the Graph2vec model in Section 6.1.3.3, which includes certain task-related information into the embedding.

### 6.1.3.1 Weighted Directed SG2V

In Section 5.4.2, we proposed 2 signed adaptations of the Graph2vec model relying on 2 variants of the WL relabeling procedure able to handle edge signs. To incorporate supplementary edge attributes in the learned representations, we follow the same procedure and propose a new variant able to handle edge weights and directions. The original Graph2vec label update rule is an iterative process where a vertex is described by a tuple constituted of its previous label, and a sorted multiset containing those of its neighbors. It is formally defined in Equation 5.3. In order to incorporate both other edge properties, we first consider the set of outgoing neighbors of a vertex $u$ and split them into 4 quartiles based on the weight of the concerned edges. Based on this breakdown, we can attribute a quartile number (i.e. 1, 2, 3 or 4) to each edge. This allows distinguishing vertices holding the same label but connected to $u$ with different weight levels. We experimentally determined that splitting the weights into quartiles is the best option, as using fewer quartiles does not provide enough possibilities to distinguish edges, and using more of them leads to too many possible labels, hence reducing the efficiency of the method.

The first variant that we proposed in Section 5.4.2, SG2Vn, uses the original label update rule except that it appends the sign of the concerned edge in front of each neighbor (Equation 5.4). Our proposition for WD-SG2Vn is to further append the quartile number in front of each neighbor when building the labels in the following manner:

$$ \ell_t(u) = f(\ell_{t-1}(u), \{[q(u, v), s(u, v), \ell_{t-1}(v)] : v \in N(u)\}), \quad (6.1) $$

where $q(u, v)$ and $s(u, v)$ are the quartile number and the sign of edge $(u, v)$, respectively, and $[]$ denotes string concatenation.

The second variant proposed in Section 5.4.2, SG2Vsb, creates 2 labels for each vertex, based on its positive and negative reachable sets. Our proposition for WD-SG2Vsb is the same as above, i.e. adding the quartile number in front of each positive and negative neighbor when building the labels. The update rules for the positive and negative labels are:
$$ \ell_t^+(u) = f(\ell_{t-1}^+(u), \{[q(u, v), \ell_{t-1}^+(v)] : v \in N^+(u)\}, \{[q(u, v), \ell_{t-1}^-(v)] : v \in N^-(u)\}) \quad (6.2) $$

$$ \ell_t^-(u) = f(\ell_{t-1}^-(u), \{[q(u, v), \ell_{t-1}^-(v)] : v \in N^+(u)\}, \{[q(u, v), \ell_{t-1}^+(v)] : v \in N^-(u)\}). \quad (6.3) $$

Once all the iterations are completed, f is applied to tuples formed by the positive and negative labels of each vertex, resulting in the final rooted subgraph labels.

### 6.1.3.2 Weighted Directed SGCN

SGCN is a graph embedding method that originally learns representations of vertices in signed graphs. In Section 5.4.3, we adapted it to handle whole graphs by adding master vertices to the graph and using their representations as the representation of the whole graph. To make this model able to use edge weights and directions, we propose a modification of its aggregation step. The original model consists of collecting features from neighboring vertices and aggregating them. In **WD-SGCN**, to obtain the representation of each vertex u, we first consider only their set of *ingoing* neighbors as they are the ones that have influence on the vertex. Then, to integrate the fact that all these neighbors have different degrees of importance depending on the weight of the edge shared with the vertex of interest, we perform a weighted aggregation of the neighbors. In other words, we weigh the features from neighboring vertices by the edge weight that we normalize to obtain a sum of 1. Then we aggregate them. This gives more importance to neighbors with a strong relationship in the resulting representation. We apply this procedure with all the 5 interconnection schemes that we proposed for SGCN.

### 6.1.3.3 Tuning Graph2vec

In addition to the standard Graph2vec [161] model presented in the previous section, we use a feature of that method to propose 3 Graph2vec variants specifically designed for our abuse detection task. Graph2vec allows labeling each vertex used in the rooted subgraph extraction process. In this way, 2 subgraphs containing vertices with identical labels are considered similar. This introduces an additional concept of similarity between vertices in addition to the graph structure. If no label is given, Graph2vec uses the vertex degree as the default label. We propose and assess the performances of 3 alternative labeling procedures that are designed for our task, and illustrated in Figure 6.3.

The first one, called **Graph2vec-author** (Figure 6.3.a), consists of using the unique identifier associated with the author of a post in our dataset. Therefore, we can identify if a same author is active in several conversations. Our intuition is that identifying the authors could allow us to consider context at a larger scale than a simple conversation. For instance, the fact that 2 authors
<figure><img src="image_15.png" /><figcaption>Figure 6.3: Examples of the 3 labeling procedures: Graph2vec-author (a), Graph2vec-distance (b), Graph2vec-target (c). The targeted vertex is represented in red.</figcaption></figure>

interact in multiple conversations and that there is a history between them could be an important
information.

With **Graph2vec-distance** (Figure 6.3.b), we compute the distance, between the vertex and the targeted vertex, and use it as its label. The purpose is to group different vertices (i.e. users) with the same label depending on their proximity with the author represented by the targeted vertex. The groups with low distances could correspond to the persons close to the author while the groups with high distances represent the users that only have brief contact with him. This strategy reduces the number of distinct labels in the graph and could make it easier to detect structural equivalence.

Finally, **Graph2vec-target** (Figure 6.3.c) is an extension of the previous strategy that further reduces the number of labels in the graph down to 2. With *Graph2vec-target*, we use a binary label that indicates whether the vertex is the targeted one or not. The intuition is that the targeted vertex plays a particular role in the graph, thus it could be important to differentiate it from others.

These labels generated by the 3 strategies are all based on the users and their proximity to the author of the targeted message. Therefore, they introduce information that is directly related to our task. For all these methods, we use representations of size 128.

## 6.2 Experiments

In this section, we present our experimental protocol (Section 6.2.1) and the performances obtained by the studied lexical and graph embedding methods (Section 6.2.2). Finally, we combine these embeddings (Section 6.2.3) as we did in Chapter 4 and summarize our findings (Section 6.2.4).
6.2.1 Experimental Protocol

The objective of the following experiments is to draw a comparison between embedding meth-
ods and feature-based approaches studied in Section 4. To this end, we use the same experimental
protocol as the one described in Section 4.3.1. We use the SpaceOrigin dataset and conduct the
experiments on the binary classification task consisting of detecting instances of *Abusive* and *Non
abusive* messages. We separate the experiments into 2 distinct aspects: textual and conversa-
tional.

The textual aspect relies on the content of the messages to perform the detection. We use the 5 lexical embedding models described previously in Section 6.1.1. As a baseline, we also include the feature-based method developed in Section 4. The 5 approaches generate embeddings at the word level. As our objective is to represent the message as a whole and not individual words, we use the standard technique of averaging the representations of all words in a message to obtain its global embedding [220]. Furthermore, the methods based on the BERT architecture, i.e. CamemBERT and FlauBERT, directly learn a global representation of the sequence through a particular token added at the very beginning of the sequence. We use this global representation as a second technique to represent the messages, in addition to the averaged representations.

The conversational aspect focuses on the use of contextual information. It relies on the con-
versational graphs extracted from the SpaceOrigin dataset. They represent the full conversations
(i.e. context) in which messages were posted. In the following experiments, we use the models
described in Section 6.1.2. We also include the feature-based baseline developed in Section 4.2.2
for reference. Additionally, we experiment with the proposed weighted directed variants of SG2V
and SGCN, and the 3 strategies proposed to tune Graph2vec for this specific task. Half of the
selected methods learn vertex embeddings and the other half focus on the whole graph. The latter
directly learns representations of the graph as a whole, we use these representations. For the
former, we use 2 strategies. First, we use only the representation of the vertex corresponding to
the author of the targeted message. Second, we average the representations of all vertices in the
graph into a single global one.

All these methods are used to learn representations of messages which are ultimately used as
input values of an SVM. As in Chapter 4.3, we perform a 10-fold cross-validation with 70%-train /
30%-test split. The performance is expressed in terms of macro F-measure.

6.2.2 Classification Results

In this section, we present the results of our experiments with lexical and graph embeddings
on the abuse detection task. We also include the content- and graph-based original methods
developed through feature engineering (Chapter 4), as a reference. In the following, we discuss
each aspect separately.
### 6.2.2.1 Lexical Embedding

Table 6.3 shows the results obtained by the baseline and the 5 lexical embedding methods that we consider. The first column denotes the scale of the generated representations: *Word* for the methods that generate representations of words that we average to obtain the representation of a message, and *Message* for BERT-based methods able to directly learn a global representation of the message. The third column shows the dimension of the learned representations. We keep the original dimension of all the pre-trained models used.

For most models, the most frequent classification error is to label a *non-abusive* message as *abusive*. This phenomenon represents approximately 60% of the total classification errors. Some non-abusive messages are quite ambiguous and therefore, difficult to classify.

<table><caption>Table 6.3: F-measures obtained by our baseline approach (based on feature engineering) and the 5 word embedding methods.</caption><thead><tr><th>Scale</th><th>Method</th><th>Dimension</th><th>F-measure</th><th>Total Runtime</th></tr></thead><tbody><tr><td>-</td><td>Baseline-Text</td><td>29</td><td>75.21</td><td>0:41</td></tr><tr><td rowspan="6">Word</td><td>Word2vec</td><td>300</td><td>71.14</td><td>04:14</td></tr><tr><td>fastText</td><td>300</td><td>76.01</td><td>04:29</td></tr><tr><td>Flair</td><td>2,048</td><td>77.33</td><td>05:25</td></tr><tr><td>CamemBERT-W</td><td>1,024</td><td><b>81.02</b></td><td>06:56</td></tr><tr><td>FlauBERT-W</td><td>1,024</td><td>78.46</td><td>07:11</td></tr><tr><td>Message</td><td>CamemBERT-M</td><td>1,024</td><td>80.96</td><td>04:59</td></tr><tr><td></td><td>FlauBERT-M</td><td>1,024</td><td>74.99</td><td>05:08</td></tr></tbody></table>

Our first observation is that all the word embedding approaches but one obtain a better F-measure than the baseline. Word2vec and fastText are among the least effective approaches for this classification task. This is not surprising since they are the 2 methods that produce fixed representations of words, i.e. a single representation for each word, independent of the context. Therefore, they are unable to distinguish homographs which can lead to confusion on the meaning of a message. The gap in terms of performance between these 2 methods can be explained by the fact that Word2vec is completely unable to treat words that were not seen during training (i.e. out-of-vocabulary words) while fastText, by construction, is able to deal with them. In the context of online messages, this feature is essential since spelling mistakes, rare and very specific terms, or even intentional obfuscation, are more than common.

The contextualized embeddings learned by Flair yield better performances, but they are lower than all other contextual methods. It seems that the character-level representation of Flair is not well suited for the abuse detection task, or at least, for our dataset.
FlaubERT-M, which learns a global representation of the message, gets lower performances
than its counterpart Flaubert-W, which creates the embedding by averaging the representations of
words composing the message. We assume that the abusive nature of a message is often based
on just a few words, and that operating directly on words allows improving their detection. However,
we do not find this situation with CamemBERT since its 2 variants obtain performances that are
not statistically different. CamemBERT-W, nonetheless, gets the best F-measure (81.02%) of all
the lexical embedding methods. This constitutes an improvement of almost 6 points compared to
our baseline.

These results highlight all the potential of word embedding methods. On this classification
task, they are able to learn representations that capture much more discriminative information
compared to the baseline based on a handcrafted set of morphological and language features
computed on the message. This directly leads to an improvement of up to 6 points in terms of
F-measure on the abuse detection task. Furthermore, these methods are not built following the
same concepts and objectives, therefore, we can suppose that they could be complementary. For
instance, Le et al. [213] show that the combination of CamemBERT and FlaubERT yields better
performances than the 2 methods used separately on a POS tagging task. We explore these
options in Section 6.2.3.

On the other hand, word embedding methods require much more time than the baseline to
obtain the representations of messages. This could be a limitation in a real-world application
where the time to classify a message is limited.

6.2.2.2 Graph Embedding

The results of our graph embedding methods and the baseline are shown in Table 6.4. The last
3 rows are the Graph2vec variants tuned to include additional information through vertex labels.
The first column indicates whether it is a *vertex* or *whole-graph* embedding method. With these
results, we want to answer 3 research questions.

* First, which scale is the most efficient for learning representations of conversational graphs (Section 6.2.2.2.1)?

* Second, which edge attributes (weights, signs or directions) should be considered in the representation learning process of graphs (Section 6.2.2.2.2)?

* Third, which vertex attribute brings more information to the representation (Section 6.2.2.2.3)?

6.2.2.2.1 Representation Scale

We evaluated 3 different strategies in Table 6.4: learning a representation of the whole graph,
using the representation of the targeted vertex, and averaging the representations of all vertices in
<table><caption>Table 6.4: F-measures of our baseline and graph embedding methods, including the 3 proposed variants of Graph2vec. In the top part, avg denotes a method averaging all vertices in the graph.</caption><thead><tr><th>Scale</th><th>Method</th><th>Dimension</th><th>F-measure</th><th>Total Runtime</th></tr></thead><tbody><tr><td>-</td><td>Baseline-Graph</td><td>477</td><td>83.40</td><td>03:51:25</td></tr><tr><td rowspan="10">Vertices</td><td>DeepWalk</td><td>128</td><td>73.72</td><td rowspan="2">14:29</td></tr><tr><td>DeepWalk-avg</td><td>128</td><td>73.54</td></tr><tr><td>Node2vec</td><td>128</td><td>74.59</td><td rowspan="2">15:01</td></tr><tr><td>Node2vec-avg</td><td>128</td><td>74.44</td></tr><tr><td>Walklets</td><td>128</td><td>74.84</td><td rowspan="2">16:21</td></tr><tr><td>Walklets-avg</td><td>128</td><td>74.21</td></tr><tr><td>BoostNE</td><td>136</td><td>71.53</td><td rowspan="2">16:00</td></tr><tr><td>BoostNE-avg</td><td>136</td><td>71.51</td></tr><tr><td>GraphWave</td><td>200</td><td>80.34</td><td rowspan="2">15:12</td></tr><tr><td>GraphWave-avg</td><td>200</td><td>79.58</td></tr><tr><td rowspan="2">k-hop GNN</td><td>k-hop GNN-avg</td><td>64</td><td>73.96</td><td rowspan="2">24:55</td></tr><tr><td></td><td></td><td>64</td><td>73.55</td></tr><tr><td rowspan="14">Whole graph</td><td>FGSD</td><td>200</td><td>71.13</td><td>06:35</td></tr><tr><td>Spectral Features</td><td>128</td><td>74.23</td><td>06:57</td></tr><tr><td>NGNN</td><td>64</td><td>74.51</td><td>16:58</td></tr><tr><td>SG2V</td><td>128</td><td>79.13</td><td>06:24</td></tr><tr><td>WD-SG2V</td><td>128</td><td>79.31</td><td>06:53</td></tr><tr><td>SGCN</td><td>128</td><td>74.74</td><td>14:14</td></tr><tr><td>WD-SGCN</td><td>128</td><td>75.02</td><td>14:50</td></tr><tr><td>Graph2vec</td><td>128</td><td>78.17</td><td>04:54</td></tr><tr><td>Graph2vec-author</td><td>128</td><td>78.24</td><td>05:10</td></tr><tr><td>Graph2vec-target</td><td>128</td><td>78.05</td><td>05:08</td></tr><tr><td>Graph2vec-distance</td><td>128</td><td><strong>80.47</strong></td><td>05:04</td></tr></tbody></table>

the graph. For these last 2 strategies, the representation of the targeted vertex always gives better performances than averaging all the vertices' representations. We can assume that the targeted vertex is the most important for our classification task and considering the other vertices does not bring additional information, and even add noise to the learned representation.

Between vertex and whole-graph approaches, there is no clear distinction in terms of F-measure. Since the graph in its entirety represents the message and its associated conversation, one could assume that embedding the whole graph could allow capturing more information than a single ver-
tex embedding. However, it seems that these graphs can be well-characterized by focusing on the vertex of the author of the *targeted message*. This could mean that the relative position of this vertex in the graph is enough to characterize the whole conversation and that vertex-level embeddings are able to capture such information.

### 6.2.2.2.2 Edge Attributes

To assess the impact of edge attributes on the learned representations, we compare the performances of SG2V (Table 6.5) and SGCN (Table 6.6) on all 8 possible combinations of edge weights, directions, and signs.

<table><caption>Table 6.5: Comparison of the *F*-measure obtained when considering the 8 different types of graphs with SG2V.</caption><thead><tr><th>Graph type</th><th></th><th>Unsigned</th><th>Signed</th></tr></thead><tbody><tr><td rowspan="2">Unweighted</td><td>Undirected</td><td>79.13</td><td>79.21</td></tr><tr><td>Directed</td><td>79.15</td><td>79.24</td></tr><tr><td rowspan="2">Weighted</td><td>Undirected</td><td>79.21</td><td>79.29</td></tr><tr><td>Directed</td><td>79.23</td><td><strong>79.31</strong></td></tr></tbody></table>

Table 6.6 shows the results for SG2V. We can observe that each attribute brings a bit of information, leading to a better representation. The best performance is obtained when considering edge weights, directions, and signs.

<table><caption>Table 6.6: Comparison of the *F*-measures obtained when considering the 8 different types of graphs with SGCN.</caption><thead><tr><th>Graph type</th><th></th><th>Unsigned</th><th>Signed</th></tr></thead><tbody><tr><td rowspan="2">Unweighted</td><td>Undirected</td><td>74.74</td><td>74.85</td></tr><tr><td>Directed</td><td>74.76</td><td>74.88</td></tr><tr><td rowspan="2">Weighted</td><td>Undirected</td><td>74.86</td><td>74.92</td></tr><tr><td>Directed</td><td>74.91</td><td><strong>75.02</strong></td></tr></tbody></table>

Table 6.6 shows the results for SGCN. The observation is similar to SG2V. All the attributes slightly improve the representation, and the best performance is obtained when considering all of them. These results confirm what we already showed in Chapter 4, that edge properties contain information that improves the quality of the representations.
6.2.2.2.3 Vertex Attributes

Regarding the Graph2vec variants that we propose, 2 of them obtain performances almost equivalent to the one of the original model. Identifying the author (Graph2vec-author) or the targeted message itself (Graph2vec-target) allows to create classes of vertices rather than treating vertices as individual components. However, this does not bring additional information useful for the detection of abusive messages. We suppose that the Graph2vec-target approach, which only distinguishes 2 classes of vertices, does not bring information discriminant enough to improve the detection performance. On the other hand, Graph2vec-author creates a distinct label for each author. Graph2vec is unable to benefit from this, probably because it is difficult to detect and capture structural equivalence in graphs with so many different vertex labels. Graph2vec-distance, the method that uses the distance between each vertex and the targeted one, obtains better results that make it the best-performing graph embedding method with a *F*-measure of 80.47. This distance is closely related to our task since it can categorize users based on their proximity to the targeted author (e.g. close friend, acquaintance, stranger). This proximity can be a key contextual element to detect an abuse. This shows that adding a task-specific information during the learning of the embeddings can improve their quality.

6.2.2.2.4 General Observations

Unlike what we observed on the textual embeddings, no graph embedding method is able to outperform the baseline. This is not surprising, because this set of features was specifically designed for this task and dataset and includes a large number of various measures, whereas the embedding methods are somewhat generic. The text baseline also includes much less features than the graph baseline, which probably made it easier to beat.

GraphWave (vertex scale approach), Graph2vec, and its signed variant SG2V (whole-graph scale) yield the best performances among graph embedding approaches. The other methods, obtain correct performances with an *F*-measure approximately ranging from 71% to 78%. Spectral Features and FGSD, which operate on the whole graph, might be penalized by the small size of our dataset and by the fact that graphs have approximately the same size and thus, possibly similar structures. DeepWalk is less efficient than Node2vec, which is in line with other studies [85], [170]. The Walklets algorithm learns multi-scale relationships in the graph. However, such relationships might not be very developed in our graphs, which could explain its lower performance. This observation could also be the reason for the very poor performance of BoostNE, which also operates on different granularity levels. Signed Graph2vec (SG2V) is one point above Graph2vec, which confirms once again that signs are important for the representation.

In the end, the best graph embedding method (Graph2vec-distance) is less efficient than our
baseline by approximately 3 points. However, embedding-based approaches still have 2 major
advantages compared to the feature-based baseline. First, they are not specifically designed for
this task or dataset and are hence more likely to be efficient in other settings. For instance, the
textual embedding models that we use are not specifically pre-trained on abusive datasets, which
illustrates this ability to be applied in various contexts. Second, embedding methods are more
scalable than hand-crafted sets of features. Computing the topological measures used in our
baseline is computationally very expensive, with a total runtime of almost 4 hours. On the other
hand, it only takes a few minutes to deal with the embedding methods on the same machine, which
makes them a lot more time-efficient.

6.2.3 Fusion of Embeddings

Past work in the literature [14] and the study conducted in Chapter 4.2.3 show that combining multiple types of information can be effective in the context of abusive message detection. In this chapter, we propose to combine some of the lexical and graph embedding methods that we used in the previous experiments. Our intuition is that these methods are based on completely different modalities, namely the text and the conversational graphs, and that their embeddings must therefore capture different information, possibly complementary. In this case, their combination should improve the classification performance. Furthermore, we also consider the combination of methods that operate on the same modality. As mentioned by Le et al. [213], the combination of multiple methods can lead to improved performance.

We use the 3 fusion strategies that we previously proposed in Chapter 4.2.3 to combine content-
and graph-based features.

In this chapter, we focus on the embedding methods which obtained the best performances in
our experiments. For the textual embeddings, we select Camembert-W, FlauBERT-W (word level),
and CamemBERT-M (message level). For the graphs, we use GraphWave (vertex-scale approach)
and Graph2vec-distance (whole-graph scale approach). We also consider the baselines for both
text and graph-based methods. Table 6.7 shows the F-measures obtained when applying the 3
fusion strategies to the best-performing methods.

Our first observation is that all 3 fusion strategies yield close fairly similar results. Generally
speaking, the hybrid fusion achieves top performance, ahead of the early fusion and the late fusion.
This is an expected result since the hybrid fusion is a combination of the other 2. All the information
available in the early fusion process is summarized in just 2 scores in the late fusion. One could
think that this is not enough to preserve all the information, but despite this process, the late fusion
performs only slightly worse than the early fusion.

As we previously showed in Chapter 4, the hybrid fusion of the 2 baselines improves the performance by more than 2 points compared to the graph baseline used alone. The fusion of the 2 CamemBERT approaches (CamemBERT-W and CamemBERT-M) does not bring a significant performance boost. Therefore, we can suppose that they both capture similar information. The same
<table><caption>Table 6.7: F-measures obtained by the fusion of the best performing embedding methods following the 3 fusion strategies. The second column indicates the performance of the best method among the 2 combined ones. The table is split into 4 parts depending on the nature of the methods combined. The first part is for the baselines, the second for the fusion of lexical embeddings, the third for the fusion between graph embeddings, and the fourth for the fusion of text and graphs.</caption><thead><tr><th rowspan="2">Methods</th><th rowspan="2">Base Perf.</th><th colspan="2">Early</th><th colspan="2">Late</th><th colspan="2">Hybrid</th></tr><tr><th>Dim.</th><th>F-m.</th><th>Dim.</th><th>F-m.</th><th>Dim.</th><th>F-m.</th></tr></thead><tbody><tr><td>Graph Base. + Text Base.</td><td>83.40</td><td>506</td><td>84.51</td><td>2</td><td>84.17</td><td>508</td><td>84.97</td></tr><tr><td>CamemBERT-W + CamemBERT-M</td><td>81.02</td><td>2,048</td><td>81.32</td><td>2</td><td>80.21</td><td>2,050</td><td>81.17</td></tr><tr><td>CamemBERT-W + FlauBERT-W</td><td>81.02</td><td>2,048</td><td>79.85</td><td>2</td><td>80.10</td><td>2,050</td><td>80.34</td></tr><tr><td>Graph2vec-dist. + GraphWave</td><td>80.47</td><td>256</td><td>82.45</td><td>2</td><td>82.07</td><td>258</td><td>81.19</td></tr><tr><td>Graph2vec-dist. + CamemBERT-W</td><td>81.02</td><td>1,152</td><td>85.66</td><td>2</td><td>85.10</td><td>1,154</td><td>86.07</td></tr><tr><td>GraphWave + CamemBERT-W</td><td>81.02</td><td>1,152</td><td>85.99</td><td>2</td><td>85.80</td><td>1,154</td><td>86.11</td></tr><tr><td>Graph Base. + CamemBERT-W</td><td>83.40</td><td>1,501</td><td>86.77</td><td>2</td><td>86.21</td><td>1,503</td><td><strong>87.06</strong></td></tr><tr><td>Graph2vec-dist + Text Base</td><td>80.47</td><td>157</td><td>82.24</td><td>2</td><td>82.21</td><td>159</td><td>82.36</td></tr></tbody></table>

applies to the fusion of CamemBERT and FlauBERT, which stays on par with the performance of CamemBERT used on its own. These lexical embeddings thus capture redundant information, which can be explained by the fact that they are both based on the same RoBERTa architecture.

For the graph-based methods, the early fusion of GraphWave and Graph2vec-distance (82.45%) reaches a performance only one point lower than the graph baseline (83.40%). On the one hand, this result acknowledges the assumption that graph embedding methods that operate on different granularity levels (vertices for GraphWave, whole-graph for Graph2vec-distance), can capture complementary information. On the other hand, the late and hybrid fusions perform worse. Thus, we hypothesize that all the complementary information captured by these 2 methods cannot be summarized in only 2 scores.

Finally, the combination of the best text-based method (CamemBERT-W) with each approach based on graphs always achieves better performance than the combination of the 2 baseline methods (84.97%). The hybrid fusions of CamemBERT-W with Graph2vec-distance and GraphWave obtain a F-measure of 86.07% and 86.11% respectively. The overall best performance is achieved by combining CamemBERT-W with the graph baseline (87.06%). Interestingly, this combination is only one point above both previous combinations, while when used on their own, Graph2vec-distance and GraphWave are 5 points behind the baseline. The best combination method is 7 points above the best embedding method used alone.
6.2.4 Results Summary

To summarize our results in this chapter, Table 6.8 shows the best performance we obtain for each type of approach studied. For lexical embeddings, the best method is CamemBERT applied to the words (CamemBERT-W) and averaged over the set of words composing a message. For graphs, this is the whole-graph embedding method Graph2vec in its variant using the distance as a vertex label (Graph2vec-distance). It is worth noting that Graph2vec is still less effective than our graph-based baseline. For fusion between text-based methods, this is the hybrid combination of CamemBERT methods applied on word and message level that outputs the best *F*-measure. The improvement is, however, very small compared to the methods used on their own. The early fusion between GraphWave (vertex-level) and Graph2vec (whole-graph level) is the best combination of graph embedding methods. Finally, the overall best performance is achieved by the fusion of text and graphs. The hybrid combination of CamemBERT with the graph-based baseline achieves a 87.06% *F*-measure, with a 86.21% precision and 87.93% recall. This result shows that, for the task of abuse detection, different modalities contain different types of information and that combining them brings real benefits compared to using them separately.

<table><caption>Table 6.8: Summary of the best-performing method in terms of *F*-measure for each type of approach studied in this chapter.</caption><thead><tr><th>Type</th><th>Method(s)</th><th>Dim.</th><th>F-m.</th></tr></thead><tbody><tr><td rowspan="2">Baseline</td><td>Baseline-Graph</td><td>477</td><td>83.40</td></tr><tr><td>Baseline-Text</td><td>29</td><td>75.21</td></tr><tr><td>Lexical embedding</td><td>CamemBERT-W</td><td>1,024</td><td>81.02</td></tr><tr><td>Graph embedding</td><td>Graph2vec-distance</td><td>128</td><td>80.47</td></tr><tr><td>Fusion text-text</td><td>CamemBERT-W + CamemBERT-M</td><td>2,048</td><td>81.32</td></tr><tr><td>Fusion graph-graph</td><td>Graph2vec-dist + GraphWave</td><td>256</td><td>82.45</td></tr><tr><td>Fusion graph-text</td><td>Baseline-Graph + CamemBERT-W</td><td>1,501</td><td><b>87.06</b></td></tr></tbody></table>

6.3 Feature Study

In Chapter 4, we identified, among the measures used in the baselines, the most discriminative features for our classification task, which we called *Best Features*. We were able to define subsets of 4 features for the text-based approach and of 10 features for the graph-based method which were sufficient to reach 97% of the original performance when considering the complete feature set. Since embedding methods provide vector representations in which dimensions are not directly interpretable, it does not make sense to apply the same method here. Instead, we propose to study
whether the best features from our baselines are well captured by the embedding methods used in
this chapter. We perform this analysis for both lexical and graph embedding approaches.

To this end, We compare the *F*-measure score obtained by each embedding method on its own,
with the score obtained by using a representation composed of the same embedding completed by
one of the best features. If the performance significantly increases, we conclude that the topological
measure was not captured by the embedding. These cases are represented in red in Figures 6.4
and 6.5. If the performance stays the same or increases by less than 0.50 point with the additional
feature, we conclude that the structural property corresponding to this topological measure is well
captured by the embedding (represented in green). If the performance increase is higher than
0.50 point but not statistically significant, we conclude that the property is only partially captured
by the embedding method (represented in orange). If the performance increase is higher than 0.50
point and statistically significant, we conclude that the property is not captured by the embedding
method (represented in red). We first study text-related features (Section 6.3.1), then graph-related
measures (Section 6.3.2).

6.3.1 Text Features

Results for text best-features addition are shown in Figure 6.4. The ratio of capital letters in the message seems to be well captured by all the embedding methods. The same observation applies to the TF-IDF score computed over the *Abuse* class except for Word2vec, which only partially captures this measure. In contrast, no method completely captures the information conveyed by the *Naive Bayes* score. Word2vec is even completely unable to capture it. This feature comes from a fully-fledged classifier and is, by far, the most important one from the content-based baseline. Therefore, we can suppose that this information is too complex to be modeled by our embedding methods. Furthermore, it integrates several types of information which might be difficult for an embedding to capture. However, almost all these methods yield a better *F*-measure than the baseline. Therefore, we can suppose that these methods might be able to capture other properties of the message that are not represented by any text-related feature, but improve the overall performance when all combined through the embedding. Another interesting observation of this study is that we can understand why Word2vec performs worse than the baseline and is the least efficient embedding method, as it fails to capture the 2 most important features of that baseline.

6.3.2 Graph Features

An interesting result shown by Figure 6.5 is that some topological measures seem to be well
captured by almost all the embedding methods (e.g. Authority score at graph level, PageRank
centrality, Degree centrality, Vertex count, and Reciprocity). Contrariwise, the Coreness score
at graph level on the Full and Before graphs are partially captured by GraphWave and Spectral
<figure><img src="image_16.png" /><figcaption>Figure 6.4: Text measures captured (green), partially captured (orange), or not captured (red) by the word embedding approaches. Each value is the difference between the *F*-measure score obtained by the embedding method on its own and the score obtained by the embedding method completed by the corresponding Best Feature.</figcaption></figure>

Features, and not captured at all by all other methods. The Closeness Centrality at the graph level and the vertex level on the Before and After graphs are well captured by some methods and partially captured by others. BoostNE fails to capture the authority at the graph level, which might explain why this method obtains the worst results among vertex embeddings. SG2V and Graph2vec, 2 whole-graph embedding methods, are the only ones to correctly capture all the features except for the 2 mentioned above. There is no surprise that they are 2 of the best-performing methods, together with GraphWave. The latter, considered as a vertex embedding method, is the only approach to capture, at least partially, all the best features. However, these 3 methods obtain lower *F*-measure scores than the baseline. We assume that they might not capture other properties of the graph which are less important but improve the performance when all combined. A possibility to combine text and graphs is to directly learn representations from these 2 aspects. We developed an experimental work in this direction (see Appendix A) based on the architecture of the textual embedding Flair [216]. It seeks to learn combined representations of text and graphs to improve the quality of the representations.

An interesting result of this study is that there is no clear difference in the type of information captured by vertex and whole-graph embedding approaches. Vertex embedding methods can capture certain graph-scale topological measures, and whole-graph embedding methods can capture some vertex-scale measures. This property may result from the relatively small size of our graphs, as the second-order neighborhood of a vertex might include the majority of the graph. Thus, differences between vertex and whole-graph embedding methods are not as important as they could be on larger graphs. Furthermore, our graphs are built around a specific vertex. This specificity might help the whole graph embeddings to capture better vertex-level information.
<figure><img src="image_17.png" /><figcaption>Figure 6.5: Topological measures captured (green), partially captured (orange), or not captured (red) by the embedding approaches. The first 4 topological measures are computed at the *Graph* level and the last 5 topological measures are computed at the *Vertex* level. Each value is the difference between the *F*-measure score obtained by the embedding method on its own and the score obtained by the embedding method completed by the corresponding Best Feature.</figcaption></figure>

## 6.4 Conclusion

In this chapter, we have experimented with embedding methods to automate the representation learning process as part of the automatic abuse detection task. Our experiments cover the textual and contextual aspects of that task. We first compared 5 lexical embedding methods to learn representations of messages in French. The contextualized embeddings were much more efficient than the fixed ones. The best approach, CamembERT, achieved up to 81.02% *F*-measure. This is a massive improvement over our baseline based on a manually crafted set of features. We then compared 11 graph embedding methods, half of them generating embeddings at the vertex scale, and the other half treating the whole graph. We proposed variants of the SG2V and SGCN models able to treat weighted, directed graphs. Both variants obtain better results than the original model
showing that edge attributes convey important information for this task. In addition, we explored a functionality offered by Graph2vec, one of these methods. We consider 3 vertex labeling strategies allowing us to integrate various information related to our task directly into the learning process. The variant using the distance between a vertex and the targeted vertex as a label was the best graph embedding method with a *F*-measure of 80.47%. Finally, we experimented with the 3 fusion strategies from Chapter 4.2.3 to combine different types of embeddings. We could show that the combination of textual and graph embeddings takes advantage of both sources of information and significantly improves the detection of abusive messages, as already observed with the features in Chapter 4.

We think that these results are very promising in multiple aspects. The textual embeddings are much more efficient than the feature engineering approaches on the abuse detection task. The embeddings are much more robust and thus, less likely to be negatively impacted by spelling mistakes and other special language features that are common in online conversations. Graph embeddings, though less efficient than the feature-engineered baseline, still demonstrate great results. This is particularly interesting since these methods are completely task-independent, much more scalable, and time-efficient.

We believe that our work opens new directions for future research. In this work, we combine lexical and graph embeddings which are learned separately. An idea could be to combine these aspects directly during the representation learning process to obtain a single embedding including all these types of information. As the messages are posted in a specific order, we could also think of an approach including a temporal aspect, for instance with a dynamic conversational graph reflecting the evolution of conversation over time.
# CONCLUSION AND PERSPECTIVES

## 7.1 Conclusion

In this thesis, we tackled the problem of combining 2 modalities, text and graphs, representing conversation content and structure, for the analysis of text-based documents. We applied it to the detection of abusive messages in online conversations. We observed that the literature was largely focused on text-based approaches because this seems to be the most straightforward way of detecting an abuse case. While effective for flagrant instances of abuse, this approach shows its limitations when it comes to detecting more intricate occurrences such as innuendos and allusions. The conversation in which messages are posted represents important contextual information that can greatly improve the detection in such situations. As is the case in many other domains, this multimodal approach combining the text and the graphs provides even better detection performances. As they each focus on different aspects of the conversation, they are able to capture different but complementary information.

To compare the content and structural aspects, we divided our studies into 2 parts. The first focuses on the textual content of the message itself, whereas the second deals with conversational graphs built to represent the conversation in which the message was posted. The key part of an abuse detection system is the way it represents messages and conversations. We first used a methodology based on feature engineering. We used a set of measures that characterize the messages and the graphs. For the content-based method, we used a mix of NLP features including morphological and topological ones. For the graph-based method, we used a wide range of topological measures defined for different scales and scopes. The representations of the messages are thus directly related to the nature of the information considered by these 2 methods. By applying them to a dataset of real-world conversations extracted from a video game, and feeding the representations to a classifier, we showed that the graph-based method obtained better results than those of the content-based method. This is a particularly important result since this aspect of the literature is largely overlooked. We then proposed a method seeking to take advantage of both the text and the graphs by combining them. Through several variants, we could confirm that this combination is beneficial to the detection of abusive messages. We were also able to show that edge attributes (i.e. weight, direction, sign) bring more discriminative power to the graph-based method. Furthermore, while it was originally designed to do the classification afterward, this
method could be adapted to a live setting as the performance is still convincing when considering
only the messages posted prior to the targeted message. The main limitation of this feature-based
framework is its important computational cost due to the large number of features, and the algo-
rithmic complexity to compute some of them. This motivated us to determine subsets of the most
important features for each method. They drastically reduce the processing time of the methods
while preserving a high performance.

Another solution to obtain representations of messages and graphs is to automatically dis-
cover them through representation learning. More precisely, embedding techniques present sev-
eral advantages over feature engineering, including being more flexible and time-efficient. As our
graph-based approach for abuse detection roughly corresponds to a graph classification task, we
proposed to use whole-graph embedding, the specific type of embedding methods dedicated to
learning representations of the whole graph at once. Motivated by the fact that edge directions and
weights improve the classification, we proposed to consider a third sort of edge attribute in the form
of signs. As there is no signed whole-graph embedding method in the literature, we developed our
own 2 methods. Signed Graph2vec (SG2V), a generalization of the unsigned whole-graph em-
bedding method Graph2vec to signed whole graphs, and a generalization of the signed vertex
embedding method SGCN to signed whole graphs. There is no benchmark to evaluate such meth-
ods, so we compiled our own benchmark composed of 3 datasets of annotated signed graphs.
It contains real vote networks, conversation networks, and artificially generated graphs. We em-
pirically showed that our 2 proposed methods obtain better classification results than standard
unsigned whole-graph and signed vertex-oriented methods.

To complete our comparison between text and graphs for the analysis of text-based documents, we extended our previous framework to replace feature-based methods with automatic embedding approaches. We compared several approaches from the literature. For the aspect focused on the content, we selected methods operating on different granularities of text. For the graph-based system, the selected methods include our 2 signed ones, also operating on different scales. We showed that, for the content, textual embeddings were much more effective than the feature-based system. This observation did not stand for the graphs, as the graph embeddings are slightly less effective. However, they still have numerous advantages in terms of computational cost and adaptability to different environments. We concluded the comparison by combining the text and graph aspects and have once again found that they are complementary, as their joint use improved the classification performance on top of the gain already brought by text representation learning. Embedding methods provide vector representations in which dimensions are not directly interpretable. To overcome this issue and provide a way to interpret our results, we proposed a framework to detect which properties are captured by the embeddings. It consists of adding a dimension to the representation vector containing a single one of the previously selected features. We then compared the performance obtained with and without this additional dimension. If the performance increased
as the feature is used, we concluded that the property associated with it was not properly captured by the considered embedding method. Otherwise, the information brought by the feature is redundant and does not impact the performance. This experiment showed that all the textual embedding methods were able to capture or partially capture the important text-based features, which is in line with the fact that textual embeddings are more effective than the feature-based system. On the other hand, most of the graph embedding methods are unable to capture variants of an important feature (i.e. the coreness centrality) which might be the reason why they are less effective than the feature-based system.

## 7.2 Perspectives

The different studies conducted in this thesis suggest diverse opportunities that remain open to extend this work. We already discussed some of them in general terms at the end of each chapter. In the following, we discuss them more precisely.

**Task reformulation:** In the literature, abuse detection is always formulated as a classification problem. Be it binary or multi-class, such a framework usually suffers from an imbalanced distribution of classes. In real-world online discussions, the prevalence of abusive messages is very low, inferior to 1% in general [90], [91]. That is a reason why most authors oversample abusive occurrences when creating datasets, by using comments from specific communities and topics with a higher probability of abuse. Hence, one could argue that these datasets are not representative of reality. As abuse occurrences are relatively rare, one could consider them as anomalies that deviate from the majority of the data. For this reason, an interesting research line would be to reformulate the abuse detection problem as an anomaly detection task, instead of a classification task, where the goal is to identify abusive comments (anomalies) that appear inconsistent with the rest of the conversation. By construction, anomaly detection methods are suitable for highly imbalanced datasets.

**Conversation datasets:** The abuse detection literature mainly focuses on content and overlooks structure, even though it is highly efficient, as we showed. One reason for this trend is the lack of experimental datasets providing suitable data. Indeed, annotated datasets of conversations are rare, and for the few that exist, the conversations are often too short to exploit the full potential of structure-based approaches. We constituted a conversation dataset [18] to mitigate this issue (see Appendix B.2), but it is also limited to rather short conversations. An interesting work would be to extend this effort to create a large-scale dataset of lengthy conversations annotated for abuse. This would open opportunities for future research based on the structure of conversations. An idea could be to use recent text generation models to create large conversations. However, such models are usually tweaked to avoid generating abusive comments. Another interesting point could be to work on a multilingual corpus, to assess the possibilities of transfer learning from one language
to another.

**Temporal aspect:** When working on the structure, we used static graphs to represent con-
versations. However, conversations evolve through time. An interesting research line would be to
directly integrate the temporal aspect of conversations, either by using representation techniques
able to simultaneously embed structural and temporal information [221] or by extracting a dynamic
graph (i.e., a sequence of graphs to represent the evolution of conversations over time). We started
to experiment with different types of context around messages based on temporality: the context
posted before, after, and the combination of both. However, this is still ongoing research, and we
believe that integrating the temporal aspect would improve the classification performance.

**Combining text and graphs:** Finally, in this thesis, we showed that it is efficient to combine
the text and graphs to detect abuse. However, we performed this combination only after the rep-
resentation step, i.e. we combined the text and graph representations learned by our different
systems. A possible improvement could be to simultaneously learn from both in order to obtain a
global representation of a message including text and graph information. We experimented with
this idea by extending the Flair [216] framework, a textual embedding framework, to include text
and graphs. We briefly present this preliminary work in Appendix A. However, the preliminary re-
sults and difficulties with this approach prevented us from developing it further, and it remains an
open work.
# Appendices
# TEXT AND GRAPH JOINT EMBEDDING

This appendix presents a preliminary work of an embedding method designed to jointly learn from text and graphs. It is based on the Flair embedding architecture [216], a textual embedding model. The preliminary results with this approach were negative, but the methodology can still be interesting so we present it. We first explain this original architecture (Section A.1), then we explain how we propose to transform this model to include both graphs and text (Section A.2).

## A.1 Flair Embedding

Flair [216] is a textual embedding model that learns contextualized representations at the character level. An input sentence is treated as a character sequence and input into a bi-directional LSTM operating on characters. Working at the character level has two major advantages over word-level models: first, it is particularly efficient when dealing with rare or misspelled words and second, it is able to better model language-related concepts such as prefixes and suffixes. The model is trained to predict the next character for each element in the sequence to process. As it uses a bi-directional LSTM, the Flair model learns 2 hidden representations for each character of the sequence: one for the forward network and one for the backward network. The contextual embedding of a word in a sentence is obtained by concatenating the forward representations of the characters located before the end of the word and the backward representations of the characters located after the beginning of the word. Figure A.1 shows an example of this process.

## A.2 Proposed Architecture

Our objective is to develop a model able to jointly learn from graphs and text. To this end, we propose a two-stepped architecture that we call Flair². The first step is similar to the standard Flair model, and is used to learn representations of words based on their character-level decomposition. The outputs of this first step are thus word-level representations. We introduce the graph representations in the second step of our proposed model. These representations can be learned from any graph embedding model presented in Section 6.1.2. The second step of Flair² uses a similar architecture as the first one, but its inputs are the word-level representations from the first step and the representation of the graph representing the corresponding conversation. Our objective with
<figure><img src="image_18.png" /><figcaption>Figure A.1: Extraction of the embedding for word "Washington" in the sentence "George Washington was born". The forward model is shown in red and the backward model in blue. Figure from [216].</figcaption></figure>

this second part of our model is two-fold. First, as a message is a sequence of words, in Chapter 6 we proposed to average the representations of all the words in a sentence to obtain its representation. Here, we think that this step could automate this aggregation process and directly learn a better manner to combine the words’ representations to obtain a representation of a message. Second, as we input both text and graph representations to the model, we think that the resulting representations could integrate information from both modality.

Combining graph and text right from the learning phase, instead of treating them separately and combining their learned representations could help better capture the complementary information of both modality.
# DATASETS

As mentioned in Chapter 2, we use the SpaceOrigin dataset throughout this thesis. We tried to experiment with some others, but we always faced limitations related to the specific needs for our task. This includes the fact that our methods require full conversations with a certain diversity in the messages and authors. In this appendix, we detail the two datasets that we tried to experiment with. First, we present Reddit (Section B.1), a dataset of conversations from the platform Reddit<fnref n="1" />, which is annotated for abuse, and explain its limitations relatively to our task. Then, we present Wikipedia Abusive Conversations (WAC) (Section B.2), a corpus that we constituted to overcome the limitations of the other existing datasets.

## B.1 Reddit

Reddit [102] is a dataset of 6,000 English language Reddit comments annotated for offensive-ness. Reddit is a social news aggregation and discussion website where users can create a *post* to share various types of content such as links, text posts, images, and videos. Users can also *comment* on an existing post and reply to a comment to start a discussion. A *post* can thus be viewed as the root of a discussion (called *thread*) divided into multiple sub-discussions.

Reddit is divided into forums called *subreddits*, each dedicated to specific topics. To constitute the Reddit dataset, Hada *et al.* sampled 808 posts from particular subreddits. This selection was made to increase the proportion of offensive comments. 50% of the selected posts are from a selection of subreddits covering diverse generic themes, 25% are from *ChangeMyView*, a reddit with posts on controversial topics and the final 25% are from random subreddits. The authors extracted the first and last 25 comments from these 808 posts and sampled 6,000 comments from this set to constitute the Reddit dataset. All these comments have then been annotated with a real-valued score between -1 (maximally supportive) and 1 (maximally offensive). The annotation was done using crowdsourcing.

The Reddit dataset thus has quality annotations for offensiveness and provides comments that are not completely independent from each others as some of them were posted in the same *thread*. However, a number of limitations prevented us from using this dataset in our experiments. We

<footnote n="1">¹https://www.reddit.com/</footnote>
describe them in the following section.

B.1.1 Limitations of Reddit

**Conversation structure:** The first limitation of Reddit is inherent to Reddit. Indeed, each *comment* on a *post* is considered as the strat of a new conversation, and each conversation can then be divided into multiple sub-conversations. A general observation is that posts with a lot of comments, often tend to generate many sub-conversations, with only very few different users interacting in each of them. This phenomenon results in conversations that are relatively short and with only a limited number of participants, which is the opposite of what our proposed methods require. Furthermore, we could observe that comments posted under the same *post* but in different sub-conversations are often completely unrelated. This is due to the fact that while the original post is the same, conversations tend to quickly deviate into other subjects.

**Missing Data:** Another limitation of this dataset is that it does not provide full conversations. Indeed, Reddit contains multiple annotated comments from the same conversation but there are missing comments between them. As the authors provide the *id* of the reddit posts, our idea was to retrieve all the missing comments to obtain full conversations. However, this showed poor results as a lot of the original comments were missing either because they were deleted by their author or because they were removed by moderators. The same problem applied with authors whose account had been deleted, making it impossible for us to correctly associate comments with their author.

The aforementioned limitations prevented us from using Reddit in our experiments and inspired us to constitute our own dataset of annotated conversations. We discuss this dataset in the next section.

B.2 Wikipedia Abusive Conversations

The Wikipedia Abusive Conversations (WAC) [18] is a large dataset of Wikipedia comments that we constituted. The particularity of this dataset is that each annotated comment is provided along with the conversation in which it was posted. Our objective with this work was to encourage further development of context- and thread-based methods in the area of abusive content detection by providing the first large scale corpus of contextualized abusive messages. WAC contains roughly 193,000 conversations and 383,000 messages annotated as being abusive or not. In Section B.2.1, we give details on the dataset and how we constructed it. In Section B.2.2, we explains the limitations of this dataset that prevented us from using it in our experiments.
## B.2.1 Proposed Corpus

The conversations in WAC come from Wikipedia talk pages, i.e. the web pages associated to Wikipedia articles where editors can exchange. Typically, editors write a message explaining which change they made on the article and why. Every Wikipedia article and user has a related talk page. To constitute WAC, we aggregated data from 2 existing datasets.

### B.2.1.1 Wikipedia Comment Corpus

The Wikipedia Comment Corpus (WCC) [90] is a corpus of discussion comments from English Wikipedia talk pages which are extracted using the revision history of each considered talk page. In January 2016, a dump of more than 63M Wikipedia talk page comments was made public. From this dump, Wulczyn et al. sampled 3 smaller datasets that were annotated by humans for different type of abuse:

* **personal attack**: abusive content directed at somebody's person rather than providing evidence;

* **aggression**: malicious remark to a person or group on characteristics such as religion, nationality or gender;

* **toxicity**: comment that can make other people want to leave the conversation.

The *Personal attack* and *Aggression* datasets contain exactly the same 115k comments while the Toxicity dataset contains more comments (159k). Among them, 77k appear in all 3 datasets. Table B.1 shows the prevalence of abusive comments in each dataset.

<table><caption>Table B.1: Main properties of the three datasets constituting the Wikipedia Comment Corpus (WCC).</caption><thead><tr><td>Dataset</td><td>Comments</td><td>Percentage abusive</td></tr></thead><tbody><tr><td>Personal attack</td><td>115,864</td><td>13.4 %</td></tr><tr><td>Aggression</td><td>115,864</td><td>14.7 %</td></tr><tr><td>Toxicity</td><td>159,686</td><td>11.5 %</td></tr></tbody></table>

This dataset thus provides high quality annotations for 383k messages. However, all these messages are unrelated and we cannot use conversation-based techniques directly on them. To obtain this conversation structure, we use a second dataset.
### B.2.1.2 WikiConv

WikiConv [104] is a large public corpus also based on Wikipedia talk pages. This corpus contains full conversations, and not only isolated comments like WCC. It was created by analyzing the history of Wikipedia talk pages. However, Wikipedia does not provide a standard post system for these pages such as those commonly used in online forums. Instead, the talk page is similar to a regular Wikipedia article page, or a wiki page in general: in theory, users have the ability to edit it by adding, modifying or removing text anywhere. However, in practice, a set of writing and formatting conventions<fnref n="2" /> allow giving structure to the various conversations taking place on the talk page. For instance, when a user adds his own post, he indents it so as to indicate its hierarchical level in the conversation tree. Figure B.1 shows an example of Wikipedia conversation under the form of the rendered talk page and the corresponding Wikicode (Wikipedia markup language).

<figure><img src="image_19.png" /><figcaption>Figure B.1: Part of the *Japan* Wikipedia article talk page: rendered page (left) and corresponding Wikicode (right).</figcaption></figure>

Therefore, Hua *et al.* defined heuristics to retrieve all the messages in a talk page. Each message is identified by an *id*. WikiConv thus provides large conversations but without high quality annotations. On objective in the next section is to combine the annotations from WCC with the conversations of WikiConv in order to obtain annotated messages along with the conversation in which it was posted.

### B.2.1.3 Combining WCC and WikiConv

Comments in WCC and WikiConv are identified by an *id* that, in most cases, can be used to link messages from both datasets. To create our WAC dataset, we retrieve the WikiConv's conversation associated with each comment in WCC. After this retrieval, all the annotated message are associated with their corresponding conversation. Note that multiple annotated messages can

<footnote n="2">²https://en.wikipedia.org/wiki/Help:Talk_pages#Replying_to_an_existing_thread</footnote>
occur in the same conversation. Because of missing data, some messages from the original WCC dataset can not be retrieved in WikiConv and are discarded. In total, the WAC corpus contains more than 2.2M unique messages split into 168,827 unique conversations. Among the 2.2M, 382,665 messages are annotated and the rest are messages that only appear in the conversation around an annotated one. The number of annotated messages and the division of annotations in all three datasets is summarized in Table B.2.

<table><caption>Table B.2: Number of annotated messages and distribution of annotations in the proposed Wikipedia Abusive Conversations (WAC) corpus.</caption><thead><tr><th>Dataset</th><th>Annotated messages</th><th>Abuse</th><th>Non-abuse</th></tr></thead><tbody><tr><td>Personal attack</td><td>113,174</td><td>14,934 (13.20%)</td><td>98,240 (86.80%)</td></tr><tr><td>Aggression</td><td>113,174</td><td>16,331 (14.43%)</td><td>96,843 (85.57%)</td></tr><tr><td>Toxicity</td><td>156,317</td><td>19,700 (12.60%)</td><td>136,617 (87.40%)</td></tr><tr><td>Total</td><td>382,665</td><td>50,965 (13.31%)</td><td>331,700 (86.69%)</td></tr></tbody></table>

The Wikipedia Abusive Conversations is thus a large scale dataset of messages annotated for 3 types of abuse and provided along with the conversation in which they were posted. However, we were unable to benefit from this dataset in our experiments because of the limitations detailed in the following section.

## B.2.2 Limitations of WAC

**Conversation Structure:** As for Reddit (Section B.1), WAC conversations are not structured as a linear sequence of messages. A conversation is usually composed of multiple sub-conversations with very few different users interacting in each of them. In the context of this thesis, his is an important limitation as our proposed methods benefit from having a diversity of users in the conversations.

**Length of conversations:** The conversations in WAC are also relatively short, with a majority of them constituted of fewer than 20 messages. Figure B.2 shows the distribution of the conversations length in the dataset. Note that the y-axis scale is logarithmic for readability reasons. We can observe that some conversations contain more than 1,000 messages but for a large majority, conversations are only 1- to 20-message long.
<figure><img src="image_20.png" /><figcaption>Figure B.2: Distribution of the conversation lengths in *Wikipedia abusive Conversations*, expressed in number of messages. The y-axis scale is logarithmic.</figcaption></figure>

For these reasons and after preliminary experiments that were inconclusive, we decided to not use the WAC dataset for our experiments.
REFERENCES

[1] M. O'Reilly, N. Dogra, N. Whiteman, J. Hughes, S. Eruyar, and P. Reilly, "Is social media bad for mental health and wellbeing? exploring the perspectives of adolescents," *Clinical Child Psychology and Psychiatry*, vol. 23, no. 4, pp. 601-613, 2018. DOI: 10.1177/1359104518775154 (cited on p. 11).

[2] S. Hinduja and J. W. Patchin, "Cyberbullying: Identification, prevention, and response," [Online]. Available: https://cyberbullying.org/Cyberbullying-Identification-Prevention-Response-2023.pdf (cited on p. 11).

[3] S. Menini, A. Palmero Aprosio, and S. Tonelli, "Abuse is contextual, what about nlp? the role of context in abusive language annotation and detection," in *ArXiv Preprint*, 2021. [Online]. Available: https://arxiv.org/pdf/2103.14916.pdf (cited on pp. 12, 22, 26, 27).

[4] H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran, "Deceiving google's perspective api built for detecting toxic comments," *CoRR*, vol. cs.LG, p. 1702.08138, 2017. [Online]. Available: https://arxiv.org/abs/1702.08138 (cited on pp. 12, 22).

[5] J. Cheng, C. Danescu-Niculescu-Mizil, and J. Leskovec, "Antisocial behavior in online discussion communities," in *9th International AAAI Conference on Web and Social Media*, 2015, pp. 61-70. [Online]. Available: https://cs.stanford.edu/people/jure/pubs/trolls-icwsm15.pdf (cited on pp. 12, 22, 23).

[6] E. Papegnies, V. Labatut, R. Dufour, and G. Linarès, "Conversational networks for automatic online moderation," *IEEE Trans. Comput. Social Systems*, vol. 6, no. 1, pp. 38-55, 2019. DOI: 10.1109/TCSS.2018.2887240 (cited on pp. 12, 24, 45-50, 59, 60, 88).

[7] A. Gandhi, K. Adhvaryu, S. Poria, E. Cambria, and A. Hussain, "Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions," *Information Fusion*, vol. 91, pp. 424-444, 2023. DOI: https://doi.org/10.1016/j.inffus.2022.09.025 (cited on p. 12).

[8] Z. Wu, C. Zheng, Y. Cai, J. Chen, H.-f. Leung, and Q. Li, "Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts," in *28th ACM International Conference on Multimedia*, 2020, pp. 1038-1046. DOI: 10.1145/3394171.3413650 (cited on p. 12).
[9] Z. Huang, X. Xu, J. Ni, H. Zhu, and C. Wang, "Multimodal representation learning for recommendation in internet of things," *IEEE Internet of Things Journal*, vol. 6, pp. 10675–10685, 2019. DOI: 10.1109/JIOT.2019.2940709 (cited on p. 12).

[10] A. H. Razavi, D. Inkpen, S. Uritsky, and S. Matwin, "Offensive language detection using multi-level classification," in *Canadian Conference on Artificial Intelligence*, 2010, pp. 16–27. DOI: 10.1007/978-3-642-13059-5_5 (cited on pp. 12, 20).

[11] E. Papegnies, V. Labatut, R. Dufour, and G. Linares, "Impact of content features for automatic online abuse detection," in *International Conference on Computational Linguistics and Intelligent Text Processing*, 2017, pp. 404–419. DOI: 10.1007/978-3-319-77116-8_30 (cited on pp. 12, 20, 27–29, 46, 52, 60).

[12] H. Watanabe, M. Bouazizi, and T. Ohtsuki, "Hate speech on twitter: A pragmatic approach to collect hateful and offensive expressions and perform hate speech detection," *IEEE Access*, vol. 6, pp. 13825–13835, 2018. DOI: 10.1109/ACCESS.2018.2806394 (cited on pp. 12, 20).

[13] R. Martins, M. Gomes, J. J. Almeida, P. Novais, and P. Henriques, "Hate speech classification in social media using emotional analysis," in *7th Brazilian Conference on Intelligent Systems*, 2018, pp. 61–66. DOI: 10.1109/BRACIS.2018.00019. (cited on pp. 12, 20).

[14] P. Mishra, M. Del Tredici, H. Yannakoudakis, and E. Shutova, "Author profiling for abuse detection," in *27th International Conference on Computational Linguistics*, 2018, pp. 1088–1098. [Online]. Available: https://www.aclweb.org/anthology/C18-1093 (cited on pp. 12, 24, 31, 45, 113).

[15] S. Nagar, S. Gupta, C. S. Bahushruth, F. A. Barbhuiya, and K. Dey, "Hate speech detection on social media using graph convolutional networks," in *International Conference on Complex Networks and Their Applications*, 2022, pp. 3–14. DOI: 10.1007/978-3-030-93413-2_1 (cited on pp. 12, 25).

[16] R. Ali, U. Farooq, U. Arshad, W. Shahzad, and M. O. Beg, "Hate speech detection on twitter using transfer learning," *Computer Speech & Language*, vol. 74, 2022. DOI: 10.1016/j.csl.2022.101365 (cited on pp. 12, 21).

[17] H. Saleh, A. Alhothali, and K. Moria, "Detection of hate speech using bert and hate speech word embedding with deep model," *Applied Artificial Intelligence*, vol. 37, 2023. DOI: 10.1080/08839514.2023.2166719 (cited on pp. 12, 21).

[18] N. Cécillon, V. Labatut, R. Dufour, and G. Linarès, "WAC: A corpus of wikipedia conversations for online abuse detection," in *12th International Conference on Language Resources and Evaluation*, 2020. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2020/pdf/2020.lrec-1.173.pdf (cited on pp. 14, 17, 25–28, 123, 130).
[19] —, "Graph embeddings for abusive language detection," *SN Computer Science*, vol. 2, no. 37, 2021. DOI: 10.1007/s42979-020-00413-7 (cited on pp. 14, 88, 97, 100).

[20] —, "Abusive language detection in online conversations by combining content- and graph-based features," *Frontiers in Big Data*, vol. 2, p. 8, 2019, ISSN: 2624-909X. DOI: 10.3389/fdata.2019.00008 (cited on pp. 14, 46).

[21] N. Cécillon, R. Dufour, and V. Labatut, "Approche multimodale par plongement de texte et de graphes pour la détection de messages abusifs," *Traitement Automatique des Langues*, vol. 62, no. 2, pp. 13–38, 2021. [Online]. Available: https://www.atala.org/sites/default/files/TAL_62_2_v2.pdf (cited on pp. 14, 97).

[22] N. Cécillon, R. Dufour, V. Labatut, and G. Linarès, "Tuning graph2vec with node labels for abuse detection in online conversations," in *11th MARAMI*, 2020. [Online]. Available: http://ceur-ws.org/Vol-2750/paper8.pdf (cited on pp. 14, 97).

[23] N. Cécillon, N. Arnk, V. Labatut, and R. Dufour, "Whole-graph representation learning for the classification of signed networks," *Submitted*, 2023 (cited on pp. 15, 70).

[24] C. Nobata, J. Tetreault, A. Thomas, Y. Mehdad, and Y. Chang, "Abusive language detection in online user content," in *25th International Conference on World Wide Web*, 2016, pp. 145–153. DOI: 10.1145/2872427.2883062 (cited on pp. 18, 21).

[25] P. Fortuna and S. Nunes, "A survey on automatic detection of hate speech in text," *ACM Computing Surveys*, vol. 51, pp. 1–30, 2018. DOI: 10.1145/3232676 (cited on p. 18).

[26] M. Dadvar, D. Trieschnigg, R. Ordelman, and F. de Jong, "Improving cyberbullying detection with user context," in *35th European Conference on IR Research*, vol. 7814, Mar. 2013. DOI: 10.1007/978-3-642-36973-5_62 (cited on pp. 18, 23).

[27] R. N. M. Mercado, H. F. C. Chuctaya, and E. G. C. Gutierrez, "Automatic cyberbullying detection in spanish-language social networks using sentiment analysis techniques," *International Journal of Advanced Computer Science and Applications*, vol. 9, no. 7, pp. 228–235, 2018. DOI: 10.14569/IJACSA.2018.090733 (cited on p. 18).

[28] O. Goga, G. Venkatadri, and K. P. Gummadi, "The doppelgänger bot attack: Exploring identity impersonation in online social networks," in *2015 Internet Measurement Conference*, 2015, pp. 141–153. DOI: 10.1145/2815675.2815699 (cited on p. 18).

[29] S. R. Sangwan and M. Bhatia, "Denigration bullying resolution using wolf search optimized online reputation rumour detection," *Procedia Computer Science*, vol. 173, pp. 305–314, 2020. DOI: 10.1016/j.procs.2020.06.036 (cited on p. 18).
[30] R. A. Monteiro, R. L. S. Santos, T. A. S. Pardo, T. A. de Almeida, E. E. S. Ruiz, and O. A. Vale, "Contributions to the study of fake news in portuguese: New corpus and automatic detection results," in *International Conference on Computational Processing of the Portuguese Language*, 2018, pp. 324–334. DOI: 10.1007/978-3-319-99722-3_33 (cited on p. 18).

[31] Y. Karimi, A. Squicciarini, and S. Wilson, "Automated detection of doxing on twitter," *ACM Human-Computer Interaction*, vol. 6, pp. 1–24, 2022. DOI: 10.1145/3555167 (cited on p. 18).

[32] D. M. Douglas, "Doxing: A conceptual analysis," *Ethics and Information Technology*, vol. 18, pp. 199–210, 2016. DOI: 10.1007/s10676-016-9406-0 (cited on p. 18).

[33] E. Spertus, "Smokey: Automatic recognition of hostile messages," in *14th National Conference on Artificial Intelligence and 9th Conference on Innovative Applications of Artificial Intelligence*, 1997, pp. 1058–1065. [Online]. Available: http://dl.acm.org/citation.cfm?id=1867616 (cited on pp. 20, 22, 46).

[34] N. D. Gitari, Z. Zuping, H. Damien, and J. Long, "A lexicon-based approach for hate speech detection," *International Journal of Multimedia and Ubiquitous Engineering*, vol. 10, no. 4, pp. 215–230, 2015. DOI: 10.14257/ijmue.2015.10.4.21 (cited on pp. 20, 31).

[35] Z. Waseem and D. Hovy, "Hateful symbols or hateful people? predictive features for hate speech detection on twitter," in *NAACL Student Research Workshop*, 2016, pp. 88–93. [Online]. Available: http://www.aclweb.org/anthology/N16-2013 (cited on pp. 20, 31).

[36] T. Davidson, D. Warmsley, M. Macy, and I. Weber, "Automated hate speech detection and the problem of offensive language," in *International AAAI Conference on Web and Social Media*, 2017, pp. 512–515. DOI: 10.1609/icwsm.v11i1.14955 (cited on pp. 20, 31).

[37] E. Abdelzaher, "Lexicon-based detection of violence on social media," *Cognitive Semantics*, vol. 5, no. 1, pp. 32–69, 2019. DOI: 10.1163/23526416-00501002 (cited on p. 20).

[38] M. Fortunatus, P. Anthony, and S. Charters, "Combining textual features to detect cyberbullying in social media posts," *Procedia Computer Science*, vol. 176, pp. 612–621, 2020. DOI: 10.1016/j.procs.2020.08.063 (cited on pp. 20, 31).

[39] D. Yin, Z. Xue, L. Hong, B. D. Davison, A. Kontostathis, and L. Edwards, "Detection of harassment on Web 2.0," in *WWW Workshop: Content Analysis in the Web 2.0*, 2009, pp. 1–7. [Online]. Available: http://www.cse.lehigh.edu/~brian/pubs/2009/CAW2/ (cited on pp. 20, 22, 23, 31).

[40] K. Dinakar, R. Reichart, and H. Lieberman, "Modeling the detection of textual cyberbullying," in *5th ICWSM / Workshop on the Social Mobile Web*, 2011, pp. 11–17. [Online]. Available: https://www.aaai.org/ocs/index.php/ICWSM/ICWSM11/paper/view/3841 (cited on pp. 20, 46).
[41] J. Salminen, H. Almerekhi, M. Milenkovi, S. Jung, J. An, H. Kwak, and B. J. Jansen, "Anatomy of online hate: Developing a taxonomy and machine learning models for identifying and classifying hate in online news media.," in *ICWSM*, Jun. 2018. [Online]. Available: https://ojs.aaai.org/index.php/ICWSM/article/view/15028 (cited on pp. 20, 29).

[42] M. Wiegand, J. Ruppenhofer, A. Schmidt, and C. Greenberg, "Inducing a lexicon of abusive words a feature-based approach," in *2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 2018, pp. 1046–1056. DOI: 10.18653/v1/N18-1095 (cited on pp. 20, 22).

[43] H. Mubarak, K. Darwish, and W. Magdy, "Abusive language detection on arabic social media," in *Proceedings of the First Workshop on Abusive Language Online*, 2017, pp. 52–56. DOI: 10.18653/v1/W17-3008 (cited on p. 20).

[44] R. Martins, J. Almeida, P. Henriques, and P. Novais, "Increasing authorship identification through emotional analysis," in *World Conference on Information Systems and Technologies*, 2018, pp. 763–772. DOI: 10.1007/978-3-319-77703-0_76 (cited on p. 20).

[45] E. Alawneh, M. Al-Fawa'reh, J. M. T., and F. M. A., "Sentiment analysis-based sexual harassment detection using machine learning techniques," in *International Symposium on Electronics and Smart Devices*, 2021, pp. 1–6. DOI: 10.1109/ISESD53023.2021.9501725 (cited on p. 20).

[46] N. Djuric, J. Zhou, R. Morris, M. Grbovic, V. Radosavljevic, and N. Bhamidipati, "Hate speech detection with comment embeddings," in *24th WWW*, 2015, pp. 29–30. DOI: 10.1145/2740908.2742760 (cited on p. 21).

[47] Q. Le and T. Mikolov, "Distributed representations of sentences and documents," in *31st International Conference on International Conference on Machine Learning*, vol. 32, 2014, pp. 1188–1196 (cited on pp. 21, 77).

[48] J. Pavlopoulos, P. Malakasiotis, and I. Androutsopoulos, "Deep learning for user comment moderation," in *Workshop on Abusive Language Online*, vol. First Workshop on Abusive Language Online, Aug. 2017, pp. 25–35. DOI: 10.18653/v1/W17-3004 (cited on pp. 21, 29).

[49] P. Mishra, H. Yannakoudakis, and E. Shutova, "Neural character-based composition models for abuse detection," in *2nd Workshop on Abusive Language Online*, vol. 2nd Workshop on Abusive Language Online (ALW2), Oct. 2018, pp. 1–10. DOI: 10.18653/v1/W18-5101 (cited on pp. 21, 96).

[50] P. Badjatiya, S. Gupta, M. Gupta, and V. Varma, "Deep learning for hate speech detection in tweets," in *26th International Conference on World Wide Web Companion*, 2017, pp. 759–760. DOI: 10.1145/3041021.3054223 (cited on pp. 21, 29).
[51] R. Kshirsagar, T. Cukuvac, K. McKeown, and S. McGregor, "Predictive embeddings for hate speech detection on twitter," in *2nd Workshop on Abusive Language Online*, 2018, pp. 26–32. DOI: 10.18653/v1/W18-5104 (cited on p. 21).

[52] R. Cao, R. K.-W. Lee, and T.-A. Hoang, "Deephate: Hate speech detection via multi-faceted text representations," in *12th ACM Conference on Web Science*, 2020, pp. 11–20. DOI: 10.1145/3394231.3397890 (cited on p. 21).

[53] O. E. Ojo, T. H. Ta, A. Gelbukh, H. Calvo, G. Sidorov, and O. O. Adebanji, "Automatic hate speech detection using deep neural networks and word embedding," *Computacion y Sistemas*, vol. 26, no. 2, pp. 1007–1013, 2022. DOI: 10.13053/CyS-26-2-4107 (cited on p. 21).

[54] N. Badri, F. Kboubi, and A. H. Chaibi, "Combining fasttext and glove word embedding for offensive and hate speech text detection," *Procedia Computer Science*, vol. 207, pp. 769–778, 2022. DOI: 10.1016/j.procs.2022.09.132 (cited on pp. 21, 29).

[55] B. Gambäck and U. K. Sikdar, "Using convolutional neural networks to classify hate-speech," in *First Workshop on Abusive Language Online*, 2017, pp. 85–90. DOI: 10.18653/v1/W17-3013 (cited on p. 21).

[56] H. Faris, I. Aljarah, M. Habib, and P. A. Castillo, "Hate speech detection using word embedding and deep learning in the arabic language context," in *9th International Conference on Pattern Recognition Applications and Methods*, 2020, pp. 453–460. DOI: 10.5220/0008954004530460 (cited on p. 21).

[57] A. Mazari, N. Boudoukhani, and A. Djeffal, "Bert-based ensemble learning for multi-aspect hate speech detection," *Cluster Computing*, 2023. DOI: 10.1007/s10586-022-03956-x (cited on p. 21).

[58] T. Caselli, V. Basile, J. Mitrovi, and M. Granitzer, "HateBERT: Retraining BERT for abusive language detection in english," in *5th Workshop on Online Abuse and Harms*, ACL, 2021, pp. 17–25. DOI: 10.18653/v1/2021.woah-1.3 (cited on p. 21).

[59] M. A. Khan, N. Yadav, M. Jain, and S. Goyal, "International conference on learning representations," in *The Art of Embedding Fusion: Optimizing Hate Speech Detection*, 2023. [Online]. Available: https://arxiv.org/pdf/2306.14939.pdf (cited on p. 21).

[60] N. Zampieri, I. Illina, and D. Fohr, "Improving hate speech detection with self-attention mechanism and multi-task learning," in *10th Language & Technology Conference: HumanLanguage Technologies as a Challenge for Computer Science and Linguistics*, 2023. [Online]. Available: https://hal.laas.fr/INRIA/hal-04017250v1 (cited on p. 21).
[61] J. H. Park and P. Fung, "One-step and two-step classification for abusive language detection on twitter," in *First Workshop on Abusive Language Online*, 2017, pp. 41–45. DOI: 10.18653/v1/W17-3006 (cited on p. 21).

[62] M. P. Akhter, Z. Jiangbin, I. R. Naqvi, M. AbdelMajeed, and T. Zia, "Abusive language detection from social media comments using conventional machine learning and deep learning approaches," *Multimedia Systems*, vol. 28, no. 6, pp. 1925–1940, 2022. DOI: 10.1007/s00530-021-00784-8 (cited on p. 21).

[63] S.-J. Bu and S.-B. Cho, "A hybrid deep learning system of cnn and lrcn to detect cyber-bullying from sns comments," in *International Conference on Hybrid Artificial Intelligence Systems*, 2018, pp. 561–572. DOI: 10.1007/978-3-319-92639-1_47 (cited on p. 22).

[64] R. Beniwal and A. Maurya, "Toxic comment classification using hybrid deep learning model," in *Sustainable Communication Networks and Application*, 2021, pp. 461–473. DOI: 10.1007/978-981-15-8677-4_38 (cited on p. 22).

[65] M. Alotaibi, B. Alotaibi, and A. Razaque, "A multichannel deep learning framework for cyber-bullying detection on social media," *Electronics*, vol. 10, 2021. DOI: 10.3390/electronics10212664 (cited on p. 22).

[66] M. Raj, S. Singh, K. Solanki, and R. Selvanambi, "An application to detect cyberbullying using machine learning and deep learning techniques," *Springer Nature Computer Science*, vol. 3, 2022. DOI: 10.1007/s42979-022-01308-5 (cited on p. 22).

[67] B. A. H. Murshed, J. Abawajy, S. Mallappa, M. A. N. Saif, and H. D. E. Al-Ariki, "Dea-rnn: A hybrid deep learning approach for cyberbullying detection in twitter social media platform," *IEEE Access*, vol. 10, pp. 25857–25871, 2022. DOI: 10.1109/ACCESS.2022.3153675 (cited on p. 22).

[68] A. Anuchitanukul, J. Ive, and L. Specia, "Revisiting contextual toxicity detection in conversations," *Journal of Data and Information Quality*, vol. 15, pp. 1–22, 2022. DOI: 10.1145/3561390 (cited on p. 23).

[69] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in *NAACL-HLT*, 2019, pp. 4171–4186. DOI: 10.18653/v1/N19-1423 (cited on p. 23).

[70] F. Vargas, F. Rodrigues de Góes, I. Carvalho, F. Benevenuto, and T. Pardo, "Contextual-lexicon approach for abusive language detection," in *International Conference on Recent Advances in Natural Language Processing*, 2021, pp. 1438–1447. [Online]. Available: https://aclanthology.org/2021.ranlp-1.161 (cited on p. 23).
[71] M. Karan and J. najder, "Preemptive toxic language detection in wikipedia comments using thread-level context," in *Third Workshop on Abusive Language Online*, 2019, pp. 129–134. DOI: 10.18653/v1/W19-3514 (cited on p. 23).

[72] H. Almerekhi, H. Kwak, B. J. Jansen, and J. Salminen, "Detecting toxicity triggers in online discussions," in *30th ACM Conference on Hypertext and Social Media*, 2019, pp. 291–292. DOI: 10.1145/3342220.3344933 (cited on p. 23).

[73] L. Gao and R. Huang, "Detecting online hate speech using context aware models," in *International Conference Recent Advances in Natural Language Processing*, 2017, pp. 260–266. DOI: 10.26615/978-954-452-049-6_036 (cited on p. 23).

[74] K. Balci and A. A. Salah, "Automatic analysis and identification of verbal aggression and abusive behaviors for online social games," *Computers in Human Behavior*, vol. 53, pp. 517–526, 2015. DOI: 10.1016/j.chb.2014.10.025 (cited on pp. 23, 24).

[75] M. Dadvar, F. de Jong, R. Ordelman, and D. Trieschnigg, "Improved cyberbullying detection using gender information.," in *12th Dutch-Belgian Information Retrieval Workshop*, 2012, pp. 23–26. [Online]. Available: http://dir2012.intec.ugent.be/system/files/proceedings/DIR2012_04_Maral_Dadvar.pdf (cited on p. 23).

[76] M. A. Al-Garadi, K. D. Varathan, and S. D. Ravana, "Cybercrime detection in online communications: The experimental case of cyberbullying detection in the twitter network," *Computers in Human Behavior*, vol. 63, pp. 433–443, 2016. DOI: 10.1016/j.chb.2016.05.051 (cited on pp. 23, 24).

[77] G. Xiang, B. Fan, L. Wang, J. Hong, and C. Rose, "Detecting offensive tweets via topical feature discovery over a large scale twitter corpus," in *21st ACM CIKM*, 2012, pp. 1980–1984. DOI: 10.1145/2396761.2398556 (cited on p. 24).

[78] D. Chatzakou, N. Kourtellis, J. Blackburn, E. De Cristofaro, G. Stringhini, and A. Vakali, "Mean birds: Detecting aggression and bullying on twitter," in *2017 ACM on Web Science Conference*, 2017, pp. 13–22. DOI: 10.1145/3091478.3091487 (cited on pp. 24, 30).

[79] C. Ziems, Y. Vigfusson, and F. Morstatter, "Aggressive, repetitive, intentional, visible, and imbalanced: Refining representations for cyberbullying classification," in *14th International AAAI Conference on Web and Social Media*, 2020. [Online]. Available: https://ojs.aaai.org/index.php/ICWSM/article/view/7345/7199 (cited on p. 24).

[80] E. F. Unsvåg and B. Gambäck, "The effects of user features on twitter hate speech detection," in *2nd Workshop on Abusive Language Online*, 2018, pp. 75–85. DOI: 10.18653/v1/W18-5110 (cited on p. 24).
[81] H. Hosseinmardi, R. I. Rafiq, R. Han, Q. Lv, and S. Mishra, "Prediction of cyberbullying incidents in a media-based social network," in *2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining*, 2016, pp. 186–192. DOI: 10.1109/ASONAM.2016.7752233 (cited on p. 24).

[82] J. Qian, M. ElSherief, E. Belding, and W. Y. Wang, "Leveraging intra-user and inter-user representation learning for automated hate speech detection," in *2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 2018, pp. 118–123. DOI: 10.18653/v1/N18-2019 (cited on p. 24).

[83] A. M. Founta, D. Chatzakou, N. Kourtellis, J. Blackburn, A. Vakali, and I. Leontiadis, "A unified deep learning architecture for abuse detection," in *10th ACM WebSci*, 2019, pp. 105–114. DOI: 10.1145/3292522.3326028 (cited on pp. 24, 96).

[84] M. Saveski, B. Roy, and D. Roy, "The structure of toxic conversations on twitter," in *Web Conference*, 2021, pp. 1086–1097. DOI: 10.1145/3442381.3449861 (cited on p. 24).

[85] A. Grover and J. Leskovec, "Node2vec: Scalable feature learning for networks," in *22nd ACM SIGKDD*, 2016, pp. 855–864. DOI: 10.1145/2939672.2939754 (cited on pp. 24, 70, 75, 101, 103, 112).

[86] P. Mishra, M. Del Tredici, H. Yannakoudakis, and E. Shutova, "Abusive language detection with graph convolutional networks," in *Conference of the North American Chapter of the ACL*, 2019, pp. 2145–2150. DOI: 10.18653/v1/N19-1221 (cited on p. 24).

[87] T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks," in *International Conference on Learning Representations*, 2017 (cited on pp. 24, 76).

[88] M. Ribeiro, P. Calais, Y. Santos, V. Almeida, and M. J. W., "Characterizing and detecting hateful users on twitter," in *International AAAI Conference on Web and Social Media*, 2018. [Online]. Available: https://arxiv.org/pdf/1803.08977.pdf (cited on p. 25).

[89] W. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning on large graphs," in *Advances in Neural Information Processing Systems*, 2017, pp. 1025–1035. [Online]. Available: https://dl.acm.org/doi/10.5555/3294771.3294869 (cited on pp. 25, 76).

[90] E. Wulczyn, N. Thain, and L. Dixon, "Ex machina: Personal attacks seen at scale," in *26th International Conference on World Wide Web*, International World Wide Web Conferences Steering Committee, Feb. 2017, pp. 1391–1399. DOI: 10.1145/3038912.3052591 (cited on pp. 25, 26, 28, 123, 131).
[91] A. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali, M. Sirivianos, and N. Kourtellis, "Large scale crowdsourcing and characterization of twitter abusive behavior," in *12th International AAAI Conference on Web and Social Media*, 2018. DOI: 10.1609/icwsm.v12i1.14991 (cited on pp. 25, 123).

[92] H. Kirk, W. Yin, B. Vidgen, and P. Röttger, "Semeval-2023 task 10: Explainable detection of online sexism," in *17th International Workshop on Semantic Evaluation*, 2023, pp. 2193–2210. DOI: 10.18653/v1/2023.semeval-1.305 (cited on pp. 26, 27).

[93] X. Yu, E. Blance, and L. Hong, "Hate speech and counter speech detection: Conversational context does matter," in *Conference of the NAACL: Human Language Technologies*, 2022, pp. 5918–5930. DOI: 10.18653/v1/2022.naacl-main.433 (cited on p. 26).

[94] T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar, "Toxigen: A large-scale machine-generated dataset for adversarial land implicit hate speech detection," in *60th Annual Meeting of the ACL*, 2022, pp. 3309–3326. [Online]. Available: https://aclanthology.org/2022.acl-long.234.pdf (cited on pp. 25–27).

[95] F. Beyhan, B. Çark, . Arn, A. Terziolu, B. Yanikoglu, and R. Yeniterzi, "A turkish hate speech dataset and detection system," in *13th Language Resources and Evaluation Conference*, 2022, pp. 4177–4185. [Online]. Available: https://aclanthology.org/2022.lrec-1.443 (cited on pp. 26, 27).

[96] B. Vidgen, T. Thrush, Z. Waseem, and D. Kiela, "Learning from the worst: Dynamically generated datasets to improve online hate detection," in *59th Annual Meeting of the ACL and the 11th International Joint Conference on NLP*, 2021, pp. 1667–1682. DOI: 10.18653/v1/2021.acl-long.132 (cited on pp. 25–27).

[97] A. Xenos, J. Pavlopoulos, and I. Androutsopoulos, "Context sensitivity estimation in toxicity detection," in *5th Workshop on Online Abuse and Harms*, 2021, pp. 140–145. DOI: 10.18653/v1/2021.woah-1.15 (cited on pp. 26, 27).

[98] T. Mandl, S. Modha, G. K. Shahi, A. K. Jaiswal, D. Nandini, D. Patel, P. Majumder, and J. Schäfer, "Overview of the hasoc track at fire 2020: Hate speech and offensive content identification inindo-european languages," in *CEUR Workshop*, 2020, pp. 87–111. [Online]. Available: https://ceur-ws.org/Vol-2826/T2-1.pdf (cited on p. 26).

[99] V. Basile, C. Bosco, E. Fersini, D. Nozza, V. Patti, F. M. Rangel Pardo, P. Rosso, and M. Sanguinetti, "Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter," in *13th International Workshop on Semantic Evaluation*, 2019, pp. 54–63. DOI: 10.18653/v1/S19-2007 (cited on pp. 26, 27).
[100] B. Vidgen, D. Nguyen, H. Margetts, P. Rossini, and R. Tromble, "Introducing cad: The con-textual abuse dataset," in *2021 Conference of the NA Chapter of the ACL: Human Language Technologies*, 2021, pp. 2289-2303. DOI: 10.18653/v1/2021.nacl-main.182 (cited on pp. 26, 27).

[101] A. Ollagnier, E. Cabrio, S. Villata, and C. Blaya, "Cyberagressionado-v1: A dataset of annotated online aggressions in french collected through a role-playing game," in *13th Language Resources and Evaluation Conference*, 2022, pp. 867-875. [Online]. Available: https://aclanthology.org/2022.lrec-1.91 (cited on pp. 26, 27).

[102] R. Hada, S. Sudhir, P. Mishra, H. Yannakoudakis, S. M. Mohammad, and E. Shutova, "Ruddit: Norms of offensiveness for english reddit comments," in *ACL-IJCNLP 2021*, 2021, pp. 2700-2717. DOI: 10.18653/v1/2021.acl-long.210 (cited on pp. 26, 27, 129).

[103] E. Papegnies, V. Labatut, R. Dufour, and G. Linarès, "Detection of abusive messages in an on-line community," in *CORIA 2017 - Conférence en Recherche d'Informations et Applications- 14th French Information Retrieval Conference*, Mar. 2017, pp. 153-168. DOI: doi:10.24348/coria.2017.16 (cited on pp. 27, 45, 46).

[104] Y. Hua, C. Danescu-Niculescu-Mizil, D. Taraborelli, N. Thain, J. Sorensen, and L. Dixon, "Wikiconv: A corpus of the complete conversational history of a large online collaborative community," in *Conference on Empirical Methods in Natural Language Processing*, vol. 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2818-2823. DOI: 10.18653/v1/d18-1305 (cited on pp. 29, 132).

[105] S. Kirchenko, I. Nejadgholi, and K. C. Fraser, "Confronting abusive language online: A survey from the ethical and human rights perspective," *Journal of Artificial Intelligence Research*, vol. 71, pp. 431-478, 2021. DOI: 10.1613/jair.1.12590 (cited on p. 29).

[106] N. Babakov, V. Logacheva, and A. Panchenko, "Beyond plain toxic: Building datasets for detection of flammable topics and inappropriate statements," *Language Resources and Evaluation*, 2023. DOI: 10.1007/s10579-023-09682-z (cited on p. 29).

[107] Z. Li, M. Rei, and L. Specia, "Multimodal conversation modelling for topic derailment detection," in *EMNLP*, 2022, pp. 5115-5127. DOI: 10.18653/v1/2022.findings-emnlp.376 (cited on p. 29).

[108] F. Syeda Faizan, L. Seemab, and L. Rabia, "Fine tuning bert for unethical behavior classification," in *International Conference on Digital Futures and Transformative Technologies*, 2021, pp. 1-6. DOI: 10.1109/ICoDT252288.2021.9441540 (cited on p. 29).
[109] T. Xu, G. Goossen, H. K. Cevahir, S. Khodeir, Y. Jin, F. Li, S. Shan, S. Patel, D. Freeman, and P. Pearce, "Deep entity classification: Abusive account detection for online social networks," in *30th USENIX Security Symposium*, 2021, pp. 4097-4114. [Online]. Available: https://www.usenix.org/conference/usenixsecurity21/presentation/xu-teng (cited on p. 29).

[110] S. Subramani, S. Michalska, H. Wang, J. Du, Y. Zhang, and H. Shakeel, "Deep learning for multi-class identification from domestic violence online posts," *IEEE Access*, vol. 7, pp. 46210-46224, 2019. DOI: 10.1109/ACCESS.2019.2908827 (cited on p. 29).

[111] M. Jorgensen, M. Choi, M. Niemann, J. Brunk, and J. Becker, "Multi-class detection of abusive language using automated machine learning," in *15th International Conference on Wirtschaftsinformatik*, 2020, pp. 1763-1775. DOI: 10.30844/wi_2020_r7-jorgensen (cited on p. 29).

[112] A. Garcia-Recuero, "Discouraging abusive behavior in privacy-preserving online social networking applications," in *25th International Conference Companion on World Wide Web*, 2016, pp. 305-309. DOI: 10.1145/2872518.2888600 (cited on pp. 30, 31).

[113] J. M. Lobo, A. Jimenez-Valverde, and R. Real, "Auc: A misleading measure of the performance of predictive distribution models," *Global Ecology and Biogeography*, vol. 17, pp. 145-151, 2008. DOI: 10.1111/j.1466-8238.2007.00358.x (cited on p. 31).

[114] S. Halligan, D. Altman, and S. Mallett, "Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach," *European Radiology*, vol. 25, pp. 932-939, 2015. DOI: 10.1007/s00330-014-3487-0 (cited on p. 31).

[115] J. Muschelli, "Roc and auc with a binary predictor: A potentially misleading metric," *Journal of Classification*, vol. 37, pp. 696-708, 2019. DOI: 10.1007/s00357-019-09345-1 (cited on p. 31).

[116] K. Orman, "Contribution to the interpretation of evolving communities in complex networks: Application to the study of social interactions," PhD thesis, LIRIS, 2014. [Online]. Available: https://www.theses.fr/2014ISAL0072 (cited on p. 33).

[117] D. J. Watts and S. H. Strogatz, "Collective dynamics of 'small-world' networks," *Nature*, vol. 393, no. 6684, pp. 440-442, 1998. DOI: 10.1038/30918 (cited on p. 36).

[118] A. Barrat, M. Barthelemy, R. Pastor-Satorras, and A. Vespignani, "The architecture of complex weighted networks," *Proceedings of the National Academy of Sciences*, vol. 101, no. 11, pp. 3747-3752, 2004. DOI: 10.1073/pnas.0400087101 (cited on p. 36).

[119] R. S. Burt, "Structural holes and good ideas," *American Journal of Sociology*, vol. 110, no. 2, pp. 349-399, 2004. DOI: 10.1086/421787 (cited on p. 37).
[120] P. Bonacich, "Factoring and weighting approaches to status scores and clique identification," *Journal of Mathematical Sociology*, vol. 2, no. 1, pp. 113–120, 1972. DOI: 10.1080/0022250X.1972.9989806 (cited on p. 37).

[121] S. Brin and L. E. Page, "The anatomy of a large-scale hypertextual Web search engine," *Computer Networks and ISDN Systems*, vol. 30, pp. 107–117, 1998. DOI: 10.1016/S0169-7552(98)00110-X (cited on p. 38).

[122] J. Kleinberg, "Authoritative sources in a hyperlinked environment," *Journal of the Association for Computing Machinery*, vol. 46, no. 5, pp. 604–632, 1999. DOI: 10.1145/324133.324140 (cited on p. 38).

[123] L. Katz, "A new status index derived from sociometric analysis," *Psychometrika*, vol. 18, no. 1, pp. 39–43, 1953. DOI: 10.1007/bf02289026 (cited on p. 39).

[124] P. Bonacich and P. Lloyd, "Eigenvector-like measures of centrality for asymmetric relations," *Social Networks*, vol. 23, no. 3, pp. 191–201, 2001. DOI: 10.1016/S0378-8733(01)00038-7 (cited on p. 39).

[125] P. F. Bonacich, "Power and centrality: A family of measures," *American Journal of Sociology*, vol. 92, no. 5, pp. 1170–1182, 1987. DOI: 10.1086/228631 (cited on p. 39).

[126] E. Estrada and J. A. Rodriguez-Velazquez, "Subgraph centrality in complex networks," *Physical Review E*, vol. 71, no. 5, p. 056103, 2005. DOI: 10.1103/PhysRevE.71.056103 (cited on p. 39).

[127] L. C. Freeman, "A set of measures of centrality based on betweenness," *Sociometry*, vol. 40, no. 1, pp. 35–41, 1977. DOI: 10.2307/3033543 (cited on p. 40).

[128] A. Bavelas, "Communication patterns in task-oriented groups," *Journal of the Acoustical Society of America*, vol. 22, no. 6, pp. 725–730, 1950. DOI: 10.1121/1.1906679 (cited on p. 40).

[129] F. Harary, *Graph Theory*. Addison-Wesley, 1969. [Online]. Available: http://www.dtic.mil/dtic/tr/fulltext/u2/705364.pdf (cited on pp. 40, 41).

[130] S. B. Seidman, "Network structure and minimum degree," *Social Networks*, vol. 5, no. 3, pp. 269–287, 1983. DOI: 10.1016/0378-8733(83)90028-X (cited on p. 41).

[131] R. Guimerà and L. A. N. Amaral, "Functional cartography of complex metabolic networks," *Nature*, vol. 433, pp. 895–900, 2005. DOI: 10.1038/nature03288 (cited on p. 41).

[132] N. Dugué, V. Labatut, and A. Perez, "Identifying the community roles of social capitalists in the Twitter network," in *IEEE/ACM International Conference on Advances in Social Network Analysis and Mining*, IEEE, 2014, pp. 371–374. DOI: 10.1109/ASONAM.2014.6921612 (cited on p. 41).
[133] R. D. Luce and A. D. Perry, "A method of matrix analysis of group structure," *Psychometrika*, vol. 14, no. 2, pp. 95-116, 1949. DOI: 10.1007/BF02289146 (cited on p. 42).

[134] S. Wasserman and K. Faust, *Social Network Analysis: Methods and Applications*, ser. Structural Analysis in the Social Sciences. Cambridge, UK: Cambridge University Press, 1994, vol. 8. [Online]. Available: http://www.cambridge.org/zw/academic/subjects/sociology/sociology-general-interest/social-network-analysis-methods-and-applications (cited on p. 43).

[135] D. R. White, F. Harary, M. Sobel, and M. Becker, "The cohesiveness of blocks in social networks: Node connectivity and conditional density," *Sociological Methodology*, vol. 31, no. 1, pp. 305-359, 2001. DOI: 10.1111/0081-1750.00098 (cited on p. 43).

[136] M. E. J. Newman and M. Girvan, "Finding and evaluating community structure in networks," *Physical Review E*, vol. 69, no. 2, p. 026113, 2004. DOI: 10.1103/PhysRevE.69.026113 (cited on p. 44).

[137] S. Fortunato, "Community detection in graphs," *Physics Reports*, vol. 486, no. 3, Community detection in graphs, 2010. DOI: https://doi.org/10.1016/j.physrep.2009.11.002 (cited on p. 44).

[138] P. Mutton, "Inferring and visualizing social networks on internet relay chat," in *8th International Conference on Information Visualisation*, 2004, pp. 35-43. DOI: 10.1109/IV.2004.1320122 (cited on p. 47).

[139] R. Dzisević and D. eok, "Text classification using different feature extraction approaches," in *Open Conference of Electrical, Electronic and Information Sciences*, 2019, pp. 1-4. DOI: 10.1109/eStream.2019.8732167 (cited on p. 52).

[140] R. Campos, V. Mangaravite, A. Pasquali, A. M. Jorge, C. Nunes, and A. Jatowt, "A text feature based automatic keyword extraction method for single documents," in *Advances in Information Retrieval*, 2018, pp. 684-691. DOI: https://doi.org/10.1007/978-3-319-76941-7_63 (cited on p. 52).

[141] D. D. Lewis, "Feature selection and feature extraction for text categorization," in *Proceedings of the Workshop on Speech and Natural Language*, 1992, pp. 212-217. [Online]. Available: https://aclanthology.org/H92-1041.pdf (cited on p. 52).

[142] Y. Chen, Y. Zhou, S. Zhu, and H. Xu, "Detecting offensive language in social media to protect adolescent online safety," in *International Conference on Privacy, Security, Risk and Trust and International Conference on Social Computing*, 2012, pp. 71-80. DOI: 10.1109/SocialCom-PASSAT.2012.55 (cited on p. 52).
[143] H. Chen, S. Mckeever, and S. J. Delany, "Presenting a labelled dataset for real-time detection of abusive user posts," in *International Conference on Web Intelligence*, 2017, pp. 884–890. DOI: 10.1145/3106426.3106456 (cited on p. 52).

[144] L. V. Batista and M. M. Meira, "Texture classification using the lempel-ziv-welch algorithm," in *Brazilian Symposium on Artificial Intelligence*, 2004, pp. 444–453. DOI: 10.1007/978-3-540-28645-5_45 (cited on p. 52).

[145] M. Rosvall and C. T. Bergstrom, "Maps of random walks on complex networks reveal community structure," *Proceedings of the National Academy of Sciences*, vol. 105, no. 4, p. 1118, 2008. DOI: 10.1073/pnas.0706851105 (cited on p. 56).

[146] G. Csardi and T. Nepusz, "The igraph software package for complex network research," *InterJournal*, vol. 1695, 2006. [Online]. Available: http://igraph.sf.net (cited on p. 59).

[147] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," *Journal of Machine Learning Research*, vol. 12, pp. 2825–2830, 2011. [Online]. Available: http://www.jmlr.org/papers/v12/pedregosa11a.html (cited on p. 59).

[148] K. Babi, S. Martini-Ipi, and A. Metrovi, "Survey of neural text representation models," *Information*, vol. 11, no. 11, 2020. DOI: 10.3390/info11110511 (cited on p. 68).

[149] T. Schomacker and M. Tropmann-Frick, "Language representation models: An overview," *Entropy*, vol. 23, no. 11, 2021. DOI: https://doi.org/10.3390/e23111422 (cited on p. 68).

[150] P. Siebers, C. Janiesch, and P. Zschech, "A survey of text representation methods and their genealogy," *IEEE Access*, vol. 10, pp. 96492–96513, 2022. DOI: 10.1109/ACCESS.2022.3205719 (cited on p. 68).

[151] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, "Self-supervised speech representation learning: A review," *IEEE Journal of Selected Topics in Signal Processing*, vol. 16, pp. 1179–1210, 2022. DOI: 10.1109/JSTSP.2022.3207050 (cited on p. 68).

[152] R. Patil, S. Boit, V. Gudivada, and J. Nandigam, "A survey of text representation and embedding techniques in nlp," *IEEE Access*, vol. 11, pp. 36120–36146, 2023. DOI: 10.1109/ACCESS.2023.3266377 (cited on p. 68).

[153] F. Incitti, F. Urli, and L. Snidaro, "Beyond word embeddings: A survey," *Information Fusion*, vol. 89, pp. 418–436, 2023. DOI: https://doi.org/10.1016/j.inffus.2022.08.024 (cited on p. 68).
[154] Z. Liu. and M. Sun, "Representation learning and nlp," in *Representation Learning for Natural Language Processing*, Springer Nature Singapore Singapore, 2023, pp. 1-27. DOI: https://doi.org/10.1007/978-981-99-1600-9_1 (cited on p. 68).

[155] M. Ou, P. Cui, J. Pei, Z. Zhang, and W. Zhu, "Asymmetric transitivity preserving graph embedding," in *22nd ACM SIGKDD*, 2016, pp. 1105-1114. DOI: 10.1145/2939672.2939751 (cited on pp. 70, 74, 75).

[156] M. Belkin and P. Niyogi, "Laplacian eigenmaps and spectral techniques for embedding and clustering," in *Advances in Neural Information Processing Systems 14*, 2002, pp. 585-591. [Online]. Available: http://papers.nips.cc/paper/1961-laplacian-eigenmaps-and-spectral-techniques-for-embedding-and-clustering.pdf (cited on pp. 70, 73, 74, 77).

[157] A. Ahmed, N. Shervashidze, S. Narayanamurthy, V. Josifovski, and A. J. Smola, "Distributed large-scale natural graph factorization," in *22nd International Conference on World Wide Web*, 2013, pp. 37-48. DOI: 10.1145/2488388.2488393 (cited on pp. 70, 74).

[158] S. Yuan, X. Wu, and Y. Xiang, "SNE: Signed network embedding," in *Pacific-Asia Conference on Knowledge Discovery and Data Mining*, ser. Lecture Notes in Computer Science, vol. 10235, 2017, pp. 183-195. DOI: 10.1007/978-3-319-57529-2_15 (cited on pp. 70, 79).

[159] J. Kim, H. Park, J.-E. Lee, and U. Kang, "SIDE: Representation learning in signed directed networks," in *World Wide Web Conference*, 2018, pp. 509-518. DOI: 10.1145/3178876.3186117 (cited on pp. 70, 79).

[160] T. Derr, Y. Ma, and J. Tang, "Signed graph convolutional network," in *18th ICDM*, 2018, pp. 929-934. DOI: 10.1109/ICDM.2018.00113 (cited on pp. 70, 80).

[161] A. Narayanan, M. Chandramohan, R. Venkatesan, L. Chen, Y. Liu, and S. Jaiswal, "graph2vec: Learning distributed representations of graphs," in *MLG*, 2017 (cited on pp. 70, 77, 84, 102, 103, 105).

[162] F. Heider, "Attitudes and cognitive organization," *Journal of Psychology*, vol. 21, no. 1, pp. 107-112, 1946. DOI: 10.1080/00223980.1946.9917275 (cited on p. 71).

[163] F. Harary, "On the notion of balance of a signed graph," *Michigan Mathematical Journal*, vol. 2, no. 2, pp. 143-146, 1953. DOI: 10.1307/mmj/1028989917 (cited on p. 71).

[164] D. Cartwright and F. Harary, "Structural balance: A generalization of Heider's theory," *Psychological Review*, vol. 63, pp. 277-293, 1956. DOI: 10.1037/h0046049 (cited on p. 71).

[165] P. Doreian and A. Mrvar, "A partitioning approach to structural balance," *Social Networks*, vol. 18, no. 2, pp. 149-168, 1996. DOI: 10.1016/0378-8733(95)00259-6 (cited on p. 72).

[166] N. Bansal, A. Blum, and S. Chawla, "Correlation clustering," in *FOCS*, 2002, pp. 238-247. DOI: 10.1109/SFCS.2002.1181947 (cited on pp. 72, 81).
[167] J. A. Davis, "Clustering and structural balance in graphs," *Human Relations*, vol. 20, no. 2, pp. 181-187, 1967. DOI: 10.1177/001872676702000207 (cited on p. 72).

[168] H. Cai, V. W. Zheng, and K. C.-C. Chang, "A comprehensive survey of graph embedding: Problems, techniques, and applications," *IEEE TKDE*, vol. 30, no. 9, pp. 1616-1637, 2018. DOI: 10.1109/TKDE.2018.2807452 (cited on p. 72).

[169] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Lin, "Graph embedding and extensions: A general framework for dimensionality reduction," *IEEE PAMI*, vol. 29, pp. 40-51, 2007. DOI: 10.1109/tpami.2007.250598 (cited on pp. 72, 73).

[170] P. Goyal and E. Ferrara, "Graph embedding techniques, applications, and performance: A survey," *Knowledge-Based Systems*, vol. 151, pp. 78-94, 2018, ISSN: 0950-7051. DOI: https://doi.org/10.1016/j.knosys.2018.03.022 (cited on pp. 73, 74, 76, 96, 112).

[171] S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," *Science*, vol. 290, no. 5500, pp. 2323-2326, 2000. DOI: 10.1126/science.290.5500.2323 (cited on p. 73).

[172] X. Liang, D. Li, M. Song, A. Madden, Y. Ding, and Y. Bu, "Predicting biomedical relationships using the knowledge and graph embedding cascade model," *PLoS ONE*, vol. 14, 2019. DOI: https://doi.org/10.1371/journal.pone.0218264 (cited on pp. 74, 75).

[173] J. Li, L. Wu, R. Guo, C. Liu, and H. Liu, "Multi-level network embedding with boosted low-rank matrix approximation," in *2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining*, 2019, pp. 49-56. DOI: 10.1145/3341161.3342864 (cited on pp. 74, 102, 103).

[174] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," in *ICLR*, 2013. [Online]. Available: https://arxiv.org/abs/1301.3781 (cited on pp. 74, 75, 98).

[175] B. Perozzi, R. Al-Rfou, and S. Skiena, "DeepWalk: Online learning of social representations," in *20th ACM SIGKDD*, 2014, pp. 701-710. DOI: 10.1145/2623330.2623732 (cited on pp. 75, 101, 103).

[176] B. Hou, Y. Wang, M. Zeng, S. Jiang, O. J. Mengshoel, Y. Tong, and J. Bai, "Customized graph embedding: Tailoring embedding vectors to different applications," *arXiv*, 2019. [Online]. Available: http://arxiv.org/abs/1911.09454 (cited on p. 75).

[177] D. Wang, P. Cui, and W. Zhu, "Structural deep network embedding," in *22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, 2016, pp. 1225-1234. DOI: 10.1145/2939672.2939753 (cited on pp. 75, 76).
[178] H. Chen, B. Perozzi, Y. Hu, and S. Skiena, "HARP: Hierarchical representation learning for networks," in *32nd AAAI Conference on Artificial Intelligence*, 2018. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewFile/16273/15922 (cited on p. 75).

[179] B. Perozzi, V. Kulkarni, and S. Skiena, "Don't walk, skip! online learning of multi-scale network embeddings," in *IEEE/ACM ASONAM*, 2017, pp. 258-265. DOI: 10.1145/3110025.3110086 (cited on pp. 75, 101, 103).

[180] H. Wang, J. Wang, J. Wang, M. Zhao, W. Zhang, F. Zhang, X. Xie, and M. Guo, "GraphGAN: Graph representation learning with generative adversarial nets," in *32nd AAAI Conference on Artificial Intelligence*, 2018, pp. 2508-2515. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16611 (cited on p. 76).

[181] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and P. S. Yu, "A comprehensive survey on graph neural networks," *IEEE Transactions on Neural Networks and Learning Systems*, vol. 32, no. 1, pp. 4-24, 2021. DOI: 10.1109/tnnls.2020.2978386 (cited on p. 76).

[182] S. Zhang, H. Tong, J. Xu, and R. Maciejewski, "Graph convolutional networks: Algorithms, applications and open challenges," in *International Conference on Computational Social Networks*, ser. Lecture Notes in Computer Science, vol. 11280, 2018, pp. 79-91. DOI: 10.1007/978-3-030-04648-4_7 (cited on p. 76).

[183] N. de Lara and E. Pineau, "A simple baseline algorithm for graph classification," *arXiv*, 2018. eprint: 1810.09155 (cs.LG). [Online]. Available: https://arxiv.org/pdf/1810.09155.pdf (cited on pp. 77, 102, 103).

[184] S. F. Mousavi, M. Safayani, A. Mirzaei, and H. Bahonar, "Hierarchical graph embedding in vector space by graph pyramid," *Pattern Recognition*, vol. 61, pp. 245-254, 2017, ISSN: 0031-3203. DOI: 10.1016/j.patcog.2016.07.043 (cited on p. 77).

[185] S. Verma and Z.-L. Zhang, "Hunt for the unique, stable, sparse and fast feature learning on graphs," in *Advances in Neural Information Processing Systems 30*, 2017, pp. 88-98. [Online]. Available: http://papers.nips.cc/paper/6614-hunt-for-the-unique-stable-sparse-and-fast-feature-learning-on-graphs.pdf (cited on pp. 77, 102, 103).

[186] A. Tsitsulin, D. Mottin, P. Karras, A. Bronstein, and E. Müller, "NetLSD: Hearing the shape of a graph," in *24th ACM SIGKDD*, 2018, pp. 2347-2356. DOI: 10.1145/3219819.3219991 (cited on p. 77).

[187] T. Huynh, T. T. Thi Ho, and B. Le, "Graph classification via graph structure learning," in *Asian Conference on Intelligent Information and Database Systems*, ser. Lecture Notes in Computer Science, vol. 13758, Springer, 2022, pp. 269-281. DOI: 10.1007/978-3-031-21967-2_22 (cited on p. 77).
[188] R. Winter, F. Noe, and D.-A. Clevert, "Permutation-invariant variational autoencoder for graph-level representation learning," in *Conference on Neural Information Processing Systems*, 2021 (cited on p. 78).

[189] L. Gutiérrez-Gómez and J.-C. Delvenne, "Unsupervised network embeddings with node identity awareness," *Applied Network Science*, vol. 4, p. 82, 2019. DOI: 10.1007/s41109-019-0197-1 (cited on p. 78).

[190] R. Al-Rfou, B. Perozzi, and D. Zelle, "Ddgk: Learning graph representations for deep divergence graph kernels," in *World Wide Web Conference*, 2019, pp. 37–48. DOI: 10.1145/3308558.3313668 (cited on p. 78).

[191] M. Niepert, M. Ahmed, and K. Kutzkov, "Learning convolutional neural networks for graphs," in *33rd International Conference on International Conference on Machine Learning*, 2016, pp. 2014–2023 (cited on p. 78).

[192] Z. Luo, L. Liu, J. Yin, Y. Li, and Z. Wu, "Deep learning of graphs with ngram convolutional neural networks," *IEEE Transactions on Knowledge and Data Engineering*, vol. 29, no. 10, pp. 2125–2139, 2017. DOI: 10.1109/tkde.2017.2720734 (cited on p. 78).

[193] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, "Neural message passing for quantum chemistry," in *Proceedings of the 34th ICML*, 2017, pp. 1263–1272 (cited on pp. 78, 85).

[194] T. Pham, T. Tran, H. Dam, and S. Venkatesh, "Graph classification via deep learning with virtual nodes," *arXiv*, vol. cs.LG, p. 1708.04357, 2017 (cited on p. 78).

[195] S. Wang, J. Tang, C. Aggarwal, Y. Chang, and H. Liu, "Signed network embedding in social media," in *17th SIAM International Conference on Data Mining*, 2017, pp. 327–335. DOI: 10.1137/1.9781611974973.37 (cited on p. 79).

[196] M. Cygan, M. Pilipczuk, M. Pilipczuk, and J. Wojtaszczyk, "Sitting closer to friends than enemies, revisited," in *Mathematical Foundations of Computer Science*, 2012, pp. 296–307. DOI: https://doi.org/10.1007/978-3-642-32589-2_28 (cited on p. 79).

[197] J. Huang, H. Shen, L. Hou, and X. Cheng, "Signed graph attention networks," in *International Conference on Artificial Neural Networks*, 2019 (cited on p. 80).

[198] Y. Li, Y. Tian, J. Zhang, and Y. Chang, "Learning signed network embedding via graph attention," in *34th AAAI Conference on Artificial Intelligence*, 2020 (cited on p. 80).

[199] N. Arnk, R. Figueiredo, and V. Labatut, "Multiplicity and diversity: Analysing the optimal solution space of the correlation clustering problem on complete signed graphs," *Journal of Complex Networks*, vol. 8, no. 6, 2021. DOI: 10.1093/comnet/cnaa025 (cited on p. 81).
[200] —, "Multiple partitioning of multiplex signed networks: Application to european parliament votes," *Social Networks*, vol. 60, pp. 83–102, 2020. DOI: 10.1016/j.socnet.2019.02.001 (cited on p. 81).

[201] B. Rozemberczki and R. Sarkar, "Characteristic functions on graphs: Birds of a feather, from statistical descriptors to parametric models," in *29th ACM International Conference on Information & Knowledge Management*, 2020, pp. 1325–1334. DOI: 10.1145/3340531.3411866 (cited on pp. 83, 100).

[202] A. Galland, "Deep learning techniques for graph embedding at different scales," PhD thesis, Université Paris sciences et lettres, 2020. [Online]. Available: https://theses.hal.science/tel-03690033/ (cited on p. 83).

[203] B. Y. Weisfeiler and A. A. Leman, "A reduction of a graph to a canonical form and an algebra arising during this reduction," *Nauchno-Technicheskaya Informatsia*, vol. 2, no. 9, pp. 12–16, 1968 (cited on p. 84).

[204] C. Fang, Z. Liu, Y. Shi, J. Huang, and Q. Shi, "Functional code clone detection with syntax and semantics fusion learning," in *International Symposium on Software Testing and Analysis*, 2020, pp. 516–527. DOI: 10.1145/3395363.3397362 (cited on p. 84).

[205] X. Zhou, Y. Zhang, Z. Li, X. Wang, J. Zhao, and Z. Zhang, "Large-scale cellular traffic prediction based on graph convolutional networks with transfer learning," *Neural Computing and Applications*, vol. 34, pp. 5549–5559, 2022. DOI: 10.1007/s00521-021-06708-x (cited on p. 84).

[206] Q.-D. Ngo, H.-T. Nguyen, H.-A. Tran, and D.-H. Nguyen, "IoT botnet detection based on the integration of static and dynamic vector features," in *8th International Conference on Communications and Electronics*, 2021, pp. 540–545. DOI: 10.1109/ICCE48956.2021.9352145 (cited on p. 84).

[207] Z. Zhang, J. Liu, X. Zheng, Y. Wang, P. Han, Y. Wang, K. Zhao, and Z. Zhang, "RSGNN: A model-agnostic approach for enhancing the robustness of signed graph neural networks," in *ACM Web Conference*, 2023, pp. 60–70. DOI: 10.1145/3543507.3583221 (cited on p. 85).

[208] P. Doreian and A. Mrvar, "Partitioning signed social networks," *Social Networks*, vol. 31, no. 1, pp. 1–11, 2009. DOI: 10.1016/j.socnet.2008.08.001 (cited on p. 94).

[209] R. Figueiredo and G. Moura, "Mixed integer programming formulations for clustering problems related to structural balance," *Social Networks*, vol. 35, no. 4, pp. 639–651, 2013. DOI: 10.1016/j.socnet.2013.09.002 (cited on p. 94).
[210] J. Padilla Montani and P. Schüller, "TUWienKBS at GermEval 2018: German abusive tweet detection," in *GermEval 2018 Workshop*, 2018, pp. 45–50. [Online]. Available: https://www.oew.ac.at/fileadmin/subsites/academiaecorpora/PDF/GermEval2018_Proceedings.pdf (cited on p. 96).

[211] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information," *Transactions of the ACL*, vol. 5, pp. 135–146, 2017. DOI: 10.1162/tacl_a_00051 (cited on p. 99).

[212] L. Martin, B. Muller, P. J. Ortiz Suárez, Y. Dupont, L. Romary, E. de la Clergerie, D. Seddah, and B. Sagot, "CamemBERT: A tasty french language model," in *ACL*, 2020, pp. 7203–7219. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.645 (cited on p. 99).

[213] H. Le, L. Vial, J. Frej, V. Segonne, M. Coavoux, B. Lecouteux, A. Allauzen, B. Crabbé, L. Besacier, and D. Schwab, "FlaubERT: Unsupervised language model pre-training for french," in *12th Language Resources and Evaluation Conference*, 2020, pp. 2479–2490. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.302 (cited on pp. 99, 109, 113).

[214] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," *arXiv*, 2019. [Online]. Available: https://arxiv.org/pdf/1907.11692.pdf (cited on p. 99).

[215] G. Wenzek, M.-A. Lachaux, A. Conneau, V. Chaudhary, F. Guzmán, A. Joulin, and E. Grave, "CCNet: Extracting high quality monolingual datasets from web crawl data," in *12th Language Resources and Evaluation Conference*, 2020, pp. 4003–4012. [Online]. Available: https://www.aclweb.org/anthology/2020.lrec-1.494.pdf (cited on p. 99).

[216] A. Akbik, D. Blythe, and R. Vollgraf, "Contextual string embeddings for sequence labeling," in *27th International Conference on Computational Linguistics*, 2018, pp. 1638–1649. [Online]. Available: https://www.aclweb.org/anthology/C18-1139 (cited on pp. 99, 117, 124, 127, 128).

[217] C. Donnat, M. Zitnik, D. Hallac, and J. Leskovec, "Learning structural node embeddings via diffusion wavelets," in *24th ACM SIGKDD*, 2018, pp. 1320–1329. DOI: 10.1145/3219819.3220025 (cited on pp. 102, 103).

[218] G. Nikolentzos, G. Dasoulas, and M. Vazirgiannis, "K-hop graph neural networks," *Neural Networks*, vol. 130, pp. 195–205, 2020. DOI: 10.1016/j.neunet.2020.07.008 (cited on pp. 102, 103).
[219] M. Zhang and P. Li, "Nested graph neural networks," in *Conference on Neural Information Processing Systems*, 2021, pp. 15734–15747. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/8462a7c229aea03dde69da754c3bbcc4-Paper.pdf (cited on pp. 102, 103).

[220] A. Rücklé, S. Eger, M. Peyrard, and I. Gurevych, "Concatenated power mean word embeddings as universal cross-lingual sentence representations," *arXiv*, 2018. [Online]. Available: https://arxiv.org/abs/1803.01400 (cited on p. 107).

[221] U. Singer, I. Guy, and K. Radinsky, "Node embedding over temporal graphs," in *International Joint Conference on Artificial Intelligence*, 2019, pp. 4605–4612. DOI: 10.24963/ijcai.2019/640 (cited on p. 124).