Using word embeddings to improve the discriminability of co-occurrence text networks

Abstract: Word co-occurrence networks have been employed to analyze texts both in the practical and theoretical scenarios. Despite the relative success in several applications, traditional co-occurrence networks fail in establishing links between similar words whenever they appear distant in the text. Here we investigate whether the use of word embeddings as a tool to create virtual links in co-occurrence networks may improve the quality of classification systems. Our results revealed that the discriminability in the stylometry task is improved when using Glove, Word2Vec and FastText. In addition, we found that optimized results are obtained when stopwords are not disregarded and a simple global thresholding strategy is used to establish virtual links. Because the proposed approach is able to improve the representation of texts as complex networks, we believe that it could be extended to study other natural language processing tasks. Likewise, theoretical languages studies could benefit from the adopted enriched representation of word co-occurrence networks.

The ability to construct complex and diverse linguistic structures is one of the main features that set us apart from all other species. Despite its ubiquity, some language aspects remain unknown. Topics such as language origin and evolution have been studied by researchers from diverse disciplines, including Linguistic, Computer Science, Physics and Mathematics BIBREF0, BIBREF1, BIBREF2. In order to better understand the underlying language mechanisms and universal linguistic properties, several models have been developed BIBREF3, BIBREF4. A particular language representation regards texts as complex systems BIBREF5. Written texts can be considered as complex networks (or graphs), where nodes could represent syllables, words, sentences, paragraphs or even larger chunks BIBREF5. In such models, network edges represent the proximity between nodes, e.g. the frequency of the co-occurrence of words. Several interesting results have been obtained from networked models, such as the explanation of Zipf's Law as a consequence of the least effort principle and theories on the nature of syntactical relationships BIBREF6, BIBREF7.
In a more practical scenario, text networks have been used in text classification tasks BIBREF8, BIBREF9, BIBREF10. The main advantage of the model is that it does not rely on deep semantical information to obtain competitive results. Another advantage of graph-based approaches is that, when combined with other approaches, it yields competitive results BIBREF11. A simple, yet recurrent text model is the well-known word co-occurrence network. After optional textual pre-processing steps, in a co-occurrence network each different word becomes a node and edges are established via co-occurrence in a desired window. A common strategy connects only adjacent words in the so called word adjacency networks.
While the co-occurrence representation yields good results in classification scenarios, some important features are not considered in the model. For example, long-range syntactical links, though less frequent than adjacent syntactical relationships, might be disregarded from a simple word adjacency approach BIBREF12. In addition, semantically similar words not sharing the same lemma are mapped into distinct nodes. In order to address these issues, here we introduce a modification of the traditional network representation by establishing additional edges, referred to as “virtual” edges. In the proposed model, in addition to the co-occurrence edges, we link two nodes (words) if the corresponding word embedding representation is similar. While this approach still does not merge similar nodes into the same concept, similar nodes are explicitly linked via virtual edges.
Our main objective here is to evaluate whether such an approach is able to improve the discriminability of word co-occurrence networks in a typical text network classification task. We evaluate the methodology for different embedding techniques, including GloVe, Word2Vec and FastText. We also investigated different thresholding strategies to establish virtual links. Our results revealed, as a proof of principle, that the proposed approach is able to improve the discriminability of the classification when compared to the traditional co-occurrence network. While the gain in performance depended upon the text length being considered, we found relevant gains for intermediary text lengths. Additional results also revealed that a simple thresholding strategy combined with the use of stopwords tends to yield the best results.
We believe that the proposed representation could be applied in other text classification tasks, which could lead to potential gains in performance. Because the inclusion of virtual edges is a simple technique to make the network denser, such an approach can benefit networked representations with a limited number of nodes and edges. This representation could also shed light into language mechanisms in theoretical studies relying on the representation of text as complex networks. Potential novel research lines leveraging the adopted approach to improve the characterization of texts in other applications are presented in the conclusion.
Complex networks have been used in a wide range of fields, including in Social Sciences BIBREF13, Neuroscience BIBREF14, Biology BIBREF15, Scientometry BIBREF16 and Pattern Recognition BIBREF17, BIBREF18, BIBREF19, BIBREF20. In text analysis, networks are used to uncover language patterns, including the origins of the ever present Zipf's Law BIBREF21 and the analysis of linguistic properties of natural and unknown texts BIBREF22, BIBREF23. Applications of network science in text mining and text classification encompasses applications in semantic analysis BIBREF24, BIBREF25, BIBREF26, BIBREF27, authorship attribution BIBREF28, BIBREF29 and stylometry BIBREF28, BIBREF30, BIBREF31. Here we focus in the stylometric analysis of texts using complex networks.
In BIBREF28, the authors used a co-occurrence network to study a corpus of English and Polish books. They considered a dataset of 48 novels, which were written by 8 different authors. Differently from traditional co-occurrence networks, some punctuation marks were considered as words when mapping texts as networks. The authors also decided to create a methodology to normalize the obtained network metrics, since they considered documents with variations in length. A similar approach was adopted in a similar study BIBREF32, with a focus on comparing novel measurements and measuring the effect of considering stopwords in the network structure.
A different approach to analyze co-occurrence networks was devised in BIBREF33. Whilst most approaches only considered traditional network measurements or devised novel topological and dynamical measurements, the authors combined networked and semantic information to improve the performance of network-based classification. Interesting, the combined use of network motifs and node labels (representing the corresponding words) allowed an improvement in performance in the considered task. A similar combination of techniques using a hybrid approach was proposed in BIBREF8. Networked-based approaches has also been applied to the authorship recognition tasks in other languages, including Persian texts BIBREF9.
Co-occurrence networks have been used in other contexts other than stylometric analysis. The main advantage of this approach is illustrated in the task aimed at diagnosing diseases via text analysis BIBREF11. Because the topological analysis of co-occurrence language networks do not require deep semantic analysis, this model is able to model text created by patients suffering from cognitive impairment BIBREF11. Recently, it has been shown that the combination of network and traditional features could be used to improve the diagnosis of patients with cognitive impairment BIBREF11. Interestingly, this was one of the first approaches suggesting the use of embeddings to address the particular problem of lack of statistics to create a co-occurrence network in short documents BIBREF34.
While many of the works dealing with word co-occurrence networks have been proposed in the last few years, no systematic study of the effects of including information from word embeddings in such networks has been analyzed. This work studies how links created via embeddings information modify the underlying structure of networks and, most importantly, how it can improve the model to provide improved classification performance in the stylometry task.
To represent texts as networks, we used the so-called word adjacency network representation BIBREF35, BIBREF28, BIBREF32. Typically, before creating the networks, the text is pre-processed. An optional pre-processing step is the removal of stopwords. This step is optional because such words include mostly article and prepositions, which may be artlessly represented by network edges. However, in some applications – including the authorship attribution task – stopwords (or function words) play an important role in the stylistic characterization of texts BIBREF32. A list of stopwords considered in this study is available in the Supplementary Information.
The pre-processing step may also include a lemmatization procedure. This step aims at mapping words conveying the same meaning into the same node. In the lemmatization process, nouns and verbs are mapped into their singular and infinite forms. Note that, while this step is useful to merge words sharing a lemma into the same node, more complex semantical relationships are overlooked. For example, if “car” and “vehicle” co-occur in the same text, they are considered as distinct nodes, which may result in an inaccurate representation of the text.
Such a drawback is addressed by including “virtual” edges connecting nodes. In other words, even if two words are not adjacent in the text, we include “virtual” edges to indicate that two distant words are semantically related. The inclusion of such virtual edges is illustrated in Figure FIGREF1. In order to measure the semantical similarity between two concepts, we use the concept of word embeddings BIBREF36, BIBREF37. Thus, each word is represented using a vector representation encoding the semantical and contextual characteristics of the word. Several interesting properties have been obtained from distributed representation of words. One particular property encoded in the embeddings representation is the fact the semantical similarity between concepts is proportional to the similarity of vectors representing the words. Similarly to several other works, here we measure the similarity of the vectors via cosine similarity BIBREF38.
The following strategies to create word embedding were considered in this paper:
GloVe: the Global Vectors (GloVe) algorithm is an extension of the Word2vec model BIBREF39 for efficient word vector learning BIBREF40. This approach combines global statistics from matrix factorization techniques (such as latent semantic analysis) with context-based and predictive methods like Word2Vec. This method is called as Global Vector method because the global corpus statistics are captured by GloVe. Instead of using a window to define the local context, GloVe constructs an explicit word-context matrix (or co-occurrence matrix) using statistics across the entire corpus. The final result is a learning model that oftentimes yields better word vector representations BIBREF40.
Word2Vec: this is a predictive model that finds dense vector representations of words using a three-layer neural network with a single hidden layer BIBREF39. It can be defined in a two-fold way: continuous bag-of-words and skip-gram model. In the latter, the model analyzes the words of a set of sentences (or corpus) and attempts to predict the neighbors of such words. For example, taking as reference the word “Robin”, the model decides that “Hood” is more likely to follow the reference word than any other word. The vectors are obtained as follows: given the vocabulary (generated from all corpus words), the model trains a neural network with the sentences of the corpus. Then, for a given word, the probabilities that each word follows the reference word are obtained. Once the neural network is trained, the weights of the hidden layer are used as vectors of each corpus word.
FastText: this method is another extension of the Word2Vec model BIBREF41. Unlike Word2Vec, FastText represents each word as a bag of character n-grams. Therefore, the neural network not only trains individual words, but also several n-grams of such words. The vector for a word is the sum of vectors obtained for the character n-grams composing the word. For example, the embedding obtained for the word “computer” with $n\le 3$ is the sum of the embeddings obtained for “co”, “com”, “omp”, “mpu”, “put”, “ute”, “ter” and “er”. In this way, this method obtains improved representations for rare words, since n-grams composing rare words might be present in other words. The FastText representation also allows the model to understand suffixes and prefixes. Another advantage of FastText is its efficiency to be trained in very large corpora.
Concerning the thresholding process, we considered two main strategies. First, we used a global strategy: in addition to the co-occurrence links (continuous lines in Figure FIGREF1), only “virtual” edges stronger than a given threshold are left in the network. Thus only the most similar concepts are connected via virtual links. This strategy is hereafter referred to as global strategy. Unfortunately, this method may introduce an undesired bias towards hubs BIBREF42.
To overcome the potential disadvantages of the global thresholding method, we also considered a more refined thresholding approach that takes into account the local structure to decide whether a weighted link is statistically significant BIBREF42. This method relies on the idea that the importance of an edge should be considered in the the context in which it appears. In other words, the relevance of an edge should be evaluated by analyzing the nodes connected to its ending points. Using the concept of disparity filter, the method devised in BIBREF42 defines a null model that quantifies the probability of a node to be connected to an edge with a given weight, based on its other connections. This probability is used to define the significance of the edge. The parameter that is used to measure the significance of an edge $e_{ij}$ is $\alpha _{ij}$, defined as:
where $w_{ij}$ is the weight of the edge $e_{ij}$ and $k_i$ is the degree of the $i$-th node. The obtained network corresponds to the set of nodes and edges obtained by removing all edges with $\alpha $ higher than the considered threshold. Note that while the similarity between co-occurrence links might be considered to compute $\alpha _{ij}$, only “virtual” edges (i.e. the dashed lines in Figure FIGREF1) are eligible to be removed from the network in the filtering step. This strategy is hereafter referred to as local strategy.
After co-occurrence networks are created and virtual edges are included, in the next step we used a characterization based on topological analysis. Because a global topological analysis is prone to variations in network size, we focused our analysis in the local characterization of complex networks. In a local topological analysis, we use as features the value of topological/dynamical measurements obtained for a set of words. In this case, we selected as feature the words occurring in all books of the dataset. For each word, we considered the following network measurements: degree, betweenness, clustering coefficient, average shortest path length, PageRank, concentric symmetry (at the second and third hierarchical level) BIBREF32 and accessibility BIBREF43, BIBREF44 (at the second and third hierarchical level). We chose these measurements because all of them capture some particular linguistic feature of texts BIBREF45, BIBREF46, BIBREF47, BIBREF48. After network measurements are extracted, they are used in machine learning algorithms. In our experiments, we considered Decision Trees (DT), nearest neighbors (kNN), Naive Bayes (NB) and Support Vector Machines (SVM). We used some heuristics to optimize classifier parameters. Such techniques are described in the literature BIBREF49. The accuracy of the pattern recognition methods were evaluated using cross-validation BIBREF50.
In summary, the methodology used in this paper encompasses the following steps:
Network construction: here texts are mapped into a co-occurrence networks. Some variations exists in the literature, however here we focused in the most usual variation, i.e. the possibility of considering or disregarding stopwords. A network with co-occurrence links is obtained after this step.
Network enrichment: in this step, the network is enriched with virtual edges established via similarity of word embeddings. After this step, we are given a complete network with weighted links. Virtually, any embedding technique could be used to gauge the similarity between nodes.
Network filtering: in order to eliminate spurious links included in the last step, the weakest edges are filtered. Two approaches were considered: a simple approach based on a global threshold and a local thresholding strategy that preserves network community structure. The outcome of this network filtering step is a network with two types of links: co-occurrence and virtual links (as shown in Figure FIGREF1).
Feature extraction: In this step, topological and dynamical network features are extracted. Here, we do not discriminate co-occurrence from virtual edges to compute the network metrics.
Pattern classification: once features are extracted from complex networks, they are used in pattern classification methods. This might include supervised, unsupervised and semi-supervised classification. This framework is exemplified in the supervised scenario.
The above framework is exemplified with the most common technique(s). It should be noted that the methods used, however, can be replaced by similar techniques. For example, the network construction could consider stopwords or even punctuation marks BIBREF51. Another possibility is the use of different strategies of thresholding. While a systematic analysis of techniques and parameters is still required to reveal other potential advantages of the framework based on the addition of virtual edges, in this paper we provide a first analysis showing that virtual edges could be useful to improve the discriminability of texts modeled as complex networks.
Here we used a dataset compatible with datasets used recently in the literature (see e.g. BIBREF28, BIBREF10, BIBREF52). The objective of the studied stylometric task is to identify the authorship of an unknown document BIBREF53. All data and some statistics of each book are shown in the Supplementary Information.
In Section SECREF13, we probe whether the inclusion of virtual edges is able to improve the performance of the traditional co-occurrence network-based classification in a usual stylometry task. While the focus of this paper is not to perform a systematic analysis of different methods comprising the adopted network, we consider two variations in the adopted methodology. In Section SECREF19, we consider the use of stopwords and the adoption of a local thresholding process to establish different criteria to create new virtual edges.
In Figure FIGREF14, we show some of the improvements in performance obtained when including a fixed amount of virtual edges using GloVe as embedding method. In each subpanel, we show the relative improvement in performance obtained as a function of the fraction of additional edges. In this section, we considered the traditional co-occurrence as starting point. In other words, the network construction disregarded stopwords. The list of stopwords considered in this paper is available in the Supplementary Information. We also considered the global approach to filter edges.
The relative improvement in performance is given by $\Gamma _+{(p)}/\Gamma _0$, where $\Gamma _+{(p)}$ is the accuracy rate obtained when $p\%$ additional edges are included and $\Gamma _0 = \Gamma _+{(p=0)}$, i.e. $\Gamma _0$ is the accuracy rate measured from the traditional co-occurrence model. We only show the highest relative improvements in performance for each classifier. In our analysis, we considered also samples of text with distinct length, since the performance of network-based methods is sensitive to text length BIBREF34. In this figure, we considered samples comprising $w=\lbrace 1.0, 2.5, 5.0, 10.0\rbrace $ thousand words.
The results obtained for GloVe show that the highest relative improvements in performance occur for decision trees. This is apparent specially for the shortest samples. For $w=1,000$ words, the decision tree accuracy is enhanced by a factor of almost 50% when $p=20\%$. An excellent gain in performance is also observed for both Naive Bayes and SVM classifiers, when $p=18\%$ and $p=12\%$, respectively. When $w=2,500$ words, the highest improvements was observed for the decision tree algorithm. A minor improvement was observed for the kNN method. A similar behavior occurred for $w=5,000$ words. Interestingly, SVM seems to benefit from the use of additional edges when larger documents are considered. When only 5% virtual edges are included, the relative gain in performance is about 45%.
The relative gain in performance obtained for Word2vec is shown in Figure FIGREF15. Overall, once again decision trees obtained the highest gain in performance when short texts are considered. Similar to the analysis based on the GloVe method, the gain for kNN is low when compared to the benefit received by other methods. Here, a considerable gain for SVM in only clear for $w=2,500$ and $p=10\%$. When large texts are considered, Naive Bayes obtained the largest gain in performance.
Finally, the relative gain in performance obtained for FastText is shown in Figure FIGREF16. The prominent role of virtual edges in decision tree algorithm in the classification of short texts once again is evident. Conversely, the classification of large documents using virtual edges mostly benefit the classification based on the Naive Bayes classifier. Similarly to the results observed for Glove and Word2vec, the gain in performance obtained for kNN is low compared when compared to other methods.
While Figures FIGREF14 – FIGREF16 show the relative behavior in the accuracy, it still interesting to observe the absolute accuracy rate obtained with the classifiers. In Table TABREF17, we show the best accuracy rate (i.e. $\max \Gamma _+ = \max _p \Gamma _+(p)$) for GloVe. We also show the average difference in performance ($\langle \Gamma _+ - \Gamma _0 \rangle $) and the total number of cases in which an improvement in performance was observed ($N_+$). $N_+$ ranges in the interval $0 \le N_+ \le 20$. Table TABREF17 summarizes the results obtained for $w = \lbrace 1.0, 5.0, 10.0\rbrace $ thousand words. Additional results for other text length are available in Tables TABREF28–TABREF30 of the Supplementary Information.
In very short texts, despite the low accuracy rates, an improvement can be observed in all classifiers. The best results was obtained with SVM when virtual edges were included. For $w=5,000$ words, the inclusion of new edges has no positive effect on both kNN and Naive Bayes algorithms. On the other hand, once again SVM could be improved, yielding an optimized performance. For $w=10,000$ words, SVM could not be improved. However, even without improvement it yielded the maximum accuracy rate. The Naive Bayes algorithm, in average, could be improved by a margin of about 10%.
The results obtained for Word2vec are summarized in Table TABREF29 of the Supplementary Information. Considering short documents ($w=1,000$ words), here the best results occurs only with the decision tree method combined with enriched networks. Differently from the GloVe approach, SVM does not yield the best results. Nonetheless, the highest accuracy across all classifiers and values of $p$ is the same. For larger documents ($w=5,000$ and $w=10,000$ words), no significant difference in performance between Word2vec and GloVe is apparent.
The results obtained for FastText are shown in Table TABREF18. In short texts, only kNN and Naive Bayes have their performance improved with virtual edges. However, none of the optimized results for these classifiers outperformed SVM applied to the traditional co-occurrence model. Conversely, when $w=5,000$ words, the optimized results are obtained with virtual edges in the SVM classifier. Apart from kNN, the enriched networks improved the traditional approach in all classifiers. For large chunks of texts ($w=10,000$), once again the approach based on SVM and virtual edges yielded optimized results. All classifiers benefited from the inclusion of additional edges. Remarkably, Naive Bayes improved by a margin of about $13\%$.
While in the previous section we focused our analysis in the traditional word co-occurrence model, here we probe if the idea of considering virtual edges can also yield optimized results in particular modifications of the framework described in the methodology. The first modification in the co-occurrence model is the use of stopwords. While in semantical application of network language modeling stopwords are disregarded, in other application it can unravel interesting linguistic patterns BIBREF10. Here we analyzed the effect of using stopwords in enriched networks. We summarize the obtained results in Table TABREF20. We only show the results obtained with SVM, as it yielded the best results in comparison to other classifiers. The accuracy rate for other classifiers is shown in the Supplementary Information.
The results in Table TABREF20 reveals that even when stopwords are considered in the original model, an improvement can be observed with the addition of virtual edges. However, the results show that the degree of improvement depends upon the text length. In very short texts ($w=1,000$), none of the embeddings strategy was able to improve the performance of the classification. For $w=1,500$, a minor improvement was observed with FastText: the accuracy increased from $\Gamma _0 = 37.18\%$ to $38.46\%$. A larger improvement could be observed for $w=2,000$. Both Word2vec and FastText approaches allowed an increase of more than 5% in performance. A gain higher than 10% was observed for $w=2,500$ with Word2vec. For larger pieces of texts, the gain is less expressive or absent. All in all, the results show that the use of virtual edges can also benefit the network approach based on stopwords. However, no significant improvement could be observed with very short and very large documents. The comparison of all three embedding methods showed that no method performed better than the others in all cases.
We also investigated if more informed thresholding strategies could provide better results. While the simple global thresholding approach might not be able to represent more complex structures, we also tested a more robust approach based on the local approach proposed by Serrano et al. BIBREF42. In Table TABREF21, we summarize the results obtained with this thresholding strategies. The table shows $\max \Gamma _+^{(L)} / \max \Gamma _+^{(G)}$, where $\Gamma _+^{(L)}$ and $\Gamma _+^{(G)}$ are the accuracy obtained with the local and global thresholding strategy, respectively. The results were obtained with the SVM classifier, as it turned to be the most efficient classification method. We found that there is no gain in performance when the local strategy is used. In particular cases, the global strategy is considerably more efficient. This is the case e.g. when GloVe is employed in texts with $w=1,500$ words. The performance of the global strategy is $12.2\%$ higher than the one obtained with the global method. A minor difference in performance was found in texts comprising $w=1,000$ words, yet the global strategy is still more efficient than the global one.
To summarize all results obtained in this study we show in Table TABREF22 the best results obtained for each text length. We also show the relative gain in performance with the proposed approach and the embedding technique yielding the best result. All optimized results were obtained with the use of stopwords, global thresholding strategy and SVM as classification algorithm. A significant gain is more evident for intermediary text lengths.
Textual classification remains one of the most important facets of the Natural Language Processing area. Here we studied a family of classification methods, the word co-occurrence networks. Despite this apparent simplicity, this model has been useful in several practical and theoretical scenarios. We proposed a modification of the traditional model by establishing virtual edges to connect nodes that are semantically similar via word embeddings. The reasoning behind this strategy is the fact the similar words are not properly linked in the traditional model and, thus, important links might be overlooked if only adjacent words are linked.
Taking as reference task a stylometric problem, we showed – as a proof of principle – that the use of virtual edges might improve the discriminability of networks. When analyzing the best results for each text length, apart from very short and long texts, the proposed strategy yielded optimized results in all cases. The best classification performance was always obtained with the SVM classifier. In addition, we found an improved performance when stopwords are used in the construction of the enriched co-occurrence networks. Finally, a simple global thresholding strategy was found to be more efficient than a local approach that preserves the community structure of the networks. Because complex networks are usually combined with other strategies BIBREF8, BIBREF11, we believe that the proposed could be used in combination with other methods to improve the classification performance of other text classification tasks.
Our findings paves the way for research in several new directions. While we probed the effectiveness of virtual edges in a specific text classification task, we could extend this approach for general classification tasks. A systematic comparison of embeddings techniques could also be performed to include other recent techniques BIBREF54, BIBREF55. We could also identify other relevant techniques to create virtual edges, allowing thus the use of the methodology in other networked systems other than texts. For example, a network could be enriched with embeddings obtained from graph embeddings techniques. A simpler approach could also consider link prediction BIBREF56 to create virtual edges. Finally, other interesting family of studies concerns the discrimination between co-occurrence and virtual edges, possibly by creating novel network measurements considering heterogeneous links.
The authors acknowledge financial support from FAPESP (Grant no. 16/19069-9), CNPq-Brazil (Grant no. 304026/2018-2). This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
The following words were considered as stopwords in our analysis: all, just, don't, being, over, both, through, yourselves, its, before, o, don, hadn, herself, ll, had, should, to, only, won, under, ours,has, should've, haven't, do, them, his, very, you've, they, not, during, now, him, nor, wasn't, d, did, didn, this, she, each, further, won't, where, mustn't, isn't, few, because, you'd, doing, some, hasn, hasn't, are, our, ourselves, out, what, for, needn't, below, re, does, shouldn't, above, between, mustn, t, be, we, who, mightn't, doesn't, were, here, shouldn, hers, aren't, by, on, about, couldn, of, wouldn't, against, s, isn, or, own, into, yourself, down, hadn't, mightn, couldn't, wasn, your, you're, from, her, their, aren, it's, there, been, whom, too, wouldn, themselves, weren, was, until, more, himself, that, didn't, but, that'll, with, than, those, he, me, myself, ma, weren't, these, up, will, while, ain, can, theirs, my, and, ve, then, is, am, it, doesn, an, as, itself, at, have, in, any, if, again, no, when, same, how, other, which, you, shan't, shan, needn, haven, after, most, such, why, a, off i, m, yours, you'll, so, y, she's, the, having, once.
The list of books is shown in Tables TABREF25 and TABREF26. For each book we show the respective authors (Aut.) and the following quantities: total number of words ($N_W$), total number of sentences ($N_S$), total number of paragraphs ($N_P$) and the average sentence length ($\langle S_L \rangle $), measured in number of words. The following authors were considered: Hector Hugh (HH), Thomas Hardy (TH), Daniel Defoe (DD), Allan Poe (AP), Bram Stoker (BS), Mark Twain (MT), Charles Dickens (CD), Pelham Grenville (PG), Charles Darwin (CD), Arthur Doyle (AD), George Eliot (GE), Jane Austen (JA), and Joseph Conrad (JC).
In this section we show additional results obtained for different text length. More specifically, we show the results obtained for GloVe, Word2vec and FastText when stopwords are either considered in the text or disregarded from the analysis.

Figure and table captions:
FIG. 1. Example of a enriched word co-occurrence network created for a text. In this model, after the removal of stopwords, the remaining words are linked whenever they appear in the same context. In the proposed network representation, “virtual” edges are included whenever two nodes (words) are semantically related. In this example, virtual edges are those represented by red dashed lines. Edges are included via embeddings similarity. The quantity of included edges is a parameter to be chosen.
FIG. 2. Gain in performance when considering additional virtual edges created using GloVe as embedding method. Each sub-panel shows the results obtained for distinct values of text length. In this case, the highest improvements in performance tends to occur in the shortest documents.
FIG. 3. Gain in performance when considering additional virtual edges created using Word2vec as embedding method. Each sub-panel shows the results obtained for distinct values of text length.
FIG. 4. Gain in performance when considering additional virtual edges created using FastText as embedding method. Each sub-panel shows the results obtained for distinct value of text length.
TABLE I. Statistics of performance obtained with GloVe for different text lengths. Additional results considering other text lengths are shown in the Supplementary Information. Γ0 is the the accuracy rate obtained with the traditional co-occurrence model and max Γ+ is the highest accuracy rate considering different number of additional virtual edges. 〈Γ+ − Γ0〉 is the average absolute improvement in performance, 〈Γ+/Γ0〉 is the average relative improvement in performance and N+ is the total number of cases in which an improvement in performance was observed. In total we considered 20 different cases, which corresponds to the addition of p = 1%, 2% . . . 20% additional virtual edges. The best result for each document length is highlighted.
TABLE II. Statistics of performance obtained with FastText for different text lengths. Additional results considering other text lengths are shown in the Supplementary Information. Γ0 is the the accuracy rate obtained with the traditional co-occurrence model and max Γ+ is the highest accuracy rate considering different number of additional virtual edges. 〈Γ+ − Γ0〉 is the average absolute improvement in performance, 〈Γ+/Γ0〉 is the average relative improvement in performance and N+ is the total number of cases in which an improvement in performance was observed. In total we considered 20 different cases, which corresponds to the addition of p = 1%, 2% . . . 20% additional virtual edges. The best result for each document length is highlighted.
TABLE III. Performance analysis of the adopted framework when considering stopwords in the construction of the networks. Only the best results obtained across all considered classifiers are shown. In this case, all optimized results were obtained with SVM. Γ0 corresponds to the accuracy obtained with no virtual edges and max Γ+ is the best accuracy rate obtained when including virtual edges. For each text length, the highest accuracy rate is highlighted. A full list of results for each classifier is available in the Supplementary Information.
TABLE IV. Comparison between the best results obtained via global and local thresholding. For each text length and embedding method, we show max Γ (L)
TABLE V. Summary of best results obtained in this paper. For each document length we show the highest accuracy rate obtained, the relative gain obtained with the proposed approach and the embedding method yielding the highest accuracy rate: GloVe (GL), Word2Vec (W2V) or FastText (FT). All the results below were obtained when stopwords were used and the SVM was used as classification method.

Questions about this paper:
Do the use word embeddings alone or they replace some previous features of the model with word embeddings?

Answers:
They use it as addition to previous model - they add new edge between words if word embeddings are similar.