Introduction and related work
Keywords are terms (i.e. expressions) that best describe the subject of a document BIBREF0 . A good keyword effectively summarizes the content of the document and allows it to be efficiently retrieved when needed. Traditionally, keyword assignment was a manual task, but with the emergence of large amounts of textual data, automatic keyword extraction methods have become indispensable. Despite a considerable effort from the research community, state-of-the-art keyword extraction algorithms leave much to be desired and their performance is still lower than on many other core NLP tasks BIBREF1 . The first keyword extraction methods mostly followed a supervised approach BIBREF2 , BIBREF3 , BIBREF4 : they first extract keyword features and then train a classifier on a gold standard dataset. For example, KEA BIBREF4 , a state of the art supervised keyword extraction algorithm is based on the Naive Bayes machine learning algorithm. While these methods offer quite good performance, they rely on an annotated gold standard dataset and require a (relatively) long training process. In contrast, unsupervised approaches need no training and can be applied directly without relying on a gold standard document collection. They can be further divided into statistical and graph-based methods. The former, such as YAKE BIBREF5 , BIBREF6 , KP-MINER BIBREF7 and RAKE BIBREF8 , use statistical characteristics of the texts to capture keywords, while the latter, such as Topic Rank BIBREF9 , TextRank BIBREF10 , Topical PageRank BIBREF11 and Single Rank BIBREF12 , build graphs to rank words based on their position in the graph. Among statistical approaches, the state-of-the-art keyword extraction algorithm is YAKE BIBREF5 , BIBREF6 , which is also one of the best performing keyword extraction algorithms overall; it defines a set of five features capturing keyword characteristics which are heuristically combined to assign a single score to every keyword. On the other hand, among graph-based approaches, Topic Rank BIBREF9 can be considered state-of-the-art; candidate keywords are clustered into topics and used as vertices in the final graph, used for keyword extraction. Next, a graph-based ranking model is applied to assign a significance score to each topic and keywords are generated by selecting a candidate from each of the top-ranked topics. Network-based methodology has also been successfully applied to the task of topic extraction BIBREF13 .
The method that we propose in this paper, RaKUn, is a graph-based keyword extraction method. We exploit some of the ideas from the area of graph aggregation-based learning, where, for example, graph convolutional neural networks and similar approaches were shown to yield high quality vertex representations by aggregating their neighborhoods' feature space BIBREF14 . This work implements some of the similar ideas (albeit not in a neural network setting), where redundant information is aggregated into meta vertices in a similar manner. Similar efforts were shown as useful for hierarchical subnetwork aggregation in sensor networks BIBREF15 and in biological use cases of simulation of large proteins BIBREF16 .
The main contributions of this paper are as follows. The notion of load centrality was to our knowledge not yet sufficiently exploited for keyword extraction. We show that this fast measure offers competitive performance to other widely used centralities, such as for example the PageRank centrality (used in BIBREF10 ). To our knowledge, this work is the first to introduce the notion of meta vertices with the aim of aggregating similar vertices, following similar ideas to the statistical method YAKE BIBREF5 , which is considered a state-of-the-art for the keyword extraction. Next, as part of the proposed RaKUn algorithm we extend the extraction from unigrams also to bigram and threegram keywords based on load centrality scores computed for considered tokens. Last but not least, we demonstrate how arbitrary textual corpora can be transformed into weighted graphs whilst maintaining global sequential information, offering the opportunity to exploit potential context not naturally present in statistical methods.
The paper is structured as follows. We first present the text to graph transformation approach (Section SECREF2 ), followed by the introduction of the RaKUn keyword extractor (Section SECREF3 ). We continue with qualitative evaluation (Section SECREF4 ) and quantitative evaluation (Section SECREF5 ), before concluding the paper in Section 6.
Transforming texts to graphs
We first discuss how the texts are transformed to graphs, on which RaKUn operates. Next, we formally state the problem of keyword extraction and discuss its relation to graph centrality metrics.
Representing text
In this work we consider directed graphs. Let INLINEFORM0 represent a graph comprised of a set of vertices INLINEFORM1 and a set of edges ( INLINEFORM2 ), which are ordered pairs. Further, each edge can have a real-valued weight assigned. Let INLINEFORM3 represent a document comprised of tokens INLINEFORM4 . The order in which tokens in text appear is known, thus INLINEFORM5 is a totally ordered set. A potential way of constructing a graph from a document is by simply observing word co-occurrences. When two words co-occur, they are used as an edge. However, such approaches do not take into account the sequence nature of the words, meaning that the order is lost. We attempt to take this aspect into account as follows. The given corpus is traversed, and for each element INLINEFORM6 , its successor INLINEFORM7 , together with a given element, forms a directed edge INLINEFORM8 . Finally, such edges are weighted according to the number of times they appear in a given corpus. Thus the graph, constructed after traversing a given corpus, consists of all local neighborhoods (order one), merged into a single joint structure. Global contextual information is potentially kept intact (via weights), even though it needs to be detected via network analysis as proposed next.
Improving graph quality by meta vertex construction
A naïve approach to constructing a graph, as discussed in the previous section, commonly yields noisy graphs, rendering learning tasks harder. Therefore, we next discuss the selected approaches we employ in order to reduce both the computational complexity and the spatial complexity of constructing the graph, as well as increasing its quality (for the given down-stream task).
First, we consider the following heuristics which reduce the complexity of the graph that we construct for keyword extraction: Considered token length (while traversing the document INLINEFORM0 , only tokens of length INLINEFORM1 are considered), and next, lemmatization (tokens can be lemmatized, offering spatial benefits and avoiding redundant vertices in the final graph). The two modifications yield a potentially “simpler” graph, which is more suitable and faster for mining.
Even if the optional lemmatization step is applied, one can still aim at further reducing the graph complexity by merging similar vertices. This step is called meta vertex construction. The motivation can be explained by the fact, that even similar lemmas can be mapped to the same keyword (e.g., mechanic and mechanical; normal and abnormal). This step also captures spelling errors (similar vertices that will not be handled by lemmatization), spelling differences (e.g., British vs. American English), non-standard writing (e.g., in Twitter data), mistakes in lemmatization or unavailable or omitted lemmatization step.
The meta-vertex construction step works as follows. Let INLINEFORM0 represent the set of vertices, as defined above. A meta vertex INLINEFORM1 is comprised of a set of vertices that are elements of INLINEFORM2 , i.e. INLINEFORM3 . Let INLINEFORM4 denote the INLINEFORM5 -th meta vertex. We construct a given INLINEFORM6 so that for each INLINEFORM7 , INLINEFORM8 's initial edges (prior to merging it into a meta vertex) are rewired to the newly added INLINEFORM9 . Note that such edges connect to vertices which are not a part of INLINEFORM10 . Thus, both the number of vertices, as well as edges get reduced substantially. This feature is implemented via the following procedure:
Meta vertex candidate identification. Edit distance and word lengths distance are used to determine whether two words should be merged into a meta vertex (only if length distance threshold is met, the more expensive edit distance is computed).
The meta vertex creation. As common identifiers, we use the stemmed version of the original vertices and if there is more than one resulting stem, we select the vertex from the identified candidates that has the highest centrality value in the graph and its stemmed version is introduced as a novel vertex (meta vertex).
The edges of the words entailed in the meta vertex are next rewired to the meta vertex.
The two original words are removed from the graph.
The procedure is repeated for all candidate pairs.
A schematic representation of meta vertex construction is shown in Figure FIGREF3 . The yellow and blue groups of vertices both form a meta vertex, the resulting (right) graph is thus substantially reduced, both with respect to the number of vertices, as well as the number of edges.
Keyword identification
Up to this point, we discussed how the graph used for keyword extraction is constructed. In this work, we exploit the notion of load centrality, a fast measure for estimating the importance of vertices in graphs. This metric can be defined as follows.
Qualitative evaluation
RaKUn can be used also for visualization of keywords in a given document or document corpus. A visualization of extracted keywords is applied to an example from wiki20 BIBREF19 (for dataset description see Section SECREF15 ), where we visualize both the global corpus graph, as well as a local (document) view where keywords are emphasized, see Figures FIGREF13 and FIGREF14 , respectively. It can be observed that the global graph's topology is far from uniform — even though we did not perform any tests of scale-freeness, we believe the constructed graphs are subject to distinct topologies, where keywords play prominent roles.
Quantitative evaluation
This section discusses the experimental setting used to validate the proposed RaKUn approach against state-of-the-art baselines. We first describe the datasets, and continue with the presentation of the experimental setting and results.
Datasets
For RaKUn evaluation, we used 14 gold standard datasets from the list of BIBREF5 , BIBREF6 , from which we selected datasets in English. Detailed dataset descriptions and statistics can be found in Table TABREF17 , while full statistics and files for download can be found online. Most datasets are from the domain of computer science or contain multiple domains. They are very diverse in terms of the number of documents—ranging from wiki20 with 20 documents to Inspec with 2,000 documents, in terms of the average number of gold standard keywords per document—from 5.07 in kdd to 48.92 in 500N-KPCrowd-v1.1—and in terms of the average length of the documents—from 75.97 in kdd to SemEval2017 with 8332.34.
Experimental setting
We adopted the same evaluation procedure as used for the series of results recently introduced by YAKE authors BIBREF6 . Five fold cross validation was used to determine the overall performance, for which we measured Precision, Recall and F1 score, with the latter being reported in Table TABREF24 . Keywords were stemmed prior to evaluation. As the number of keywords in the gold standard document is not equal to the number of extracted keywords (in our experiments INLINEFORM0 =10), in the recall we divide the correctly extracted keywords by the number of keywords parameter INLINEFORM1 , if in the gold standard number of keywords is higher than INLINEFORM2 .
Selecting default configuration. First, we used a dedicated run for determining the default parameters. The cross validation was performed as follows. For each train-test dataset split, we kept the documents in the test fold intact, whilst performing a grid search on the train part to find the best parametrization. Finally, the selected configuration was used to extract keywords on the unseen test set. For each train-test split, we thus obtained the number of true and false positives, as well as true and false negatives, which were summed up and, after all folds were considered, used to obtain final F1 scores, which served for default parameter selection. The grid search was conducted over the following parameter range Num keywords: 10, Num tokens (the number of tokens a keyword can consist of): Count threshold (minimum support used to determine potential bigram candidates): Word length difference threshold (maximum difference in word length used to determine whether a given pair of words shall be aggregated): INLINEFORM0 , Edit length difference (maximum edit distance allowed to consider a given pair of words for aggregation): INLINEFORM1 , Lemmatization: [yes, no].
Even if one can use the described grid-search fine-tunning procedure to select the best setting for individual datasets, we observed that in nearly all the cases the best settings were the same. We therefore selected it as the default, which can be used also on new unlabeled data. The default parameter setting was as follows. The number of tokens was set to 1, Count threshold was thus not needed (only unigrams), for meta vertex construction Word length difference threshold was set to 3 and Edit distance to 2. Words were initially lemmatized. Next, we report the results using these selected parameters (same across all datasets), by which we also test the general usefulness of the approach.
Results
The results are presented in Table TABREF24 , where we report on F1 with the default parameter setting of RaKUn, together with the results from related work, as reported in the github table of the YAKE BIBREF5 .
We first observe that on the selection of datasets, the proposed RaKUn wins more than any other method. We also see that it performs notably better on some of the datasets, whereas on the remainder it performs worse than state-of-the-art approaches. Such results demonstrate that the proposed method finds keywords differently, indicating load centrality, combined with meta vertices, represents a promising research venue. The datasets, where the proposed method outperforms the current state-of-the-art results are: 500N-KPCrowd-v1.1, Schutz2008, fao30 and wiki20. In addition, RaKUn also achieves competitive results on citeulike180. A look at the gold standard keywords in these datasets reveals that they contain many single-word units which is why the default configuration (which returns unigrams only) was able to perform so well.
Four of these five datasets (500N-KPCrowd-v1.1, Schutz2008, fao30, wiki20) are also the ones with the highest average number of keywords per document with at least 33.23 keywords per document, while the fifth dataset (citeulike180) also has a relatively large value (18.42). Similarly, four of the five well-performing datasets (Schutz2008, fao30, citeulike180, wiki20) include long documents (more than 3,900 words), with the exception being 500N-KPCrowd-v1.1. For details, see Table TABREF17 . We observe that the proposed RaKUn outperforms the majority of other competitive graph-based methods. For example, the most similar variants Topical PageRank and TextRank do not perform as well on the majority of the considered datasets. Furthermore, RaKUn also outperforms KEA, a supervised keyword learner (e.g., very high difference in performance on 500N-KPCrowd-v1.1 and Schutz2008 datasets), indicating unsupervised learning from the graph's structure offers a more robust keyword extraction method than learning a classifier directly.
Conclusions and further work
In this work we proposed RaKUn, a novel unsupervised keyword extraction algorithm which exploits the efficient computation of load centrality, combined with the introduction of meta vertices, which notably reduce corpus graph sizes. The method is fast, and performs well compared to state-of-the-art such as YAKE and graph-based keyword extractors. In further work, we will test the method on other languages. We also believe additional semantic background knowledge information could be used to prune the graph's structure even further, and potentially introduce keywords that are inherently not even present in the text (cf. BIBREF29 ). The proposed method does not attempt to exploit meso-scale graph structure, such as convex skeletons or communities, which are known to play prominent roles in real-world networks and could allow for vertex aggregation based on additional graph properties. We believe the proposed method could also be extended using the Ricci-Oliver BIBREF30 flows on weighted graphs.

Question: How are meta vertices computed?
Answer: Meta vertex candidate identification. Edit distance and word lengths distance are used to determine whether two words should be merged into a meta vertex (only if length distance threshold is met, the more expensive edit distance is computed)., The meta vertex creation. As common identifiers, we use the stemmed version of the original vertices and if there is more than one resulting stem, we select the vertex from the identified candidates that has the highest centrality value in the graph and its stemmed version is introduced as a novel vertex (meta vertex).

Question: How are graphs derived from a given text?
Answer: The given corpus is traversed, and for each element INLINEFORM6 , its successor INLINEFORM7 , together with a given element, forms a directed edge INLINEFORM8 . Finally, such edges are weighted according to the number of times they appear in a given corpus. Thus the graph, constructed after traversing a given corpus, consists of all local neighborhoods (order one), merged into a single joint structure. Global contextual information is potentially kept intact (via weights), even though it needs to be detected via network analysis

Question: In what sense if the proposed method interpretable?
Answer: Unanswerable