Wiki Entity Summarization Benchmark

## Abstract

Abstract Entity summarization aims to compute concise summaries for entities in knowledge graphs.
Existing datasets and benchmarks are often limited to a few hundred entities and discard graph structure in source knowledge graphs. This limitation is particularly pronounced when it comes to ground-truth summaries, where there exist only a few labeled summaries for evaluation and training.
We propose WikES (Wiki Entity Summarization Benchmark), a comprehensive benchmark comprising of entities, their summaries, and their connections. Additionally, WikES features a dataset generator to test entity summarization algorithms in different areas of the knowledge graph.
Importantly, our approach combines graph algorithms and NLP models, as well as different data sources such that WikES does not require human annotation, rendering the approach cost-effective and generalizable to multiple domains. Finally, WikES is scalable and capable of capturing the complexities of knowledge graphs in terms of topology and semantics. WikES features existing datasets for comparison. Empirical studies of entity summarization methods confirm the usefulness of our benchmark. Data, code, and models are available at: https://github.com/msorkhpar/wiki-entity-summarization .

## 1 Introduction

Knowledge Graphs (KGs) are a valuable information representation: interconnected networks of entities and their relationships enable machine reasoning to empower question answering [, ], recommender systems [], information retrieval []. KGs may comprise millions of entities representing real-world objects, concepts, or events.

Yet, the size and complexity of these KGs progressively expand, rendering it increasingly challenging to convey the essential information about an entity in a concise and meaningful way [, ]. This is where entity summarization becomes relevant. Entity summarization (ES) [] is the process of generating a concise and informative summary that captures the most salient aspects of the entity description, based on the information available in the KGs. In ES, the entity *description* refers to all the triples involving such an entity. For instance, Figure [1](#S1.F1) illustrates a set of relationships surrounding the entity Ellen Johnson Sirleaf in a KG, along with a possible summary for this entity. Extensive descriptions can overwhelm users and exceed the capacity of typical user interfaces, making it challenging to identify the most relevant triples. Entity summarization addresses this issue by computing an optimal compact summary for an entity, selecting a size-constrained subset of triples [].

Figure: Figure 1: KG subgraph of entity Ellen Johnson Sirleaf: arrows depict the subgraph of relationships to other entities, and labels indicate their roles. Selecting the bold edges as entity summaries of the most relevant triples may reduce information overload while concisely describing the entity.
Refer to caption: /html/2406.08435/assets/x1.png

Despite advances in entity summarization techniques [], the development and evaluation of these methods are hindered by a number of limitations in the benchmarks and datasets [, ].
The first limitation of the current benchmarks is the small dataset size, encompassing only a few hundred entities. Second, the generation of ground-truth summaries for testing mostly relies on expensive and lengthy manual annotation. Moreover, the dependence on a few human annotators often biases the data towards the annotators’ preferences and knowledge. Third, existing benchmarks often disregard the wealth of information in the knowledge graph structure.

To address the above limitations, we propose:

- •
Novel WikES benchmark for ES based on summaries and graphs from Wikidata and Wikipedia.
- •
Subgraph extraction method preserving the complexity of real-world KGs; subsampling using random walks and proportionally preserving node degrees, WikES captures the structure of the entities up to the second-hop neighborhood, thereby ensuring that the connections in WikES accurately reflect those in the source KG.
- •
Comprehensive summaries for *any* entity in the KG, ensuring that summaries are both relevant and contextually rich by deriving them directly from corresponding Wikipedia abstracts, minimizing human bias, as these abstracts are created and reviewed by several experts. In this manner, WikES is scalable, enabling it to generate large benchmark resources efficiently with high-quality annotation.
- •
Automatic entity summarization dataset generator allows for the creation of arbitrarily large datasets, encompassing various domains of knowledge.

## 2 Existing Datasets

Here, we review the existing datasets for entity summarization. Table [1](#S2.T1) provides an overview and statistics of the current datasets in this field. FACES and INFO datasets have a higher density than the entities in the Entity Summarization Benchmark (ESBM). It is also clear that LMDB and FACES are not connected graphs, that challenge graph-based learning methods where the information cannot easily propagate in disconnected networks. Specifically, FACES consists of 12 connected components, which complicates the learning process for graph embedding methods by limiting the richness of information that can be leveraged from the graph.

**Table 1: Entity summarization datasets in terms of number of entities $|\mathcal{V}|$, triples $|\mathcal{E}|$, number of ground-truth summaries (target entities), density as $\displaystyle|\mathcal{E}|/\binom{|\mathcal{V}|}{2}$, graph connectivity, number of components, sampling method to select entities and subgraph, and minimum / maximum node degree.**
| Metric | DBpedia (ESBM) | LMDB (ESBM) | FACES | INFO |
| --- | --- | --- | --- | --- |
| Entities ($|\mathcal{V}|$) | 2 721 | 1 853 | 1 379 | 1 410 |
| Relations ($|\mathcal{E}|$) | 4 436 | 2 148 | 2 152 | 2 019 |
| Target Entities | 125 | 50 | 50 | 100 |
| Density | 0.0005 | 0.0006 | 0.0011 | 0.0010 |
| Sampling method | Not specified | Not specified | Not specified | Not specified |
| Connected-graph | Yes | No | No | Yes |
| Num-comp | 1 | 2 | 12 | 1 |
| Min Degree | 1 | 1 | 1 | 1 |
| Max Degree | 125 | 208 | 88 | 100 |

We provide here a comprehensive description of each dataset or benchmark:

- •
ESBM []: The Entity Summarization Benchmark (ESBM) is the first benchmark to evaluate the performance of entity summarization methods. ESBM has three versions; v1.2 is the latest and most extensive version. This version comprises 175 entities, with 150 from DBpedia [] and 25 from LinkedMDB []. The summaries comprise triples selected by $30$ “researchers and students“ annotators. Each entity has exactly $6$ summaries. Despite encompassing two datasets, ESBM has several limitations. First, the entity sampling method is not explained. In particular, some triples in the neighborhood of the entity are missing in the datasets. Second, there are no connections among the entities in the neighborhood, nor any two-hop neighborhood. Third, the expertise and background of the annotators are not assessed nor disclosed.
Due to the expensive annotation process, the dataset size is small.
- •
FACES []
is a dataset from DBpedia (version 3.9) [auer2007dbpedia] and includes $50$ randomly selected entities, each with at least $17$ different types of relations. Similar to ESBM, the FACES ground-truth is also generated manually.
- •
INFO [] contains $100$ randomly selected entities from $10$ classes in DBpedia. It comprises two sets of ground-truth summaries, REF-E and REF-W. REF-E summaries comprise a selection of triples from five experts adhering to a 140-character limit, similar to typical Google search result snippets. REF-W summaries are obtained by one expert who reads the abstract sections of the respective entities on Wikipedia and selects neighboring entities that closely match the Wikipedia abstracts.
The number of ground-truth summaries per entity varies, as some experts evaluate multiple entities. This inconsistency complicates the evaluation process. The expertise of the annotators remains unspecified.

In contrast, our benchmark uses Wikidata to automatically map entities from Wikipedia to Wikidata. This automation allows us to efficiently generate summaries for any number of entities. Unlike previous work, we use the Wikipedia abstract as a summary instead of manual annotators. Each abstract is a collaboration of many users; as such, it should not introduce obvious biases. Additionally, with this process, we ensure high-quality and cost-effective summaries. Furthermore, we present the characteristics of our dataset in Table [3(c)](#A1.T3.st3). The WikES benchmark includes a larger number of entities and relations than existing datasets. It is a connected graph containing approximately 500 seed nodes. Further details regarding the specific characteristics of our dataset are provided in Section [3.4](#S3.SS4).

## 3 The WikES Benchmark

A *Knowledge Graph* $\mathcal{KG}=(\mathcal{V},\mathcal{R},\mathcal{T})$ is a directed multigraph consisting of entities $\mathcal{V}=\{v_{1},\dots,v_{n}\}$, relationships $\mathcal{R}$, and triples $\mathcal{T}\subseteq\mathcal{V}\times\mathcal{R}\times\mathcal{V}$. The set of edges $\mathcal{E}=\{(i,j)\mid v_{i},v_{j}\in\mathcal{V}\wedge\exists r\in\mathcal{R}\text{ s.t. }(v_{i},r,v_{j})\in\mathcal{T}\}$ contains pairs of nodes connected by a relationship.

The *$t$-hop neighborhood* $\mathcal{N}_{t}(v_{i})$ of node $v_{i}$ is the set of nodes reachable from $v_{i}$ within $t$ edges when ignoring edge directions.

A *summary* for an entity $v_{i}$ is a subset $\mathcal{S}(v_{i})\subseteq\Delta_{t}(v_{i})$ of triples from the $t$-description of $v_{i}$, where the $t$-*description* of an entity $v_{i}\in\mathcal{V}$ in a knowledge graph $\mathcal{KG}$ is the set $\Delta_{t}(v_{i})=\{(s,p,o)\in\mathcal{T}\mid s\in\mathcal{N}_{t}(v_{i})\vee o\in\mathcal{N}_{t}(v_{i})\}$ of triples in which one of the entities is in the $t$-hop neighborhood of $v_{i}$.

Entity summarization for an entity $v_{i}\in\mathcal{V}$ in a knowledge graph $\mathcal{KG}$ aims to find a summary $\mathcal{S}(v_{i})$ that maximizes some score among all possible summaries for $v_{i}$, i.e.,

$$ $\underset{\begin{subarray}{c}\mathcal{S}(v_{i})\subseteq\Delta_{t}(v_{i})\\ |\mathcal{S}(v_{i})|=k\end{subarray}}{\operatorname*{\arg\>\max}}{\mathrm{score}(\mathcal{S}(v_{i}))},$ (1) $$

### 3.1 Extracting Summaries from Wikidata using Wikipedia Abstracts

We extract summaries for each Wikidata item using Wikipedia abstracts and infoboxes. Each abstract is a joint effort of many users and experts, which ensures quality and accuracy. Leveraging Wikipedia, we avoid time-consuming manual annotation and enable the automatic generation of large-scale datasets.

Wikidata is a free and collaborative knowledge base that collects structured data to support Wikipedia and other Wikimedia projects. It includes descriptions and labels for entities. The descriptions offer in-depth details, while the labels serve as concise identifiers, facilitating efficient data retrieval and integration in subsequent steps.
We load all Wikidata items XML dump files published on 2023/05/01(^1^11[https://dumps.wikimedia.org/wikidatawiki/](https://dumps.wikimedia.org/wikidatawiki/)) as entities $\mathcal{V}$ alongside their properties as relationships $\mathcal{R}$ into a graph database(^2^22[https://neo4j.com](https://neo4j.com)). The result is a graph that connects all Wikidata items and statements. We include items if they (1) are not marked as redirects, (2) belong to the main Wikidata namespace, and (3) have an English label or description. Additionally, we load metadata for each Wikidata item and property, including labels and descriptions, into a relational database(^3^33[https://www.postgresql.org/](https://www.postgresql.org/)).

Wikipedia pages contain infoboxes, abstracts, page content, categories, references, and more. Links to other Wikipedia pages are referred to as mentions. We detect these mentions in the abstracts and infoboxes of Wikipedia pages to use them later for labeling the summaries in Wikidata.
We extract and load all the content from the XML dump files of Wikipedia pages, published on 2023/05/01(^4^44[https://dumps.wikimedia.org/enwiki/](https://dumps.wikimedia.org/enwiki/)), into a relational database under the same conditions as Wikidata: the pages must be in English and not redirected.

Summary annotation.
We annotate the summaries in Wikidata using the corresponding Wikipedia pages.
For each Wikipedia page corresponding to a Wikidata entity, we iterate through all connected Wikidata items using Wikidata properties. If a connected Wikidata item is mentioned in the Wikipedia abstract and infobox, we annotate the Wikidata item with the corresponding Wikidata property as part of the summary.

Wikidata is a directed multigraph, which means that each entity (Wikidata item) can be connected to another entity via multiple relations (Wikidata properties). Yet, links in Wikipedia are not labeled; as such, we need to select one of the relations for the summary. To annotate the correct Wikidata property as part of the summary, we employ the DistilBERT model []. DistilBERT is a fast and lightweight model with a reduced number of parameters compared to the original BERT model. This way, we can efficiently process large amounts of data while maintaining high-quality embeddings for accurate relation selection.

Concretely, we first embed the abstract of the Wikidata item for which we are generating summaries using DistilBERT. We then calculate the cosine similarity between the embedding of the abstract and the embeddings of each candidate relation. Finally, we add the relation with the highest cosine similarity to the abstract embedding to the summary.
This approach ensures that the most relevant Wikidata property is selected for the summary based on its semantic similarity to the Wikipedia abstract.

### 3.2 Capturing the Graph Structure

Here we introduce the WikES generator algorithm. The main idea is to sample a connected graph that preserves the original graph structure. To this end, we employ random walks [].

A random walk is a stochastic process defined as a sequence of steps, where the direction and magnitude of each step are determined by the random variable
$\mathrm{X}_{t+1}=\mathrm{X}_{t}+\mathrm{S}_{t}$
where $\mathrm{X}_{t}$ represents the position at time $t$, and $\mathrm{S}_{t}$ is the step taken from position $\mathrm{X}_{t}$.

The process is a Markov process, characterized by its memoryless property:

$$ $P(X_{t+1}=x|X_{t}=x_{t},X_{t-1}=x_{t-1},\dots,X_{0}=x_{0})=P(X_{t+1}=x|X_{t}=x_{t})$ (2) $$

In adapting this concept to our work, we redefine the number of random walks assigned to nodes based on their degrees, ensuring the distribution remains proportional to real data. This is achieved through logarithmic transformation and normalization. The logarithmic transformation is applied to reduce the impact of high-degree nodes and also low-degree nodes, making it more manageable for the random walk. Given a graph with node degrees $\{d_{1},d_{2},\ldots,d_{i}\}$, the log-transformed degree for node $i$ is $L_{i}=\log(d_{i})$. These values are then normalized:

$$ $N_{i}=\frac{L_{i}-\min(\{L\})}{\max(\{L\})-\min(\{L\})}$ (3) $$

where $N_{i}$ is the normalized logarithmic degree of node $i$. Finally, the number of random walks $R_{i}$ assigned to each node is:

$$ $R_{i}=\text{round}\left(\text{minRW}+N_{i}\times(\text{maxRW}-\text{minRW})\right)$ (4) $$

Here, minRW and maxRW are the user-defined minimum and maximum limits for random walks. This adaptation ensures that the random walks are proportional to the normalized logarithmic degree of each node, reflecting the true structure of the network.
For a small dataset we set $\text{minRW}=100$ and $\text{maxRW}=300$; for a medium dataset $\text{minRW}=150$ and $\text{maxRW}=600$; for a large dataset, $\text{minRW}=300$ and $\text{maxRW}=1800$. This ensures that the random walks are tailored to both the scale and the complexity of the dataset. Importantly, our approach can be used to extract further subgraphs at the scale needed for benchmarking in a given scenario.

Moreover, the random walk sampling process requires a set of seed nodes as a starting point. In our case, the seed nodes represent the target entities we are interested in. The seed nodes can be any Wikidata Item Identifier, Wikipedia title, or Wikipedia ID of the Wikipedia pages. We collect the seed nodes on the condition that they have at least $k$ (default $k=5$) common entities with the abstract section and the infobox in the Wikipedia pages. Therefore, this model is flexible, allowing you to choose any seed nodes from any domain as an input.
In the datasets that we generated, we collect seed nodes from []. This paper has published information about individuals from various domains. The authors collected data from multiple Wikipedia editions and Wikidata, using deduplication and cross-verification techniques to compile a database of 1.6 million individuals with English Wikipedia pages. The seed nodes that we use include actor, athletic, football, journalist, painter, player, politician, singer, sport, writer, lawyer, film, composer, novelist, poet, and screenwriter.
Using combinations of these seed nodes, we generate four sets of datasets, with each set having small, medium, and large versions. In Table [9](#A1.T9) in Section [A](#A1) in the supplementary material, we present the seed nodes and their proportions for each dataset and their corresponding train-test-val splits.

### 3.3 WikES Generator

We discuss how WikES is created, and how further benchmarks can be generated without the need for manual annotators.
Algorithm [1](#alg1) details the generator, which consists of the following steps.

Step1: Retrieve summaries of each seed node (explained in Section [3.1](#S3.SS1))

Step2: Expand the graph using the random walk method in Section [3.2](#S3.SS2). Set the random walk’s length $n$ (default $n=2$), which means it explores up to the $n$-hop neighborhood of each seed node.

Step3: Check if the graph is connected. If it is, done. If not, identify all disconnected components and sort them by size, from largest to smallest. In each iteration, connect smaller components to the largest component using $h$ connections. Utilize the shortest path method, selecting paths that are equal to or less than a minimum path length $l$. Continue connecting nodes from the smaller component to the larger one until $h$ nodes are connected. After each iteration, check graph connectivity again. If all components are connected to the largest component, the algorithm ends. Otherwise, re-sort components and increase $l$ by 1. Repeat until the graph is a single connected component.

Figure: Algorithm 1 WikES Generator

### 3.4 WikES Datasets

We generate three sizes for each of the four datasets, obtaining 12 datasets. We present their characteristics in Table [3(c)](#A1.T3.st3) in section  [A](#A1).
The number of entities in the small datasets ranges from approximately 70$k$ to 85$k$, and the number of relations ranges from around 120$k$ to 135$k$. In the medium datasets, the number of entities ranges from 100$k$ to 130$k$, and the number of relations ranges from 195$k$ to 220$k$. The number of entities in the large datasets ranges from approximately 185$k$ to 250$k$, and the number of relations ranges from around 397$k$ to 470$k$.
The average runtime for generating small graphs is approximately $128$ seconds; for medium-sized graphs, it is approximately $216$ seconds; and for large graphs, it is approximately $512$ seconds.
We construct the train-test-validation split for each dataset with $70\%$ for training, $15\%$ for testing, and $15\%$ for validation. Detailed information about the run time, as well as the number of nodes and relations for these splits, is available on our GitHub repository. All graphs in each train-test-validation splits are connected.

## 4 Empirical Evaluation

We study the quality of WikES using the following metrics:

F-Score. Let $\mathcal{S}_{m}$ the summary obtained by a summarization method and $\mathcal{S}_{h}$ the ground-truth summary. We compare $\mathcal{S}_{m}$ with $\mathcal{S}_{h}$ using the F1-score based on precision $P$ and recall $R$:

$$ $\displaystyle\mathrm{F}1=\frac{2{\cdot}P{\cdot}R}{P+R},\ \text{where}\ \mathrm{P}=\frac{|\mathcal{S}_{m}\cap\mathcal{S}_{h}|}{|\mathcal{S}_{m}|}\ \text{and}\ \mathrm{R}=\frac{|\mathcal{S}_{m}\cap\mathcal{S}_{h}|}{|\mathcal{S}_{h}|}$ (5) $$

The F1 score lies within [0,1]. High F1 indicates that $\mathcal{S}_{m}$ is closer to the ground-truth $\mathcal{S}_{h}$.

Mean Average Precision (MAP). This metric is particularly suitable for evaluating ranking tasks because it takes into account the order of the predicted triples. MAP calculates precision at each position $i$ in the predicted summary and averages these values over all relevant summary triples. It reflects both the relevance and the ranking quality of the predicted summaries. MAP, unlike F1-score, does not depend on a specific value of $k$. This makes it a robust metric for assessing how well a summarization method ranks the relevant triples.

$$ $\mathrm{MAP}=\frac{1}{N}\sum^{N}_{n=1}\dfrac{\sum^{|\mathcal{S}_{m}^{(n)}|}_{i=1}\begin{cases}\mathrm{Precision}@i(\mathcal{S}_{h}^{(n)})&\mathrm{if}\ \mathrm{Rel}(n,i)\\ 0&\mathrm{otherwise}\end{cases}}{|\mathcal{S}_{h}^{(n)}|}$ (6) $$

where $N$ is the total number of entities, $\mathcal{S}_{h}^{(n)}$ is the set of ground-truth summary triples for a particular entity $v_{n}$, $\mathcal{S}_{m}^{(n)}$ is the set of predicted summary triples for the entity $v_{n}$, $\mathrm{Precision}@i$ is the precision at the $i$-th position in the predicted summary, and $\mathrm{Rel}(n,i)$ indicates whether the $i$-th predicted triple for entity $v_{n}$ is relevant (i.e., it belongs to $\mathcal{S}_{h}^{(n)}$). MAP scores are in the range [0,1], where a higher MAP indicates better performance in terms of correctly predicting relevant summary triples. To account for the varying lengths of the ground-truth summaries in real-world data, we also calculate MAP and F-score (which we refer to as dynamic MAP and dynamic F-score) by setting the length of the generated summary ($|\mathcal{S}_{m}|$) equal to the length of the corresponding ground-truth summary ($|\mathcal{S}_{h}|$).

We analyze our dataset and compare it with the ESBM benchmark using statistical measures such as frequency and inverse frequency of entities and relations. We calculate the F-score and MAP score for the top-5 and top-10 of both the ESBM dataset and our WikiProFem. We choose top-5 and top-10 because we only have ground-truth summaries for top-5 and top-10 in the ESBM dataset.
The F-score and MAP results for ESBM are presented in Figure [2](#S4.F2). The statistics show that for DBpedia, the F-score using inverse relation frequency outperforms the random baseline by 0.15 for top-5 and by 0.34 for top-10. Furthermore, when using inverse entity frequency, DBpedia achieves an even higher F-score, surpassing the random baseline by 0.07 for top-5 and by 0.15 for top-10.
For LMDB, we observe a similar trend when using inverse frequency. The F-score surpasses the random baseline by 0.10 for top-5 and by approximately 0.15 for top-10. Additionally, when employing entity frequency, LMDB achieves an F-score that is around 0.17 higher than the baseline for top-5 and 0.07 higher for top-10.
The results demonstrate that ESBM exhibits a strong bias towards entity, reverse entity, and relation frequency. For Map score, we are exactly observing the same behavior for ESBM. We believe that the bias comes from the fact that the datasets are small, their second-hop neighborhood is not considered, and the relations between their first-hop neighbors are not considered.
On the other hand, Figure [11](#A3.F11) shows the F-score for top-$5$, top-$10$ and dynamic F-score on WikES. Since the length of summaries varies with the abstract, we calculate the F-score for each seed node based on its summary length. Results show that WikES F-score is close to random for different statistics, thus rejecting the hypothesis of obvious biases. We observe a minor bias towards node frequency in small datasets. Yet, as we increase the size of the dataset, this bias disappears. We observe a similar behavior with MAP in Figure [12](#A3.F12)
Furthermore, we use *the entire* Wikidata to measure the F-score for our seed nodes. Thus, importantly, we observe that our dataset’s F-score trend is comparable to that of the entire data, especially our large dataset. We also extracted the first-hop neighborhood of all our seed nodes and observed a small bias in the F-score top-5 and dynamic F-score. We conclude that adding the two-hop neighborhood makes the sample follow the graph distribution.
Thus, WikES is an unbiased benchmark that retains the source KG distribution.

Figure: Figure 2: F1 score and MAP for frequency statistics on ESBM datasets.

Figure: Figure 3: F1 for frequency statistics on WikiProFem.

Figure: Figure 4: MAP for frequency statistics on WikiProFem.

We evaluate the performance of different entity summarization methods on our benchmark, and provide all implementations in the WikES GitHub repository.

- •
PageRank [] ranks nodes in a graph based on the structure of incoming links, with the idea that more important nodes are likely to receive more links from other nodes.
- •
RELIN [] is a weighted PageRank algorithm that evaluates the relevance of triples within a graph structure. We have re-implemented this model according to the specifications in the referenced paper. On our smaller dataset version, RELIN takes approximately 6 hours to compute all summaries.
- •
LinkSum [] is a two-step, relevance-centric method that combines PageRank with an adaptation of the Backlink algorithm to identify relevant connected entities. We have re-implemented it according to the paper. The LinkSum method initially takes 10 hours to compute the backlinks for each node in the small version of our dataset. By parallelizing the implementation, we reduced this to one hour. Additionally, the Backlink algorithm itself initially takes 100 minutes, but with parallelization, this was reduced to 10 minutes for the small version of our dataset.

Due to the inefficiency of the methods, we use a smaller version of WikES for evaluation. The results in Table [2](#S4.T2) show that LinkSum outperforms both RELIN and PageRank. These findings suggest that models capable of exploiting the graph structure while handling large-scale datasets and maintaining high accuracy in entity summarization are valuable for such real-world KGs, such as WikES.

**Table 2: Performance comparison of entity summarization models on the small version of WikES. The models are evaluated with different topK values (5 and 10) and a dynamic setting.**
|  |  | topK = 5 | topK = 10 | Dynamic |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Dataset | F-Score | MAP | F-Score | MAP | F-Score | MAP |
| PageRank | WikiLitArt | 0.024 | 0.01 | 0.081 | 0.02 | 0.175 | 0.046 |
|  | WikiCinema | 0.003 | 0.001 | 0.041 | 0.005 | 0.146 | 0.028 |
|  | WikiPro | 0.060 | 0.02 | 0.169 | 0.049 | 0.288 | 0.109 |
|  | WikiProFem | 0.032 | 0.01 | 0.093 | 0.024 | 0.145 | 0.036 |
| RELIN | WikiLitArt | 0.093 | 0.035 | 0.148 | 0.054 | 0.208 | 0.080 |
|  | WikiCinema | 0.071 | 0.023 | 0.127 | 0.038 | 0.209 | 0.068 |
|  | WikiPro | 0.125 | 0.053 | 0.200 | 0.086 | 0.273 | 0.127 |
|  | WikiProFem | 0.111 | 0.050 | 0.179 | 0.081 | 0.219 | 0.095 |
| LinkSum | WikiLitArt | 0.184 | 0.080 | 0.239 | 0.109 | 0.225 | 0.127 |
|  | WikiCinema | 0.119 | 0.048 | 0.152 | 0.060 | 0.135 | 0.068 |
|  | WikiPro | 0.249 | 0.127 | 0.347 | 0.190 | 0.350 | 0.242 |
|  | WikiProFem | 0.195 | 0.097 | 0.236 | 0.127 | 0.213 | 0.136 |

## 5 Conclusion

We introduce WikES (Wiki Entity Summarization Benchmark), a benchmark for KG entity summarization which provides a scalable dataset generator that eschews the need for costly human annotation.
WikES uses Wikipedia abstracts for automatic summary generation, ensuring contextually rich and unbiased summaries. It preserves the complexity and integrity of real-world KGs through a random walk sampling method that captures the structure of entities down to their second-hop neighborhoods.
Empirical evaluations demonstrate that WikES provides high-quality large-scale datasets for entity summarization tasks, and that it captures the complexities of knowledge graphs in terms of topology, making it a valuable resource for evaluating and improving entity summarization algorithms.

## Appendix A Appendix

**Table 3: Generated Datasets in terms of number of entities $|\mathcal{V}|$, triples $|\mathcal{E}|$, ground-truth summaries, density as $\displaystyle|\mathcal{E}|/\binom{|\mathcal{V}|}{2}$, graph connectivity, number of components, sampling method to select the entities and the subgraph, minimum and maximum node degree and, running time.**
| Metric | WikiLitArt | WikiCinema | WikiPro | WikiProFem |
| --- | --- | --- | --- | --- |
| Entities ($|\mathcal{V}|$) | 85 346 | 70 753 | 79 825 | 79 926 |
| Relations ($|\mathcal{E}|$) | 136 950 | 126 915 | 125 912 | 123 193 |
| Target Entities | 494 | 493 | 493 | 468 |
| Density | 0.000018 | 0.000018 | 0.000019 | 0.000019 |
| Sampling method | Random Walk | Random Walk | Random Walk | Random Walk |
| Connected-graph | Yes | Yes | Yes | Yes |
| Num-comp | 1 | 1 | 1 | 1 |
| Min Degree | 1 | 1 | 1 | 1 |
| Max Degree | 2172 | 3005 | 2060 | 3142 |
| Run-time (seconds) | 91.934 | 118.014 | 126.119 | 177.63 |

### A.1 Additional WikES details

- •
Dataset and Metadata: The dataset is available at [https://github.com/msorkhpar/wiki-entity-summarization](https://github.com/msorkhpar/wiki-entity-summarization). We generate four datasets in three sizes: small, medium, and large. Each size has a train-test-validation split. In Table [9](#A1.T9), we provide information on the proportion of seed nodes in each of the datasets. Moreover, Table [3(c)](#A1.T3.st3) provides detailed information such as the number of entities $|\mathcal{V}|$, triples $|\mathcal{E}|$, ground-truth summaries, density, graph connectivity, number of components, sampling method used to select the entities and the subgraph, minimum and maximum node degree, and running time for each of the datasets.
Moreover, metadata is also in the same github repository.
- •
WikES toolkits: We offer a comprehensive toolkit designed to facilitate working with our datasets. This toolkit includes features for downloading, loading, and manipulating pre-generated graph datasets. You can access our toolkit at [https://pypi.org/project/wikes-toolkit/](https://pypi.org/project/wikes-toolkit/)
- •
Dataset Formats: We generate our dataset in CSV format. The entity files are formatted according to Table [4](#A1.T4). Additionally, we provide a file that contains the categories of the target entities, as detailed in Table [8](#A1.T8). The predicate files, described in Table [6](#A1.T6), contain predicates along with their corresponding labels and descriptions. The triple file, presented in Table [7](#A1.T7), includes the subject, object, and predicate IDs of the nodes (Wikidata items) and edges (Wikidata predicates). The ground-truth file, shown in Table [8](#A1.T8), contains the subject, object, and predicate. Predicates in groun-truth entities are also marked as ground-truth. Moreover, we provide the graph version of our dataset in GraphML and PKL formats in our release [https://github.com/msorkhpar/wiki-entity-summarization/releases/tag/1.0.5](https://github.com/msorkhpar/wiki-entity-summarization/releases/tag/1.0.5).
- •
URL to metadata record: Since we have several datasets, in our GitHub Repository, we provide the Crossiant URL metadata. The metadata for WikiLitArt small dataset is [https://github.com/msorkhpar/wiki-entity-summarization/releases/download/1.0.5/WikiLitArt-m.json](https://github.com/msorkhpar/wiki-entity-summarization/releases/download/1.0.5/WikiLitArt-m.json). The metadata format is consistent across all datasets. The metadata files are included alongside the datasets in the GitHub release.
- •
Preprocessing URL: You can find our preprocessing code for cleaning and preparing Wikipedia and Wikidata at the following link: [https://github.com/msorkhpar/wiki-entity-summarization-preprocessor](https://github.com/msorkhpar/wiki-entity-summarization-preprocessor).
Access and loading of preprocessed:
–
Neo4j database: [https://github.com/msorkhpar/wiki-entity-summarization-preprocessor/releases/tag/Neo4j-1.0.0](https://github.com/msorkhpar/wiki-entity-summarization-preprocessor/releases/tag/Neo4j-1.0.0).
–
PostgreSQL: [https://github.com/msorkhpar/wiki-entity-summarization-preprocessor/releases/tag/PostgreSQL-1.0.0](https://github.com/msorkhpar/wiki-entity-summarization-preprocessor/releases/tag/PostgreSQL-1.0.0).
- •
Authors responsibility statement and License: The authors are held responsible for copyright infringement, but assume no responsibility or liability for any misuse of the data. This project is licensed under the CC BY 4.0 License. See here [https://github.com/msorkhpar/wiki-entity-summarization/blob/main/LICENSE](https://github.com/msorkhpar/wiki-entity-summarization/blob/main/LICENSE)
- •
WikES Generator Code: The code for running the WikES generator is available in the GitHub repository at [https://github.com/msorkhpar/wiki-entity-summarization](https://github.com/msorkhpar/wiki-entity-summarization). The code allows to generate the same datasets in the paper or create your own custom datasets.
- •
Maintenance and Long Term Preservation The authors of WikES are dedicated to the ongoing maintenance and preservation of this dataset. This includes tracking and resolving issues identified by the community post-release. We will closely monitor user feedback through the GitHub issue tracker. The data is hosted on GitHub, ensuring reliable and stable storage.
- •
Intended users: The intended users are NLP and knowledge graph researchers who wish to generate summaries using the textual information of the entities (nodes) in knowledge graphs. The suitable use case for this dataset is evaluating entity summarization models to determine their ability to accurately detect summaries.

**Table 4: {variant}-{size}-{dataset_type}-entities.csv file contains entities. An entity is a Wikidata item (node) in our dataset. variant_index refers to the dataset id (detailed information is in our Github).**
| Field | Description | Datatype |
| --- | --- | --- |
| id | Incremental integer starting by zero | int |
| entity | Wikidata qid, e.g. ‘Q76‘ | string |
| wikidata_label | Wikidata label (nullable) | string |
| wikidata_desc | Wikidata description (nullable) | string |
| wikipedia_title | Wikipedia title (nullable) | string |
| wikipedia_id | Wikipedia page id (nullable) | long |

**Table 5: {variant}-{size}-{dataset_type}-root-entities.csv contains root entities. A root entity is a seed node described previously.variant_index refers to the dataset id (detailed infomation is in our Github).**
| Field | Description | Datatype |
| --- | --- | --- |
| entity | id key in Table [4](#A1.T4) | int |
| category | category | string |

**Table 6: {variant}-{size}-{dataset_type}-predicates.csv contains predicates. A predicate is a Wikidata property or a describing a connection. variant_index refers to the dataset id (detailed information is in our Github).**
| Field | Description | Datatype |
| --- | --- | --- |
| id | Incremental integer starting by zero | int |
| predicate | Wikidata Property id, e.g. ‘P121‘ | string |
| predicate_label | Wikidata Property label (nullable) | string |
| predicate_desc | Wikidata Property description (nullable) | string |

**Table 7: {variant}-{size}-{dataset_type}-triples.csv contains triples. A triple is an edge between two entities with a predicate. variant_index refers to the dataset id (detailed information is in our Github).**
| Field | Description | Datatype |
| --- | --- | --- |
| subject | id key in Table [4](#A1.T4) | int |
| predicate | id key in Table [6](#A1.T6) | int |
| object | id key in Table [4](#A1.T4) | int |

**Table 8: {variant}-{size}-{dataset_type}-ground-truths.csv contains ground truth triples. A ground truth triple is an edge marked as a summary for a root entity.**
| Field | Description | Datatype |
| --- | --- | --- |
| root_entity | entity in Table [4](#A1.T4) | int |
| subject | id key in Table [4](#A1.T4) | int |
| predicate | id key in Table [6](#A1.T6) | int |
| object | id key in Table [4](#A1.T4) | int |

**Table 9: Seed nodes categories for each dataset. "Entire graph" refers to using the seed nodes and generating the data without train-test-val splits. In train-test-val, each of the datasets is a single weakly connected graph.**
| Dataset | Seed Nodes Categories |
| --- | --- |
| WikiLitArt | Entire graph: actor=150, composer=35, film=41, novelist=24, painter=59, poet=39, screenwriter=17, singer=72, writer=57 |
| Train: actor=105, composer=24, film=29, novelist=17, painter=42, poet=27, screenwriter=12, singer=50, writer=40 |  |
| Val: actor=23, composer=5, film=6, novelist=4, painter=9, poet=6, screenwriter=2, singer=11, writer=8 |  |
| Test: actor=22, composer=6, film=6, novelist=3, painter=8, poet=6, screenwriter=3, singer=11, writer=9 |  |
| WikiCinema | Entire graph: actor=405, film=88 |
| Train: actor=284, film=61 |  |
| Val: actor=59, film=14 |  |
| Test: actor=62, film=13 |  |
| WikiPro | Entire graph: actor=58, football=156, journalist=14, lawyer=16, painter=23, player=25, politician=125, singer=27, sport=21, writer=28 |
| Train: actor=41, football=109, journalist=10, lawyer=11, painter=16, player=17, politician=87, singer=19, sport=15, writer=20 |  |
| Val: actor=9, football=23, journalist=2, lawyer=3, painter=3, player=4, politician=19, singer=4, sport=3, writer=4 |  |
| Test: actor=8, football=24, journalist=2, lawyer=2, painter=4, player=4, politician=19, singer=4, sport=3, writer=4 |  |
| WikiProFem | Entire graph: actor=141, athletic=25, football=24, journalist=16, painter=16, player=32, politician=81, singer=69, sport=18, writer=46 |
| Train: actor=98, athletic=18, football=17, journalist=9, painter=13, player=22, politician=57, singer=48, sport=14, writer=34 |  |
| Val: actor=21, athletic=4, football=3, journalist=4, painter=1, player=5, politician=13, singer=11, sport=1, writer=5 |  |
| Test: actor=22, athletic=3, football=4, journalist=3, painter=2, player=5, politician=11, singer=10, sport=3, writer=7 |  |

### A.2 Parameters for Running the WikES Generator

Table [10](#A1.T10) shows the parameters required for running the WikES Generator. The table provides a description of the parameters and their default values, where applicable. A detailed explanation on how to run the generator can be found in our GitHub repository.

**Table 10: Parameters for Running WikES Generator**
| Parameter | Description | Default Value |
| --- | --- | --- |
| min_valid_summary_edges | Minimum number of valid summaries for a seed node | 5 |
| random_walk_depth_len | Depth length of random walks (number of nodes in each random walk) | 3 |
| bridges_number | Number of connecting path bridges between components | 5 |
| max_threads | Maximum number of threads | 4 |
| min_random_walk_number | Minimum number of random walks for each seed node, explained | 100 for small, 150 for medium, and 300 for large |
| max_random_walk_number | Maximum number of random walks for each seed node | 300 for small, 600 for medium, and 1800 for large |

### A.3 Technologies

Table [11(b)](#A1.T11.st2) presents the versions of the technologies and configurations that we use in this work.

**Table 11: Technology and Configuration Details for Daatset Generations**
| Technology | Version/Details |
| --- | --- |
| Java | Version 21 |
| Spring Boot | Version 3 |
| Docker | Version 24.0.8 |
| Python | Version 3.10 |
| PostgreSQL | Version 16.3 |
| Neo4j | Version 5.20.0-community |
| Wikipedia XML Article Dump Files | Published by Wikimedia on 2023/05/01 |
| Wikidata XML Article Dump Files | Published by Wikimedia on 2023/05/01 |

## Appendix B Datasheet

### B.1 Motivation

- 1.
For what purpose was the dataset created? The motivation behind this dataset generator is to foster research on entity summarization and provide a tool to generate arbitrary-size datasets eschewing the cost of human annotation. We create an easy-to-use executable generator, in which users can create connected graphs by passing their desired seed nodes.
Furthermore, we provide some curated datasets because there has not been a large dataset in this field, and consequently, scalable models have not been developed in this area. To address this issue, we generate large datasets alongside their medium and small versions for research. By running the current models using our generated datasets, we can pinpoint scalability issues of current models and nurture research in this area.
Unlike previous benchmarks, we were interested in generating datasets that follow the patterns of real-world graphs and do not have a bias towards any metric that does not exist in real data. In Section [4](#S4), we study our dataset for different biases and compare it to the ESBM benchmark, which is the most recent entity summarization benchmark to date.
- 2.
Who created the dataset and on behalf of which entity? The dataset is fully created by the authors of this paper.

### B.2 Distribution

- 1.
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? Yes, the dataset is open and fully accessible.
- 2.
How will the dataset be distributed (e.g., tarball on website, API, GitHub)? The WikES generator, WikES benchmark and, the code for developing the baselines will be distributed through our GitHub repository.
- 3.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
- 4.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

### B.3 Maintenance

- 1.
Who will be supporting/hosting/maintaining the dataset? The authors of the paper.
- 2.
How can the owner/curator/manager of the dataset be contacted (e.g., email address)? The owner/curator/manager(s) of the dataset can be contacted through the following emails: Mohammad Sorkhpar (msorkhpar@sycamores.indstate.edu), Saeedeh Javadi (saeedeh.javadi@studenti.polito.it), and Atefeh Moradan (atefeh.moradan@cs.au.dk).
- 3.
Is there an erratum? No, but the users can submit github issues or contact the authors. If errors are found in the future, we will release errata on the main web page for the dataset ([https://github.com/msorkhpar/wiki-entity-summarization](https://github.com/msorkhpar/wiki-entity-summarization)).
- 4.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? Yes, the datasets will be updated whenever necessary to ensure accuracy, with announcements made accordingly. These updates will be posted on the main webpage for the dataset ([https://github.com/msorkhpar/wiki-entity-summarization](https://github.com/msorkhpar/wiki-entity-summarization)).
- 5.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted?) N/A.
- 6.
Will older versions of the dataset continue to be supported/hosted/maintained? Yes, older versions of the dataset will continue to be maintained and hosted
- 7.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Yes, as described above.

### B.4 Composition

- 1.
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Each instance is a document of a Wikidata item, containing some of its immediate neighbours sampled from the Wikidata Knowledge Graph.
- 2.
How many instances are there in total (of each type, if appropriate)? See Section [3.4](#S3.SS4)
- 3.
Does the dataset contain all possible instances or is it a sample of instances from a larger set? It is a sample of instance of a larger dataset from Wikipedia and Wikidata. See section  [3.2](#S3.SS2)
- 4.
Is there a label or target associated with each instance? Yes, each target entity includes target variables.
- 5.
Is any information missing from individual instances? No, we include all the information from the intersection of Wikipedia with Wikidata.
- 6.
Are there recommended data splits (e.g., training, development/validation, testing)? We have generate a train-validation-test for each of the graphs that we generate. See section [3.4](#S3.SS4)
- 7.
Are there any errors, sources of noise, or redundancies in the dataset? Not that we are aware of.
- 8.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The dataset is self-contained.
- 9.
Does the dataset contain data that might be considered confidential? No.
- 10.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.

### B.5 Collection Process

- 1.
How was the data associated with each instance acquired? Each instance in this dataset is acquired from raw XML data dumps published by the Wikimedia Foundation on 2023/05/01. The data is then cleaned and processed by our pre-processing module to extract entities, their relations, and their associated summaries. For more information on the summary annotation process, please refer to Section [3.1](#S3.SS1), “Summary Annotation.”
- 2.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? We utilized an AWS EC2 r5a.4xlarge machine to clean the data and build the datasets using a fully automated pipeline. However, Wikipedia admins and contributors can be considered indirect annotators of this dataset.
- 3.
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Regular employees and two master students were involved in the data collection process. However, the annotation process was fully automated, utilizing publicly available information.
- 4.
Does the dataset relate to people? No.
- 5.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? We sample the dataset from Wikimedia organization dump files on 2023/05/01.

### B.6 Uses

- 1.
Has the dataset been used for any tasks already? We have solely used the dataset for training and evaluating entity summarization baselines in the paper.
- 2.
What (other) tasks could the dataset be used for? Even though, the main target of this dataset is the Entity summarization task it might also be used for link prediction tasks in knowledge graphs.
- 3.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
This dataset is based on cleaned and annotated pre-processed information from Wikimedia published dump files on 2023/05/01. Using this pre-processed dump, one can generate a new dataset based on their desired seed node. However, to generate an updated version of the Wikidata knowledge graph, the pre-processing dumps should be regenerated from updated dump files. The instructions and code for this task are publicly available in our preprocessing URL.
- 4.
Are there tasks for which the dataset should not be used? No.

## Appendix C Experiments

We include the experiments in Section [4](#S4) for all of our datasets below.

Figure: Figure 5: F1 for frequency statistics on WikiLitArt.

Figure: Figure 6: MAP for frequency statistics on WikiLitArt.

Figure: Figure 7: F1 for frequency statistics on WikiCinema.

Figure: Figure 8: MAP for frequency statistics on WikiCinema.

Figure: Figure 9: F1 for frequency statistics on WikiPro.

Figure: Figure 10: MAP for frequency statistics on WikiPro.

Figure: Figure 11: F1 for frequency statistics on WikiProFem.

Figure: Figure 12: MAP for frequency statistics on WikiProFem.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: --> Why the automated mapping from Wikipedia to Wikidata allows for better summarization over entities as stated in "Wikidata to automatically map entities from Wikipedia to Wikidata. This automation allows us to efficiently generate summaries for any number of entities"
--> Why did authors only choose these algorithms for comparison?
--> Why these algorithms are not also tested on all the benchmark datasets which are given in the related work?
soundness: 2
strengths: --> The authors introduce a benchmark dataset for entity summarization over wikidata.
summary: The paper introduces a benchmark dataset for Wiki entity summarization called WikES with the help of random walks.
weaknesses: --> Why the automated mapping from Wikipedia to Wikidata allows for better summarization over entities as stated in "Wikidata to automatically map entities from Wikipedia to Wikidata. This automation allows us to efficiently generate summaries for any number of entities"
--> The motivation behind choosing these 4 algorithms for comparison is missing.
--> A thorough comparison with all the benchmark datasets given in the related work is missing. 
--> The main contribution on why this benchmark dataset is needed should be motivated more clearly.
--> The overall results in table two are very low, what would be the reason behind that?

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. Have you evaluated the correctness of property identification results based on the DistillBERT in terms of entity summarization? 
2. Why the left side of the Equation (2) equal to the right side? 
3. How would the minRW, maxRW, and minRW affect the random walk results for graphs in different scales? Especially how they would affect the quality of the entity summarization datasets? 
4. What is the meaning of Real-first and Real in the Figure 3? They are not explained in the main text.
soundness: 2
strengths: 1. The new benchmark WIKES is the first ES benchmark that does not require human annotation. And the generation method could be easily applied to generate other ES datasets with diverse topics and scales. 
2. WIKES is the largest ES benchmark compared to existing benchmarks, which make it possible to explore the effectiveness of the ES methods over large scale datasets. 
3. Some results on the smallest datasets of WIKES  are presented, giving baseline results for further researches.
summary: This paper introduces a new entity summarization(ES) benchmark, called WIKES, sourced from Wikidata and Wikipedia. Compared existing ES benchmarks, WIKES is generated automatically without relying on human labeling. WIKES also contains complex topology and semantics of the knowledge graph by including 2-hop connected sub-graphs  and diverse topics.
weaknesses: 1. Relying Wikipedia’s abstract to generate the ES datasets is cost-efficient and novel. But this makes the entity summarization generated based on the abstract  text rather than the triples of the entities in the knowledge graph. This might cause the entity summarization in WIKES not the gold entity summarization of the entities. 
2. The DistillBERT is used to annotate the property that should be included in the summarization. The correctness of the final property is not evaluated, which is important to the quality of the WIKES in terms of entity summarization. 
3. The datasets evaluation is not comprehensive. For example, 
    (1). Figure 3 only shows the F1 evaluation on WIkiProFem, part of the WIKES benchmark. The F1 score on other subdatasets are not presented. 
    (2). Table 2 shows that results of entity summarization methods on the smallest WIKES datasets. But the midium and the large WIKES datasets are not tested. It is not clear what would be the performance of current summarization methods on these two datasets. 
4. The dataset quality are not analyzed, for example, the correctness and diversity that are important for ES benchmark. 
5. Minor points and typos: 
   (1). In line 119, there is an extra question mark after “(version 3.9)”. 
   (2). The citation format in the main text seems not correct.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1: In your paper, entity summarization is derived from Wikipedia abstracts and infoboxes, primarily using extraction-based methods. However, you did not explore any text generation or abstractive summarization models (such as GPT or T5) to create summaries from the knowledge graph or the Wikipedia text. Given the recent advancements in text generation, have you considered evaluating the performance of generative models for entity summarization, especially in cases where the structured data might be sparse or incomplete? What are the potential reasons for not including these methods in your benchmark?

Typos:
- Line 34, Entity summarization (ES) => Entity Summarization (ES)
- Line 119 “(version 3.9) ? and”
- Line 122 “INFO” may need to be bold.
soundness: 3
strengths: S1: The WIKES benchmark is scalable, leveraging automatic summary generation from Wikipedia and Wikidata without relying on costly manual annotations, making it applicable to large datasets across various domains. The use of random walk-based subgraph extraction ensures that the structure of knowledge graphs is preserved, capturing both topological and semantic complexities of entities while maintaining computational efficiency.

S2: This paper provides a thorough evaluation of multiple graph-based entity summarization methods (e.g., PageRank, RELIN, GATES), allowing for direct comparison of unsupervised and supervised approaches, highlighting the advantages of graph-based methods.

S3: The overall presentation is good and the authors provided source code for review.
summary: This paper introduced a novel large-scale benchmark, named as WIKES, for entity summarization (ES) in knowledge graphs. Existing benchmarks are limited in size and often rely on expensive and time-consuming manual annotations. In contrast, WIKES aims to overcome these limitations by using Wikipedia abstracts and knowledge graph structures from Wikidata to automatically generate high-quality entity summaries without the need for manual annotators.
This paper proposed a dataset generation algorithm by combining graph algorithms with natural language processing models. The automatic processing makes the dataset be scalable and suitable for multiple domains.
Finally, this paper  offers a comprehensive evaluation of several entity summarization methods, including both unsupervised (e.g., PageRank, RELIN, LinkSum) and supervised models (e.g., GATES). It highlights that unsupervised methods, particularly graph-based approaches, often outperform supervised models in large-scale knowledge graphs.
weaknesses: W1: The summarization methods are limited to graph-based summarization techniques. The authors may need to evaluate some text generation methods. A broader comparison with recent NLP-based summarization techniques could add more depth.

W2: The paper focuses on scalability but only evaluating the small version of their dataset. The methods without efficiency concerns could be used to conduct evaluation on the large version to show the effectiveness of the proposed dataset.

W3: The random walk-based graph expansion approach may not always capture the most semantically relevant information for all types of entities. While the two-hop neighborhood approach is computationally efficient, it may miss out on key relationships that are further away in the graph but still contextually important for the entity. The authors could have considered a dynamic approach, where the hop count is adjustable based on the entity or relationship type.

W4: The authors could provide some qualitative examples of the generated summaries of model like LinkSum in Appendix .

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: Apart from my questions in Weaknesses, I have the following further questions/comments.

Q1. For relation selection, you compared relation embedding with abstract embedding. The latter covers a very large piece of text. How about comparing with the embedding of the sentence where the relation value is mentioned?

Q2. Line 321-323, I did not understand why the bias comes from the small size and/or the incomplete edges of the sampled graphs. An extended explanation would be appreciated.

Q3. LinkSum performed best in your experiments. Is it possible that its good performance came from its use of Backlinks, which coincides with your approach to generating gold-standard summaries (based on entities mentioned in Wikipedia abstracts)? It may represent a bias of your ground truth.
soundness: 3
strengths: S1. A programmatically generatable benchmark for entity summarization. Can be larger and better than existing ones. It represents a useful contribution to the community. I like it.

S2. Generally reasonable approach to benchmark generation. The generated gold-standard summaries are likely to have high quality, despite lacking verification.

S3. Well-written paper. Very easy to read.
summary: This paper presents a method to automatically create a dataset for evaluating entity summarization. Compared with existing datasets, with this new approach:
- larger datasets can be programmatically generated,
- mutli-hop (empirically 2-hop in the paper) relations are allowed,
- a connected subgraph is guaranteed to be sampled.
weaknesses: W1. Missing some important technical details and experiments related to the quality of the automatically generated gold-standard summaries.
- Line 175-176, how did you identify 'mentions' of Wikidata items in a Wikipedia page? If only relying on hyperlinks, recall could be low. Essentially it is an entity linking problem. I would like to see more details about its implementation. I also want to see a technical comparison of your approach with https://doi.org/10.1016/j.ipm.2013.12.001 which also relies on Wikipedia abstract to create gold-standard entity summaries.
- The above-mentioned entity linking, and your heuristic relation selection method, both can be inaccurate. Have you conducted experiments to evaluate the quality of your generated gold-standard summaries?
- Line 138-140, according to this problem definition, do you consider or ignore literals?

W2. While your sampled graphs can be arbitrarily large, it seems that your gold-standard summaries are limited in the following aspects.
- I am not sure whether generating a larger graph is really helpful for the entity summarization task. According to Table 1(b)(c)(d), while graphs differ in size, the number of gold-standard summaries remains almost unchanged, about 500 entities, if I understood correctly. You criticized existing benchmarks for their small size in terms of the number of entities with gold-standard summaries (e.g., 175 in ESBM), but your datasets are not significantly larger.
- All these ~500 entities are instances of person, while there are many other types of entities in DBpedia/Wikidata and in previous benchmarks like ESBM. Why did you limit your datasets to person entities? It introduces a bias.
- Only entities with English labels are considered. This is acceptable, but not necessary IMO.

Meta-review: This paper introduces Wiki Entity Summarization Benchmark (WikES) which is really an approach to generate entity summarization datasets. While reviewers agreed that this constitutes a potentially useful direction, the motivation and even the characterization of WikES as a "benchmark" seems somewhat confusing. For example, as pointed out by reviewer F6iB, there is limited evaluation of the generated data, though this seems critical for a benchmarking paper.

Decision: Reject