Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces ‘EventNarrative’ : an event centric knowledge graph to text dataset, which is about 6 times larger than the current largest parallel dataset. The paper highlights the data creation process in detail and provides a final synopsis of the generated EventNarrative dataset. The sources used are EventKG along with wikidata, enhanced with full text from wikipedia. This is followed by entity matching and Narrative graph generation after various methods of filtering of the events, and a separate module to capture dates. The final dataset is then compared compared with existing datasets by size and data sparsity. There is also a qualitative analysis conducted, using human annotators, to ascertain the quality of the data. Further, a benchmark evaluation is also done.

 The main contributions of the paper are as follows:
1. A large-scale, event-centric, parallel knowledge graph-to-text dataset of over 230,000 KG to text pairs, spanning over 7,000 event types, and over 650,000 triples
2. A comprehensive entity matching and knowledge graph-to-text matching algorithm that automatically pairs KGs to natural language texts
3.  Benchmark evaluations and baselines results on EventNarrative

strengths: The proposed dataset - 'EventNarrative' is 6 times larger than the current largest parallel dataset. It covers events across a wise range of domains and time periods. The knowledge graph-to-text pairs are generated automatically using a rich ontology, which paired with a its large scale nature, would form dense and information graphs, The qualitative analysis conducted, with the help of 3 human annotators, on 500 randomly sampled Knowledge graph-to-text pairs showed good accuracy values - ~96% for entities and 95% for relations.
The authors have provided a comparison of the dataset across other popular datasets - WebNLG, AGENDA, GenWiki (full and fine). WebNLG and EventNarrative, both are parallel, have all entities from the knowledge graph present in the narrative and all entities are linked to the graph triples. However, EventNarrative is a significantly larger dataset. Additionally, as EventNarrative is sourced from wikipedia, various existing tools can be used on this dataset. This will be beneficial for the broader research community to carry out further experiments and evaluate models in the domain.

weaknesses: In the dataset creation, there are several limitations which have been highlighted by the authors in section 2.4.
These talk about missing out on entities found within the narrative test, due to incompleteness of wikidata, which is used as a key source in the data creation. 
They also highlight the poor performance of co reference resolution systems in event centric data.
In addition to these, it is not clear if any sort of semantic similarity is checked for the events or properties in this process. If only an exact match is being considered, it may lead to missing or double counting of certain events or properties.

In section 3.3, the qualitative analysis is described. 
Here, 500 randomly sampled KG to text pairs are chosen for manual checking. As there are 230k such pairs in the data, 500 seems to be a small sample size. The results produced in this analysis, may be biased and will possibly not be a correct representation of the entire dataset. 

In section 4.1, the authors describe the experimental setup for the benchmark evaluation. This is done in detail and will definitely help in reproducing the results. However, there is no insight on the reasoning behind choosing the particular parameters mentioned.
correctness: The claims in the submission seem to be correct and appropriate references and comparisons have been provided with existing work. There is also qualitative analysis done to ascertain the quality of the dataset which has given a good score, in the selected sample dataset.
The dataset creation process is conducted in a thorough manner and the steps have been described in detail. For each step, the necessary steps were taken as needed. - eg. addition of the date module in keeping with the wikipedia style manual etc. The reasonings and explanation for these were given as well. - Section 2.
In section 4, the benchmark evaluation process is described. This is done in 2 steps:
1. modeling the problem with a graph transformer-based network - GraphWriter
2. treating the problem as a summarisation task by fine-tuning on pre-trained language models - BART and T5


clarity: There are some subsections which are a little hard to follow as there are a lot of details present - sections 2.1 and 2.2. These highlight the initial stages of the dataset creation process. There are a lot of steps which have been condensed into paragraphs in these sections. If they could be broken down into bullet points or restructured, it may be easier to follow. 
A real example from the dataset, for these steps, would also help the reader understand better, in addition to the flowchart already provided (Figure 2).

Also,  the section on Related Datasets (Section 5) should be moved up. This will help build context on these datasets which have been used for comparisons throughout sections 3 and 4.
relation to prior work: The authors have provided a comparison of EventNarrative with other popular datasets - WebNLG, AGENDA, GenWiki (full and fine) in section 3. Section 5 talks about all the mentioned datasets. The comparison is done on the following fronts: total number of knowledge graph components, number of tokens in the narratives, overall mean statistics, domain, ontology etc. Table 1 and Table 2 highlight these in detail.
The paper states, on the basis of these analyses, that "EventNarrative is the only dataset that is large-scale, ontology-based, utilizes an open real-world KG, may be used for supervised learning, and contains fully connected graphs that are fully contained within a text narrative"
additional feedback: 1. The AGENDA dataset is first mentioned in line 48. The source for this should be cited here. It has been cited in later occurrences (line 174 and later), but the first mention should have the relevant citation.
2. Section 5 can be moved up in the structure of the paper. Section 5 describes the related datasets which have been used for comparisons and analyses all throughout section 3 and 4. It makes sense to introduce the datasets before these sections.
3. In section 2, there are a lot of steps which highlight the data creation process. It becomes a little difficult to follow as there are multiple steps. It would be very helpful if you could provide a relevant example to help explain the process.

4.In section 3.3, the qualitative analysis is described. It'll be helpful if you could provide more detail on this process. Why was a sample size of 500 pairs chosen? Were there any steps taken to ascertain no or low bias in this data? Why do you think this sample will be a good representation of a dataset with 230,000 pairs?

5. In section 4.1, the experimental setup for the benchmark evaluation. The paper provides the specific details of the parameters and models chosen. How were these parameters decided? Did you conduct experiments with other configurations? Can you share the comparison details for these or any explanation on why the mentioned configuration is preferred?
review_title: Review on - EventNarrative: A large-scale Event-centric Dataset for Knowledge Graph-to-Text Generation 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this resource paper, the authors provide the EventNarrative dataset for event-centric knowledge graph to text generation. The dataset was created by careful linking (and refining) the elements in EventKG and Wikipedia articles.  The main contributions include the EventNarrative dataset itself, the procedure for the linking and refining, and an evaluation of the baseline models on this dataset.  
strengths: The EventNarrative dataset created by the authors is claimed to be the largest dataset for event-centric KG to text generation. The dataset is created based on existing well established data sources e.g., EventKG, Wikidata, and Wikipedia. In this sense, it is a good contribution to the research problem of event-centric KG to text generation. 

Because of the dataset is created based on easily accessible and highly accountable sources, there is no major concern on the raw quality of the data sources. 


weaknesses: Authors lists a few limitations in Section 2.4 on the dataset itself mainly because of the creation process and the ignorance of co-reference resolution.  Other than the limitation on the dataset creation itself, it is not clear the unique characteristics of event-centric KG compared to generic KB with respect to the task of KG to text generation. The authors may want to provide a stronger justification on why such a dataset is essential to study KG to text generation. 

Another weakness is on the quality of the dataset creation. The authors evaluated 500 randomly sampled KG-text pairs and reported a very good score. However, there is no evaluation during the multiple steps in the dataset creation. It would be very helpful if the authors could evaluate a sample of instances during each step in the creation, for example, the accuracy of entity linking, and also provide case studies on the errors. Because some steps involve rule-based approaches or RegEx, to be more confident about the quality of the dataset, proper evaluation during the creation process will be helpful. 

A related issue is: how to make corrections to the dataset. There are errors noted in evaluating the 500 samples. The question here is: what is the best way to correct some of these errors once the data is related to the research community.  

correctness: The overall creation process is reasonable. Nevertheless, the authors do not present the detailed creation process in a more formal way (e.g., the entity matching algorithm). 
clarity: The paper is well written in general. To improve, the Introduction section can be better organized. The few paragraphs in the Introduction can be better arranged to define event, provide the background of event-centric KB, and the current status of the KB to text generation. It reads to me that the authors emphasize the dataset is largest for multiple times, but do not discuss much on the quality of the dataset.   


relation to prior work: The authors did a good job to position this dataset with existing ones (to my best knowledge). 
additional feedback: The authors do not give much discussion on the results reported in Table 4. What is worrying here is the significant differences between the absolute values of the measures across different models on the same dataset. 

The authors have well addressed most of my comments through feedback.
review_title: A large-scale dataset for event-centric knowledge graph to text generation 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper provides a new KG-to-text benchmark based on events and their narratives. It is larger than existing similar benchmarks, thus facilitating supervised learning of large neural models.
By the means of various filtering methods, the authors ensure a close correspondence between the text and the KG, i.e., the text is guaranteed to be predictable from the knowledge stored in the graph. Starting from human-authored Wikipedia text, the text, however, becomes more artificial by replacing all entities with their canonical strings from the graph. In some cases, this even leads to unnatural text, when, e.g., a year is present in the KG but the text contained a full date. The result then has phrases like "between 2014 and 2014". These limits are discussed and, based on a human evaluation, these errors are rare.
The event-centricity of the new benchmark is also a useful contribution that previous benchmarks lack although this type of formal knowledge is particularly suitable to be written out as text.

(modified: 20 Jul 2021)
strengths: - large-scale
- event-centric
- parallel (filtered to guarantee good matches between texts and KGs)
- explicit links between graph nodes and entity strings
weaknesses: The paper does not evaluate a lot of baselines and their results could be discussed in more detail.
I also have some doubts about the way their experiments are described. Cf. below 'Correctness'.

after rebuttal: most of these weaknesses are explained and so the paper only needs to incorporate these clarifications.
correctness: The data collection seems reasonable and correct.

Regarding the baseline evaluation, I understand that BART and T5 are evaluated on a random subset of the dev portion. But the whole dev portion is used for model selection? That is a weird choice. Or do you mean that you evaluate only on a random subset of the test split after selecting your model on the dev split?
Also, are the results in Table 4 comparable? Is GraphWriter evaluated on the same exact subset of the data as BART and T5? Also, is the same random set of data used to evaluate T5 and BART or do you sample different subsets here?
If the data splits were not the same, the evaluation is a lot less meaningful.
clarity: L157-158, L193: your graphs are certainly not all "fully connected". You proably mean just "connected", right? "Fully connected" means that every node is connected to every other node. You mean that you don't have disconnected graphs, as I understand it.
relation to prior work: The work offers a detailed analysis and comparison with existing popular graph-to-text generation benchmarks.

The work would benefit from mentioning the contemporaneous work "WikiGraphs"[1].


[1] https://www.aclweb.org/anthology/2021.textgraphs-1.7
additional feedback: Your section 5 describes the existing datasets rather than compares them to your existing work. So I recommend putting it nearer to the beginning because it provides context for readers that are less familiar with existing datasets. After reading your analysis in section 3, I found that section 5 does not provide much more insight. A different order might work better.

As one of the filter criterions was that each sentence must contain at least two graph entities, I wonder if something like a serialization plan could be constructed for the text-graph pairs, i.e., the order of the triples as they appear in the text or some correspondence between sentences and triples. Recent work [2] has shown that this kind of annotations can be very useful.

Typo in L122: knowledge KG -> KG


[2] https://www.aclweb.org/anthology/2020.acl-main.224
review_title: Exciting new dataset, needs clarification about baseline evaluation

Meta-review: In this paper the authors provide a new very large dataset of 230k graphs for knowledge graph to text generation.  All reviewers found the large dataset itself and the process by which it was generated from reliable sources to be an important contribution with significant benefit over prior work.
Decision: Accept