Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors build a new corpus for information extraction which is larger comparing to the prior public corpora and contains information not existing in current corpora. The dataset can be useful in other applications. The paper is well written and easy to follow. It also provides details about the corpus. However, there are some questions for the authors: 1) It uses the NLP pipeline and the MinIE-SpaTe system. When you get the results, do you evaluate to what extent that the results are correct? 2) In Section 3.4, the author mentioned the correctness is around 65%, what do you do for those incorrect tuples? 3) Have you tried any task-based evaluation on your dataset?
review_title: OPIEC: An Open Information Extraction Corpus

final_score: rating: 7: Good paper, accept
review_content: This paper presents a dataset of open-IE triples that were collected from Wikipedia with the help of a recent extraction system. 

This venue seems like an ideal fit for this paper and I think it would make a good addition to the conference program. While there is little technical originality, the overall execution of the experimental part is quite good and I like that the authors focused in their report on describing how they filtered the output of the employed IE system and that they present interesting examples from the conducted filtering steps.

I particularly liked the section on comparing the new resource to the existing knowledge bases from the same source (YAGO, DBpedia), I think it makes a lot of sense to pick resources that leverage other parts of Wikipedia (category system, ...) and not the main article text, and to look into how coverage of the these different approaches relates.

It would have been nice to also compare against other datasets/triple stores/... that used open-IE to extract from Wikipedia. A couple of references discussing such are listed in the paper, e.g., DefIE or KB-Unify seem like good candidates.

review_title: Good paper on producing a triple store from Wikipedia articles.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes the creation of OPIEC -- an Open IE corpus over English Wikipedia.
The corpus is created in a completely automatic manner, by running an off-the-shelf OIE system (MinIE), which yields 341M SVO tuples. Following, this resource is automatically filtered to identify triples over named entities (using an automatic NER system, yielding a corpus of 104M tuples), and only entities which match entries in Wikipedia (5.8M tuples).

On the positive side, I think that resources for Open IE are useful, and can help spur more research and analyses.

On the other hand, however, I worry that OPIEC may be too skewed towards the predictions of a specific OIE system, and that the work presented here consists mainly of running off-the-shelf can be extended to contain more novel substance, such as a new Open IE system and its evaluation against this corpus, or more dedicated manual analysis. For example, I believe that most similar resources (e.g., ReVerb tuples) were created as a part of a larger research effort.

The crux of the matter here I think, is the accuracy of the dataset, reported tersely in Section 5.3, in which a manual analysis (who annotated? what were their guidelines? what was their agreement?) finds that the dataset is estimated to have 60% correct tuples. Can this be improved? Somehow automatically verified?

Detailed comments:

- I think that the paper should make it clear in the title or at least in the abstract that the corpus is created automatically by running an OIE system on a large scale. From current title and abstract I was wrongfully expecting a gold human-annotated dataset.

- Following on previous points, I think the paper misses a discussion on gold vs. predicted datasets for OIE, and their different uses. Some missing gold OIE references:
Wu and Weld (2010),  Akbik and Loser (2012), Stanovsky and Dagan (2016).

- Following this line, I don't think I agree with the claim in Section 4.3 that "it is the largest corpus with golden annotations to date". As far as I understand, the presented corpus is created in a completely automated manner and bound to contain prediction errors.

- I think that some of the implementation decisions seem sometimes a little arbitrary. For instance, for the post-processing example which modifies 
(Peter Brooke; was a member of; Parliament) to (Peter Brooke; was ; a member of Parliament), I think I would've preferred the original relation, imagining a scenario where you look for all members of parliament (X; was a member of; Parliament), or all of the things Peter Brooke was a member of (Peter Brooke; was a member of; Y) seems more convenient to me.

Minor comments & typos:

- I assume that in Table 1, unique relations and arguments are also in millions? I think this could be clearer, if that's the indeed the case.

- I think it'd be nice to add dataset sizes to each of the OPIEC variants in Fig 1.

- End of Section 3.1 "To avoid loosing this relationship" -> "losing this relationship"

- Top of P. 6: "what follows, we [describe] briefly discuss these"

- Section 4.5 (bottom of p. 9) "NET type" -> "NER type"?
review_title: Interesting analysis, but may not be enough content

Meta-review: metareview: This paper describes a new Open IE corpus over English Wikipedia. All the reviewers agree this paper is suitable for this venue and the dataset is useful. Overall, the paper is well-written and the experiments are convincing. Despite the novelty of this paper is  relatively thin, it is a decent paper.  
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)