Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper studies the problem of fine grained entity recognition (\fger) and typing . They initially present an excellent analysis of how fine grained entity type sets are not a direct extension of the usual coarse grained types (person, organization, etc) and hence training \fger systems on data annotated on coarse grained typed datasets would lead to low recall systems. Secondly, they show that automatically created fine grained typed datasets are also not sufficient since it leads to low recall because of noisy distant supervision. This analysis was both an interesting read and also sets the stage for the main contribution of this work.

The main contribution of this work is to create a high recall large training corpora to facilitate \fger and typing. The authors propose a staged pipeline which takes in input (a) a text corpus linked to a knowledge base (using noisy distant supervision), (b) a knowledge base (and also a list of aliases for entities) and (c) a typed hierarchy. All of these are readily available. The output is a high-recall linked corpus with linked entities to the KB and the type hierarchy. To show the usability of the training corpus the paper performs excellent intrinsic and extrinsic analysis. In the intrinsic analysis, they show that the new corpus indeed has a lot more mentions annotated. In the extrinsic analysis, they show that state art of the models trained on the new corpus has very impressive gains in recall when compared to the original existing datasets and also wiki dataset with distant supervision. There is a loss in precision though, but it is fairly small when compared to the massive gains in recall. This experiments warrants the usability of the generated training corpus and also the framework they stipulate which I think everyone should follow.

Great work!

Questions:

1. In table 3, can you also report the difference in entities (and not entity mentions). I would be interested to see if you were able to find new entities. 
2. Are you planning to release the two training datasets (and if possible the code ?)

Suggestion: The heuristics definitely work great but I think we can still do better if parts of stage  2 and 3 were replaced by a simple machine learned system. For example, in stage 2, just relying on heuristics to match to the prefix trees would result in always choosing the shortest alias and would be problematic if aliases of different entities share the same prefix. Restricting to entities in the document would definitely help here but still there might be unrecoverable errors. A simple model conditioned on the context would definitely perform better. Similarly in sentence 3, the POS based heuristics would just be more robust if a classifier is learned. 
review_title: Super-exciting stuff, great analysis and overall a well written and executed paper

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a new way to get distant supervision for fine grained NER using wikipedia, freebase and a type system. The paper explores using links in wikipedia as entities, matching them to freebase, and then expanding the using some string matching, and then finally pruning sentences using heuristics. Methods are compared on FIGER. The paper also introduces a dataset, but it is not fully described. 

One interesting aspect about this paper is, as far as I can tell, one of the few works actually doing mention detection, and exploring the role mention detection. 

That being said, its unclear what is new about the proposed source of supervision. The first two stages seem similar to standard methods and the last method (generally speaking pruning noisy supervision), has also been explored (e.g. in the context of distant supervision for IE, http://www.aclweb.org/anthology/P18-2015, and see Section 2.2) . Its also not clear to me what specifically targets good mention detection in this method. The experiments do somewhat argue that mention detection is improving, but not really on FIGER, but instead on the new dataset (this inconsistency causes me some pause). 

All that being said, I don't think it would matter much if the supervision were incorporated into an existing system (e.g. https://github.com/uwnlp/open_type or https://github.com/MurtyShikhar/Hierarchical-Typing) and demonstrated competitive results (I understand that these systems use gt mention, but any stand in would be sufficient).  Table 5 on the other hand has some results that show baselines that are significantly worse than the original FIGER paper (without gt mentions) and the proposed supervision beating it (on the positive side, beating the original FIGER results too, but not included it in the table, see Section 3 of http://xiaoling.github.io/pubs/ling-aaai12.pdf ). So in the end, I'm not convinced on the experimental side.

This paper could be significantly improved on the evaluation. Incomplete reference to previous work on FIGER and insufficient description of their new datasets are areas of concern. Cursory reference, instead of evaluation, on new fine grained datasets (like open-type) also seem like missing pieces of a broader story introducing a new form of supervision.

review_title: New distant supervision method focusing on mention coverage, but incomplete evaluation 

final_score: rating: 7: Good paper, accept
review_content: This paper presents a methodology for automatically annotating texts from Wikipedia with entity spans and types. The intended use is the generation of training sets for NER systems with a type hierarchy going beyond the common four types per, loc, org, misc.

I think this paper is a good fit for AKBC, given that entity recognition and linking systems are an essential part of knowledge-base population. This should be good for interesting discussions.

A big caveat of this paper is that, while the approach discussed by the authors is generally sound, it is very tailored towards Wikipedia texts, and it is unclear how it can be generalized beyond. Since this approach is meant for downstream training of NER systems, these systems will be limited to Wikipedia-style texts. A statement from the authors regarding this limitation would have been nice. Or maybe it's not a limitation at all? If the authors could show that NER systems trained on this Wikipedia-style texts do perform well on, say, scientific texts, that would be already good.

A second issue I see is that Stage III of the algorithm filters out sentences that might have missing entity labels. This makes sense, provided that the discarded sentences are not fundamentally different from the sentences retained in the NER tagger training set. For example, they could be longer on average, or could have more subordinate clauses, or just more entities on average, etc. This is something the authors should look into and should report in their paper. If they find that there are differences, an experiment would be nice that applies an NER tagger trained on the produced data to these sentences and verifies its performance there.

A minor point: The authors claim their approach scales to thousands of entity types, which I find a bit of an overstatement, given that the dataset produced by the authors has <1.5k different types (which is already quite a lot).
review_title: Good paper with two caveats

Meta-review: metareview: This paper design a framework based on heuristic approaches to automatically construct a dataset for the FgER task. The paper is solid and present nice experimental comparisons. Overall, it is clear and easy to follow.  However, as pointed out by the reviewers, the proposed approach is heuristic and may not be general enough for handling other tasks or data in other domains. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)