Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents a novel solution to an interesting problem - when KBs are automatically expanded user feedback is crucial to identify incorrect and missing entity attributes and relations. More specifically, in the case of entity identity uncertainty, enabling the user feedback as part of the entity resolution mentions, appears novel and important. 
The paper is well written and organized. 

Points for improvement:
- It would be interesting to see more in-depth analysis on examples where the proposed approach fails, and based on this to also outline open issues and future work. 
- The human computation aspects of the paper are lacking sufficient explanation in terms of implementation in real settings, as well as positioning with related work in human computation research
- it would be interesting to know, what is the experimental design that authors would consider for an evaluation with actual user feedback vs. the simulated one
review_title: Relevant contribution for humans-in-the-loop in KB construction

final_score: rating: 3: Clear rejection
review_content: This paper presents a hierarchical framework for integrating user feedback for KB construction under identity uncertainty.

1. it is unclear about the algorithm implementation, such as what is the implementation of feedback mention.

2. There is a definition about attribute map, but I can't find where the model uses it. Same thing for the precision of a node pair.

3. How to calculate the function g(.) is unclear either. 

4. In section 5.2, the COMPLETE definition seems not correct, it is still the definition of PURE.

5. The example for constructing positve/negative feedback is too vague.

6. The experiment section needs more analysis including qualitative result.
review_title: lack of details, few novelty

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a method to integrate user feedback into KBs in the presence of identity uncertainty, a problem that arises in the integration of new data in knowledge base construction. The proposed method represents user feedback as feedback mentions and uses an online algorithm for integrating these mentions into the KB.

The paper targets an important problem in knowledge base construction, i.e., integrating user feedback in the online setting. The proposed hierarchical model looks reasonable and effective. And overall, the work is well presented. 

The paper makes an algorithmic contribution. The contribution is, however, limited from the perspective of human computation. The experiment uses simulated user feedback for evaluating the method. In real-world settings, user feedback can be skewed to certain types (e.g., negative feedback) or be noisy (so the feedback is not reliable). How would these affect the result?
review_title: A nice algorithmic contribution for integrating user feedback for KBC

Meta-review: metareview: The paper presents an interesting methodology. The results are interesting, however the paper really misses out on an in-depth discussion and reflection of the pros and cons of this approach as well as on a proper related work comparison to similar approaches. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)