Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: Enterprise Alexandria automatically builds knowledge bases with typed entities from private enterprise datasets. To build a KBC and identify typed entities, EA uses a probabilistic program and does not use meta-data and human annotations, which are generally not available in enterprise datasets anyways.

The authors ran a user study and they show that users need a variety of entity types and users wish to engage with different types in different ways. The authors claim that the study helps in the design of EA's user model. 

Evaluations in EA focused on two aspects: (1) the precision and coverage of EA's identified entity types  against the precision and coverage of outputs from 4 BERT-based models and (2) the scalability of EA's incremental clustering. The results show EA has gains in terms of precision. 


Strong Points: 
1. Explains the challenges, motivation, and novelty fairly well. 
2. Evaluations proved gains in precision with EA compared to BERT models, which are the state-of-the-art methods.
3. Performance evaluation proved how well the incremental clustering scaled in terms of memory usage and runtime.
4. The world cloud visualization shows how relevant the most frequent entity types in the dataset are to the dataset topics. Also, 40% of the user-expected types (from the user study) were retrieved by EA. 


Weak points: 
1. It's not clear how the user study contributes to the design of EA itself. 
2. Don't entity types differ from one dataset domain to another? The results show that 40% of the user-expected types were retrieved by EA, which is great, but the domains covered from the user study (entity types related to multinational corporation data and types related to data found in the tourism and travel industry) are different from the domain datasets from the evaluation (types expected of Enron and Anonymous Eyes-on). How were the differences in domain addressed when comparing the types from EA and the types identified by the users from the user study? 
3. While there are gains for precision across both datasets in the evaluations, it's not the same for relative coverage. It would be interesting to see how relative coverage fares on a third dataset (if it is even feasible to find such a third dataset). 
4. Word cloud visualizations and the datasets associated need a bit more context. A few sentences describing what datatypes are expected based on a given dataset would help. 

Overall, the paper coupled with the evaluation results are promising. However, there are certain aspects of the paper that needs to be addressed. 
review_title: Challenging, real world problem with evaluations that could be better

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work extends the Alexandria system for KB construction [Winn et al.,19] , to the particular scenario of constructing enterprise-specific knowledge bases. The main system extensions described in the paper are:
(1) Extending Alexandria’s templates mechanism to support (semi-)structured documents, that are common in enterprise scenarios such as emails & calendar events. This allows the system to leverage the availability of semi structured data when building the KB. 
 (2) Improved entity disambiguation mechanism which considers the number and length of words, as well as variants stemming  from reasonable enterprise scenarios 
(3) Type discovery mechanism - The original entity discovery process described in [Winn et al.,19] is extend to  allow the system to partially discover types. This is very practical in the Enterprise scenario, where the entity types may vary from enterprise to enterprise, as oppose to the case of general KBs.  However, the new types are learned for entities with already known types, so this does not extend to the general case of arbitrary type discovery.
Also, An additional incremental computation mechanism is introduced which allow to incorporate manual annotation/curation


Overall this work showcase and interesting and useful extension of Alexandria, although the technical contributions seem incremental and without much technical depth. The main issue with this work is that the experimental evaluation section requires more clarifications and experiments. 

Strong points:

S1 The paper extends Alexandria to particularly construct an enterprise-specific knowledge base, which is an interesting, practical use case 

S2 The extensions proposed in Section 3 takes the main differences between the general KB and the enterprise KB. 

S3 The Experiments section provides an interesting comparison between the dedicated approach proposed in the paper with with several baselines for general-purpose NER.

Weak Points (refer to the Detailed Comments Section for more information)

W1 The introduction and related work section are somewhat confusing and do not properly justify the particular work and the contribution made. 

W2 The qualitative study does not seem informative and useful in the rest of the paper

W3 The extensions proposed to Alexandria do not demonstrate much novelty and technical depth

W4 The experimental section lacks several important clarifications and additional experiments, as detailed in commends D4 and D5


Detailed Comments:

D1 Positioning: The contribution is not clearly stated in the introduction, as the authors first claims that EA is specifically designed to answer a privacy constraint that forbids manual annotation of enterprise data. However in the contribution section the authors do not address the privacy issue anymore. Also, the abstract should state that EA extends Alexandria. 

D2 The purpose of the qualitative study is not clear. The authors asked the participants to list typed entities relevant to their organization, and conclude that “a wide variety of entity types are important for work”. What exactly is wide? Are all these entities necessary? How is this a conclusion from the study? And in any case,  48 entity types obtained for two different organizations, sounds really small in comparison with general knowledge bases (e.g. dbpedia) which contains thousands of entities. 

D3 While the extensions of Alexandria  seem appropriate to the enterprise scenario, they do not show great technical depth and novelty. 

D4 The authors compared their approach to several baselines based on pre-trained language models that were fine tuned for NER tasks with limited, predefined number of entity types. If we are interested in specific types, isn’t it better to perform the fine tuning over some more relevant types? This may greatly improve the precision of these baselines. 

D5 Several Aspects in the experiments are not clear: 
(i) Which particular entities were labeled in the datasets? Of which/ how many types? What are the negative entity examples? As one dataset is anonymous and the other is Enron + manual labels, this requires clarification 
(ii) What is the relative coverage measure and why was it used instead of standard measures such as  recall?
(iii) Why does the EA version with >100 evidence obtained lower precision scores? Intuitively it should be more “correct” w.r.t. the entities it identifies, on the account of the recall. 
(iv) Why wasn’t Alexandria a baseline in your experiments? It seems reasonable to show that the existing system does not properly identify the desired entities and types in an enterprise data scenario.  
(v) The evaluation of the the type retrieval mechanism, one of the major contributions of the paper, is lacking, to my opinion, as it only shows a world cloud to exemplify the discovered types. There should be an experiment that measure the quality/utility/correctness of the types discovered.
(vi) The details regarding the performance of the system, which rely on the incremental/clusterting mechanism of Section 4, are lacking a comparison with the approach described in [Winn et al.,19]  in order to demonstrate that the optimization proposed in the paper is indeed beneficial. 

review_title: Interesting and useful System that needs more work and clarifications on the experimental side 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors have presented a dedicated new system for automatically constructing a knowledge base from private enterprise data. This system is an extension of an existing system designed for Web-based information extraction.  It employs a probabilistic program to generate both the document and its metadata from a set of latent entities and adds the ability to learn the set of types associated with each entity from a pre-defined type hierarchy.  . This work also presents a real-world evaluation showing that the developed system scales well and significantly improves the precision of the extracted knowledge compared to four baselines on three real-world datasets. 


Strengths:

 - the problem this work handles is well-motivated, and the paper is well written and easy to follow. 
- the authors explain very clearly their contributions, challenges, and novel extensions to Alexandria. 
- the presented system supports online settings, where the documents are uploaded not at a single time point and the memory consumption exceeds the capacity of a single machine, which is a real-life scenario for enterprise data. 
- the authors compare their proposed system with state-of-the-art baselines, showing their advantages in real-life scenarios. 

Weaknesses:

- the presented user study is very interesting and nicely demonstrates
why a wide variety of entity types are important. However, this study includes a very small number of participants, and important details regarding the grouping of the categories are missing.
-It would be of interest to evaluate each component's marginal contribution, e.g., using other clustering methods to better explain the reviewers' choices. 
review_title: Well-motivated real-world problem, the main contributions are nicely explained.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper describes a system named “Enterprise Alexandria” for knowledge base construction with typed entities. The approach uses relevant methods.
Overall, the paper is difficult to read and follow. It is very difficult to decode the main contributions of the paper. Some parts are very briefly described and it is difficult to connect and understand different parts of the paper. The evaluation is also questionable - 1) the focus is unclear - what has been evaluated?, and 2) the comparison/experiments with the related methods improperly designed.
In summary, there are some contributions but they are not well presented. The paper is a mix of presentation of a system/approach/workflow and description of few methods -> the paper lacks focus. E.g. it would be better if the paper describes the system only, or focus on a particular method, e.g. entity linking or entity typing. In the current form it is a mixture and confuses the reader.
Considering my comments above I do not recommend acceptance of the paper. 
 More comments below.

==

= Abstract

“ability to process both the textual information and the structured metadata available in each document in an online fashion“
-> it is not clear what is meant by “online fashion”

“The knowledge discovery process uses a probabilistic program defining the process of generating the data item from a set of unknown typed entities.”
-> complex sentence

Section 4. Incremental clustering
-> it is unclear the purpose of the entity correctness task. It is applied, but it is unclear if any follow-up action is then. Also, there are missing details about how it actually works.

Section 6. Experimental Evaluation
-> It is unclear what exactly has been evaluated? I.e. what part of the pipeline? Typing, linking, matching?
-> the setup of the experiments are questionable. The (BERT) models have been trained and perform typing with just a few types (3, 16 types), while the model proposed in the paper performs typing of over 152. The comparison is unbalanced. More proper setup would be to train/configure the BERT to perform typing with the same amount of types - that would be fair comparison.

Section 6.4. Type Retrieval
-> Instead of word cloud figures the authors could consider providing in-depth analysis on the performance of entity typing.

review_title: lack of focus, unclear contributions and difficult to read and follow

Meta-review: The reviews are mixed and some of the concerns, mainly ones that are related to the presentation, remain after the authors' response. Due to the paper's strengths (good motivation, interesting system, promising even if imperfect evaluation), we have decided to recommend acceptance. Please carefully follow the reviewers' comments to improve the writing for the camera ready version.
Decision: Accept