Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The authors present Alexandria, a system for unsupervised, high-precision knowledge base construction. Alexandria uses a probabilistic program to define a process of converting knowledge base facts into unstructured text. The authors evaluate Alexandria by constructing a high precision (typically 97%+) knowledge base for people from a single seed fact. 

Strengths:

--although it is unusual to combine and intro and related work into a single section, I enjoyed the authors' succinct statement (which I agree with) about the 'holy grail of KB construction'. Overall, I think the writing of the paper started off on a strong note.

--the paper has a lot of promise in considering a unified view of KB construction. Although I am not recommending an accept, I hope the authors will continue this work and submit a revised version for future consideration (whether in the next iteration of this conference, or another)

Weaknesses:

--It is not necessary to be putting 'actual programs' (Figure 1). It does not serve much of a purpose and is inappropriate; the authors should either use pseudocode, or just describe it in the text. 

--In Web-scale fact extraction, the domain-specific insight graphs (DIG) system should also be mentioned and cited, since it is an end-to-end system that has been used for KGC in unusual domains like human trafficking. 

--Starting from Section 2, the whole paper starts reading a bit like a technical manual/specification document. This kind of paper is not appropriate for an academic audience; the paper should have been written in a much more conceptual way, with actual implementations/programs/code details relegated to a github repo/wiki, and with a link to the same in the paper.
review_title: Although the paper has a lot of promise, it is not suitable for publication in an academic conference in its present form, in my view. I advise the authors to review all the comments, and prepare a significantly revised version for future publication.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper uses a probabilistic program describing the process by which facts describing entities can be realised in text, and a large number of web pages, to learn to perform fact extraction about people using a single seed fact.

Despite the prodigious computational cost (close to a half million hours of computation to acquire a KB only about people) I found the scale at which this paper applied probalistic programming exciting.  It suggests that providing a structural prior in this form, and then learning the parameters of a model with that structure is a practical technique that could be applied widely.

Questions that arose whilst reading:  the most direct comparison was with a system using Markov Logic Networks, in which the structural prior takes the form of a FOL theory. A more direct comparison would have been useful - in particular, an estimate of the difficulty of expressing and equivalently powerful model, and the computational cost of trainining that model, in MLN.

Quite a lot of tuning was required to make training tractable (for outlying values of tractable) - this limits the likely applicability of the technique.

The paper suggests in future work an extension to open domain fact extraction, but it is not clear how complex or tractable the require prob program would be. The one in the paper is in some respects (types mainly) specific to the facts-about-people setting.

It is unclear why theTAC-KBP Slot Filling track  was mentioned, given that performance on this track does not seem to have been evaluated. An informal evaluation suggesting beyond SoA performance is mentioned, but not useful details given. This significantly weakens what otherwise could be a stand-out paper
review_title: Learning to extract facts with little prior except a structural prior from a probabilistic program

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: 
The paper is about a knowledge base that is constructed with a probabilistic model (Markov logic network, as implemented in Infer.NET). The system is expansive and covers many important aspects, such as data types and values, templates for extraction and  generation, noise models. The model addresses three kinds of large scale inferences: (1) template learning, (2) schema learning, and (3) fact retrieval.   The knowledge base construction approach is evaluated against other knowledge base approaches, YAGO2, NELL, Knowledge Vault, and DeepDive.

The paper is a very interesting overview over the knowledge base Alexandia, that would inspire interesting discussions at the AKBC conference. 

My only pet peeve are a set of initial claims, intended to be distinguishing this approach from others, which are simply not true:
-  "Alexandia's key differences are its unsupervised approach" -- clearly Alexandria requires prior knowledge on types, and furthermore weakly supervision for template learning etc. Unsupervised means "not require any clue about what is to be predicted". 
- "KnowledgeVault cannot discover new entities" -- I'd be surprised.
- "DeepDive uses hand-constructed feature extractors" -- It is a matter of preferences whether one seeds the extraction with patterns or data. This statement does not convince me.
While the discussion of these differences is important, I suggest the authors use more professional language.



review_title: Very interesting overview of an existing knowledge base, will inspire interesting discussions

Meta-review: metareview: The authors propose Alexandria, a probabilistic programming approach to AKBC. The core idea of the approach is to use a generative probabilistic program to model natural language expressions of facts using templates, and then reverse this process to learn new templates and properties from text. Evaluation for a small domain showed promising results.

The critical consensus was that this paper presents an interesting idea and should be accepted. There were several concerns about the representation of related work and some concerns about the evaluation and results, particularly the restriction to a single domain and the computational costs of the system, and criticism about the balance of high-level and low-level ideas in the writing. The authors have addressed some of these concerns in the rebuttal and in a draft revision.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)