Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: I like the mission of this nearly 2-decade old project and this seems like a reasonable thing to try, an epsilon increment on the idea of populating a class-level graph for the purposes of supporting a textbook. All told, they take a bunch of existing ideas and tack them on to something that is already there, integrating feedback into the tool is a great idea in particular.

However the paper reads like a progress report or masters project summary, and comes to no conclusion about the methods other than simply offering some metric numbers.  It describes what has been done, but has no hypothesis that was tested.  For this to be science,  we'd need at the very least a comparison to some baseline, an evaluation of different approaches and comparison of their advantages and disadvantages.
review_title: Interesting ideas, not much science

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overall, I like this work. It is well written and easy to follow. It feels like there is still a fair amount of work to get to well-rounded conclusions, but this seems like a good stage to already make initial results available for the community. Here and there, the text could be more clear by only discussing what has been done already and relegating all future steps to the discussion at the end. A further major aspect that needs improvement is comparing the obtained results to baselines. My comments on several aspects are below:

One aspect which is always hard is to differentiate between instances/entities/individuals and classes/types/concepts. I think you are doing a good job at explaining that. But, I wonder whether it is really that important to distinguish the two.

You mention that 'extracting rule definitions is out of the reach of automated methods'. I think this is not completely true. I have seen presentations on methods for doing that. Perhaps they are not performing well enough for your purpose?

Selecting 20 out of the 100 available relations, based on you thinking they are 'confusing' or their low usage frequency is a weakness of your work. I would have liked to see you using these 100 relations and then gain an insight on whether what you would a priori thought to be confusing and low frequency relations, really lead to issues. This would be an interesting ablation study, rather than something to exclude from the outset.

In 3.1 you describe your model for ATE. It remains unclear to me whether you still free the weights of the BERT model while training for the prediction. If you only allow the weights of your softmax layer to be trained, you might not be taking the full advantage of the possibilities of your model.
In the same section you also mention you use 'standard techniques for optimizing'. Please provide more detail here. Small differences might have large effects. Also, I would hope you could share the full implementation of the work done as this would make it possible for others to run comparative studies.
Also, there are a lot of existing methods for this step you could compare with. 

I section 3.2, you mention you use sentences with two terms identified in the sentence. I do not get why this is limited to two and sentences with more terms are not included for this.

For your experiments I would have expected more attention paid to the creation of good baselines. For example systems like NELL have been used for relation extraction before and also systems like Yago have extensive implementations for relation extraction from text. Even using a simple baseline based on Hearst patterns (which you are aware about), would be expected. Besides, there are also specific works that use distributional semantics for automatic taxonomy induction. These would be only limited to some of the relations types you consider, but would still be very relevant.

In section 3.3, I think it is very nice that you include the option of flipping the relationship in the correct direction. Even nicer would be to see whether you can use that information to train a model and do this flipping task automatically.

One limitation of your work is that (co-)references are not handled.  This is in part because you feed your models one sentence at a time.  I think there is still an interesting direction where you would give your model longer pieces of text including coreferences and see whether it can figure those out while extracting the relations. 

In 4.3. You mention you employed biologists to check the correctness of the psychology text annotations as well. Did they consider annotating this data harder than annotating the biology text? Were there more inter-annotator disagreements?

In section 4.3, you describe your use of a 'sophisticated denoising algorithm' (with reference to 37 and 9). Please give a summary or at least an intuition on what that is doing in your paper.

In 3.2 and 5.2, you reduce your set further to only 6 relations, while earlier you mentioned 20. This becomes very limiting.


review_title: An interesting experiment on an interesting research topic

final_score: rating: 7: Good paper, accept
review_content: Strength: 
1. The labeling task is designed for the textbook reading by students across large libraries of textbooks to obtain plenty of data for relation selection.
2. The framework can handle diverse relationships that include taxonomic, structural, functional and causal relationships.  
3. This paper also considers SOTA BERT-based model for sub-tasks . 


Weekness:

The paper mentions active learning will be considered later, which indeed is a good next step for this work.

One related work can be cited.
"Constructing Taxonomies from Pretrained Language Models, Chen et al, NAACL 2021" 

review_title: This paper proposes to construct ontology graph by adapting BERT and leveraging a novel crowd sourced relationship selection task, which shows that this approach is able to bootstrap the ontology graph creation for further refinement and improvement through human effort.

Meta-review: This paper presents results of a project to bootstrap ontology graphs from student textbooks.

A major weakness of the approach is the limited comparison with prior work. Clearly, it is a non-trivial problem to compare such a complex system to prior work, given the number of different components involved, but more effort could have gone into it. On the plus side, the approach draws on pre-trained language models and thus shows some of the new possibilities compared to some of the work from the past.

The reviewers felt that the approach and results currently seem a bit premature, and various components could be improved. In light of this, the title of the paper should be made more specific to better describe this as a case study on specific kinds of data.

The authors have promised to make their dataset available, which should be valuable for the community.
Decision: Accept