Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ### Summary

This paper proposes a variant of multi-instance learning (MIL) for biomedical entity extraction called abstractified multi-instance learning (AMIL), which uses entity type information to construct bags of instances. In the paper, the authors first point out that a long-tail distribution of fact triples (i.e., (entity1, relation, entity2)) diminishes the advantage of the standard MIL since many bags might end up with duplicated instances (i.e., heavy upsampling is required to fill all the bags). This is particularly a problem in the biomedical data given that 50+% of fact triples appear in less than three sentences in PubMed. Instead, AMIL groups training instances by UMLS semantic types, which are shared across different entities. This approach results in more diverse instances in the bags, maximizing the benefit of MIL. For evaluation, the authors construct their own training/dev/test split from PubMed and UMLS. The BioBERT-base RE models are trained with AMIL and compare with the prior work (Amin et al., 2020) retrained on the same data. The main results show that AMIL models outperform the baseline by large margin. The authors further investigate performance on rare/common triples and 17 different relationship representations.


### Overall Review

Overall, this paper is clearly written and carefully explains the proposed methods and motivations behind them. The idea of AMIL is quite simple, but backing off to more general information is a clever solution to alleviate the data sparsity issues in biomedical RE data. Evaluations are fair, and the strong experimental results support the AMIL’s effectiveness. It’s interesting to see that diverse bags by AMIL (using type-level information) can gain over the standard MIL (using entity-level information) without losing performance. In addition, improvement on rare triples indicates that having diverse sentences in bags is crucial.  Although extensive architecture search on relationship representations is useful, it’s hard to see why “L” works best among others given the model’s black-box nature. The instance grouping is briefly mentioned in Section 4.1, but more detailed discussion would be helpful for reproducibility.


### Pros

1. Proposing AMIL that shows advantages on biomedical RE.
2. Experimental setup seems to be fair and appropriate. Analyzing performance on rare/common triples is informative.
3. Results are quite strong and promising. 


### Cons

1. AMIL assumes an access to high-quality KBs such (e.g., UMLS) which might not be available in some data settings.
2. Although instance grouping is a key step of AMIL, it is not discussed in detail.


### Questions / Suggestions

1.	> "The UMLS semantic network is curated by human experts for decades and provides a rich ontology of biomedical concepts...”

      How would you evaluate the annotation quality of semantic types? What’s the average number of types per entity? Are they always consistent? Or, it depends on the editor? It would be nice to show some entities with types (possibly in appendix). 

2. When you group training instances, do you always use all type combinations? Have you investigated good/bad type combinations? Do you end up with similar bags if many entity pairs share the same type sets?

3. Have you tried your version of MIL baseline (matching everything except grouping)?


### Minor points

- I see “Amin et al.” in many places (missing (2020)).
- In Table 2, the numbers from Dai et al. (2019) and Amin et al. (2020) are not directly comparable with your numbers (correct?). It would be nice to say it in the caption explicitly.

review_title: Nice extension of multi-instance learning for biomedical RE supported by strong results

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents explores the benefits of abstractifying relationships, i.e., taking into account the type signatures of its instances, to improve relation extraction from biomedical text corpora. The paper applies this technique on a state-of-the-art method for relation extraction based on BioBERT. The experimental evaluation on a handful of relation representations suggests that abstractified multi-relational extraction with middle mention attention pools and entity end markers outperforms the state of the art, and is particularly useful for tail relationships, i.e., relationships for which there are not many sentences that assert them. The results show that for those triples, both the aforementioned techniques provide a performance boost (Table 3).

The paper is overall easy to read and the experimental results are encouraging. The experimental evaluation is also comprehensive. I have nevertheless a few minor remarks.

- Problem formulation

I think the paper lacks a clear and proper problem formulation, e.g., something like, given a text corpus T and a knowledge graph G, our goal is to extract a set X of triples, etc., etc. I managed to have a clearer picture of the problem rather late (when reading the experiments). Readers of the paper may benefit from an early formal problem formulation in order to know to which extent they could apply the techniques proposed in the paper.

- Datasets

While Section 3 suggests that the relationships "narrower" and "broader" were excluded for extraction, the paper enhances the underlying embedding model with abstractified triples, i.e., predicate signatures. Where do exactly those signatures come from? Is there an rdf:type/type predicate for entities?

Moreover, why did the authors exclude the relationships  "narrower", "broader", and "synonymous". Is it because they are trivial to extract? I think the paper should make clearer the rationale behind this decision.

Also, the paper should provide some hints of the quality of the corpus annotation process. How accurately are the entity mentions annotated? (Section 4.1). Studying this aspect might be interesting and useful for settings where structured knowledge is scarce and/or noisy.

- Evaluation

-- The paper states: "We also compare to Dai et al.(2019) but, due to a difference in data, comparisons to AUC and F1 are not possible but we are able to compare precision on the top K predictions."
I would recommend the authors to provide a more detailed justification.

-- The difference between corpus-level and sentence-level evaluation is not clear. Is this difference only based on the distinction between rare and common triples?

-- A website or git repository with access to the data and source code would be great!
review_title: Valid contribution and encouraging results

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
This papers studies distantly supervised biomedical RE, where the data have a long-tail distribution. 
The authors hypothesize that formulating bags using distinct entity pairs results in many bags with too few sentences, which may hinder performance. 

The authors then propose abstractified multi-instance learning (AMIL) that formulates bags using UMLS semantic type (i.e., biomedical named entity type). AMIL outperforms the previous state-of-the-art method in Amin et al. 2020.  Most of the performance gain comes from "rare" tuples, validating the hypothesis in the beginning. The author also empirically compared 17 different methods to construct the representation, where they find the type L achieves the best performance.

Strength
- The proposed method AMIL is a intuitive and effective extension to MIL targeting long-tail data distribution in distantly supervised RE. AMIL outperforms previous state-of-the-art in Biomedical RE.
- The authors did a careful analysis on pre-processing and construction of representation. These small decisions actually influence performance a lot. Such empirical advice can be useful for other researchers.
- The paper is well-organized. The figures are clear and informative.

Weakness
- Dataset creation and evaluation protocol may have defects. See the question below.
- The work may be further strengthened with additional baselines. Since AMIL benefits from the UMLS semantic types, one baseline method I can think of is to include trainable embeddings for these types and include them in constructing relation representation (which is used in pre-BERT methods such as PCNN). 

Question
- Since the construction of bag is different in AMIL, is it possible that sentence-level attention (Lin et al., 2016) becomes beneficial for AMIL?
- I wonder whether sentence-level evaluation is reasonable and whether it gives a good estimate of the performance. The test set is taken from a bigger dataset constructed with distant supervision, so the test data can still be very noisy. 
- I'm not sure if listing performance from Dai et al. (2019) and Amin et al. (2020) is reasonable. The dataset construction in this paper seems to be different, so the numbers are not comparable.
review_title: Interesting contribution on distantly supervised relation extraction for biomedical domain

Meta-review: This paper proposes a variant of the canonical multi-instance learning (MIL) paradigm called abstractified MIL (AMIL), motivated in particular by the task of biomedical relation extraction. This paper presents a simple but clever approach to handling the "long tail" / data-sparse difficulties of distantly-supervised biomedical RE, which is clearly explicated and empirically evaluated, and which presents strong results. While reviewers had some questions around the dataset generation and evaluation protocols, and a few other minor aspects, the reception was overall very strong.
Decision: Accept