Venue: AKBC 2019 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an interesting combination of semi-supervised learning and ensemble learning for information extraction, with experiments conducted on a biomedical relation extraction task. The proposed method is novel (to certain degree) and intuitively plausible, but I'm not entirely convinced of its effectiveness. 

Pros:

- Novel combination of semi-supervised learning and ensemble learning for information extraction
- Good discussion of literature
- Paper is well written and easy to follow

Cons:

- Experimental design and results are not supportive enough of the effectiveness of the proposed method.

Specifically, I think the following problems undermine the claims:

- The experiments are conducted on a single dataset. I understand it may be hard to get more biomedical RE datasets, but the proposed method is in principle not limited to biomedical RE (even though the authors limited their claim to biomedical RE). In general I think it needs multiple datasets to test a generic learning methodology.

- Only one split of training data is used, and the unlabeled data to labeled data are close in scale. It would be really informative if there are experiments and performance curves with different labeled subsets of different sizes. Also, (after under-sampling) 4500 unlabeled vs. 2000 labeled seems less impressive and may be not very supportive of the usefulness of the proposed method. What will happen if there are only 200 labeled examples? 400?

- No comparison with other semi-supervised learning methods for information extraction, so it's not clear how competent the proposed method is compared with other alternatives. 

- The fact that the mean base learner performance is often on par with the meta-learner, and simple majority vote of the weak labels can largely outperform the meta-learner may suggest a better meta-learner other than LSTM should have been used.
review_title: An interesting paper in methodology, but I'm not entirely convinced by the experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to semi-supervised method for relation classification, which trains multiple base learners using a small labeled dataset, and applies an ensemble of them to annotate unlabeled examples for semi-supervised learning. They conducted experiments on a BioCreative shared task for detecting chemical-protein interactions from biomedical abstracts. The experimental results suggest that ensemble could help denoise the self supervised labels. Overall, this is an interesting direction to explore, but the paper can also improve significantly in several aspects.

First, the discussion about related work is a bit lacking and at place slightly misguided. The authors seem to be unaware of a growing body of recent work in biomedical machine reading using indirect supervision, e.g., Deep Probabilistic Logic (EMNLP-18). These methods have demonstrated success in extracting biomedical relations across sentences and beyond abstracts, using zero labeled examples. Likewise, in discussing semi-supervised approaches and cross-view learning, the authors should discuss their obvious connections to recent progress such as Semi-Supervised Sequential Modeling with Cross-View Learning (EMNLP-18) and EZLearn (IJCAI-18).

Additionally, the authors seem to pitch the proposed method against standard weak supervision approaches. E.g., the paper stated "For the creation of the weak labels, we use classifiers pretrained in a small labeled dataset, instead of large Knowledge Bases which might be unavailable." Effectively, the authors implied that distant supervision is not applicable here because it requires "large knowledge bases". In fact, the precise attraction of distant supervision is that knowledge bases are generally available for the relations of value, even though their coverage is sparse and not up-to-date. And past work has shown success with small KBs using distant supervision, such as recent work in precision oncology:
	- Distant Supervision for Relation Extraction beyond the Sentence Boundary (EACL-17)
	- Cross-Sentence N-ary Relation Extraction with Graph LSTMs (TACL-17)

A labeled dataset contains strictly more information than the annotated relations, and arguably is harder to obtain than the corresponding knowledge base. Some prior work even simulated distant supervision scenarios from labeled datasets, e.g., DISTANT SUPERVISION FOR CANCER PATHWAY EXTRACTION FROM TEXT (PSB-15). These won't detract significance in exploring ensemble learning, but the proposed direction should be considered as complementary rather than competing with standard weak supervision approaches.

The paper should also discuss related datasets and resources other than BioCreative. E.g., BioNLP Shared Task on event extraction (Kim et al. 2009) is influential in biomedical relation extraction, followed by a number of shared tasks in the same space.

Another major area that can be improved is the technical and experimental details. E.g.:

The ensemble of base learners is key to the proposed method. But it's unclear from the paper how many base learners have been considered, what are their types, etc. At the high level, the paper should specify the total number of candidate base learners, distribution over major types (SVM, NN, ...), the types of chosen centroids, etc. For each type, the paper should include details about the variation, perhaps in supplement. E.g., SVM kernels, NN architectures. 

Table 3 shows the mean F1 score of base learners, which raises a number of questions. E.g., what're the top and bottom performers and their scores? If the mean is 53, that means some base learner learning from partial training set performs similar to LSTM learning from the whole training set, so it seems that out right some learner (supposedly not LSTM) is more suited for this domain? It's unclear whether LSTM is included in the base learner. If it was, then its low performance would mean that some top performers are even higher, so the ensemble gain could simply stem from their superiority. In any case, the lack of details makes it very hard to assess what's really going on in the experiment.

A key point in the paper is that high-capacity methods such as LSTM suffers from small dataset. While in general this could be true, the LSTM in use seems to be severely handicapped. For one, the paper suggested that they undersampled in training LSTM, which means that LSTM was actually trained using less data compared to others. They did this for imbalance, but it is probably unnecessary. LSTMs are generally less sensitive to label imbalance. And if one really has to correct for that, reweighting instances is a better option given the concern about small training set to begin with.

The paper also didn't mention how the word embedding was initialized in LSTM. If they're randomly initialized, that's another obvious handicap the LSTM suffers from. There are publicly available PubMed word2vec embedding, not to more recent approaches like Elmo/Bert. 

Minor comments:

In Table 3, what does it mean by "F1 of weak labels"? Does it mean that at test time, one compute the ensemble of predictions from base learners?

The paper uses D_G in one place and D_B in another. Might be useful to be consistent.

"Text trimming" sounds quite odd. What the paragraph describes is just standard feature engineering (i.e., taking n-grams from whole text vs. in between two entities).

Fig 3: it's unclear which line is which from the graph/caption. One has to guess from the analysis, 

Fig 5: what was used in tSNE? I.e., how is each instance represented? It's also hard to make any conclusion from the graph, as the authors' tried to make the point about distributions.

review_title: The paper proposes a potentially interesting direction, though needs to improve in many aspects.

final_score: rating: 7: Good paper, accept
review_content: -summary
This paper addresses the problem of generating training data for biological relation extraction, looking specifically at the biocreative chemprot dataset. The authors weakly label data by using a set of weak classifiers, then use those predictions as additional training data for a meta learning algorithm.

-pros
- nicely written paper with good exposition on related works
- good analysis experiments

-cons
- experiments all on a single dataset
- methodology isn’t very novel

-questions
- In table 3, majority vote of weak classifiers outperforms the meta learning. Are these two numbers comparable and if so, does this mean an ensemble of weak classifiers is actually the best performing method in your experiments.


review_title: review 

Meta-review: metareview: Given a few base-learners, can we learn a meta-learner that performs better than the base-learners? The paper addresses this question in the context of bio-medical relation extraction where training data is often small. A number of base-learners are learned (SVM, LSTM, Logistic Regression etc) on a small training data which are further used to annotate a large amount of unsupervised data. Using data-programming techniques of Ratner et al. 2016, a set of n-weak labels are created. Then a discriminative model (which is called meta-learner here) is further trained on these weak labels to predict the final label.

The reviewers had many concerns on the initial version. Some of the main ones are below:

1) Reviewer 3 suggests using an additional dataset.
2) Reviewer 1 did a great job in suggesting plenty of literature.
3) Reviewer 1 also suggested making several empirical results clear.

The revised version addresses these concerns with several additions. I am satisfied with the revisions, and I believe the paper will be a good addition to the conference. Based on the revised version, I recommend accepting the paper.

A suggestion for improvement: The paper relies a lot on the data programming work of Ratner et al. 2016, 2017, which is explained at a high level in Section 3 and 4. A paper should be self-sufficient when it comes to understanding. Please describe these methods formally and in necessary detail. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)