Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper tackles the semi-supervised relation classification problem with only one manually annotated example for each relation. The method is largely based on the BERT matching model (Baldini Soares et al., 2019). The authors propose to (1) also take entity types as input, (2) use BERT matching to get noisy labels on the unlabeled data, (3) take a noisy channel and an entity type reconstruction objective to train the model on the self-labeled data. The setting is new, so the authors compared their method to BERT matching and a previous bootstrap method. The improvement compared to those selected baseline models is significant, but the performance is in general very low and there is a large gap to the supervised model. There is an ablation study showing the effectiveness of the entity type reconstruction and the noisy channel. 

Strength:

The proposed method is very intuitive (noisy channel and self-labeling) and the performance is strong (compared to the baselines).

Weakness: 

1. In the experiment, the authors discard all the "no relation" sentences. "No relation" or N/A data are the most challenging part of relation extraction/classification problems, and ignoring them makes it a very toy setting.

2. This one-shot + weakly-supervised setting is not well motivated. First, I didn't quite understand why the authors take the "one-shot" setting instead of taking the "few-shot" setting. Few-shot is in general more applicable in the real world. Second, the authors emphasizes that they are taking a "true few-shot" setting, but they actually take a "toy" weakly supervised setting, where they take the sentences from the supervised dataset, instead of taking an open corpus. Considering the authors also exclude the no-relation instances from the dataset, it makes the setting even easier.

3. This is a minor point: I feel the writing and the structure of the paper could be further improved. The setting is not clearly explained at the beginning. 

Question: 

1. I'm curious about why the entity type reconstruction can help improve the accuracy. The input already includes the entity types (in the case of TACRED, those are even gold entity types), then why does it matter to "reconstruct" the entity types? Shouldn't the model be able to directly use this information?
review_title: Solid improvement, but I'm skeptical about the setting

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the problem of one-shot relation classification, in which a model is given a sentence with two entities identified, along with the entity types. The model must predict the label of the relationship, given only unlabeled data and a single labeled example from each possible relationship label. The proposed technique, NoelA, first uses weak supervision by labeling the unlabeled data to train a classifier. Each unlabeled example is embedded with a language model such as BERT and given the label of the nearest neighbor among the labeled examples. Then, the weak labels are used to train a multi-task model. One task head predicts the relationship type. The other is an auto-encoder task that predicts back the entity types of the entities in the sentence. The model also includes dispersion regularization.

Experiments on TACRED and reWiki80 using three language models (BERT, GPT2, and SpanBERT). NoelA offers significant improvements over the baselines of using language models directly for labeling, or training a classifier alone on the weak labels. Ablations also show that the three components of the loss function all contribute to the positive results.

The combination of bootstrapping and multi-task learning is a nice experiment. It's a straightforward pipeline of two common techniques (unless learning to reconstruct entity types is more novel, see question below).

### Questions and comments
- Bootstrapping from one-shot supervision is nice, but how directly tied is it to the benefit of learning to reconstruct entity types. Do any of the references  [Hancock et al., 2018, Ma et al., 2019, Tran et al., 2020] also use an autoencoder component to learn from entity types?
- Something that could strengthen the paper is a discussion of how even a few labeled examples per class could fit into the framework. Some few-shot learning methods can easily handle a variety. Do the authors have an opinion about how best to apply NoelA in the few-shot setting?
- Regarding the claim "Therefore, we force the model to capture the entity type bias by
reconstructing the entity types of h, t." in Section 4.2, is there any way to know that this is really encoding a bias specific to entity types? How do we know, for example, it's not just a generic type of regularization?
- In section 5.1, it says " We manually created a data-agnostic exemplar for each relation in which head and tail entities were randomly selected and mostly unseen in the two datasets." It would be nice to clarify what "mostly unseen" means.
review_title: Solid Contribution to one-shot relation classification

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper proposes a weakly supervised relation classification model using pre-trained language models (LM). It considers the very challenging setting in which only one example of relation type is available. However, LM matching-based predictions which align a test sentence to a relation exemplar, are usually incorrect. This paper proposes to learn from those noisy labeling. 

Given a sentence and two entities (and their types), their model first encodes the sentences and then mean pools over the entity span. It then concatenates the entity representation with the representation of their types and passes through a feedforward network (with ReLU). Since the LM-based predictions are usually incorrect, the paper takes a noisy-channel approach in which they model the co-occurrence between the predictions of the noisy and the true label by the model. Since the correct relation is not known, they marginalize over the random variable representing true relation. The intuition behind this loss is that for relations that are easily confused by the model, the loss function increases the probability of the true relations as well (assuming LM-based predictions would be somewhat meaningful and will assign reasonable weight to the ground truth relation as well). Since entity types are helpful for relation classification, the paper has another objective for reconstructing the entity types of the two entities.

The paper conducts experiments on two datasets — TACRED and reWiki80 using multiple LMs (Bert, GPT2, SpanBert). The method achieves modest improvements over the BERT matching system (6% - 9%). However, the absolute performance is pretty low in general (24% for TACRED and 33% for reWiki80). The difference between a model trained with supervision is much higher (> 80% for TACRED and >70% for reWiki80), suggesting that their setup is indeed very hard. 

### Strengths:

1. This paper considers a very hard (true one-shot) setting for weakly supervised relation extraction. 
2. The noisy channel approach is pretty interesting and has not been applied to few-shot relation extraction before.
3. The paper is well written in general.

### Weakness:

1. Few modeling details are not well motivated (e.g. dispersion regularization term)
2. The bootstrapping baseline should be moved from the appendix to the main section of the paper.

### Question for Authors:

1. Side note: Please number the equations, it makes referring to them easier.
Since the equation for computing the probability of r' marginalizes over all relations, what is stopping from all the p(r | s, h, t) to increase? How do you ensure that the prob of only relations which has high q(r' | r) increase? Would it make sense to also include a term that decreases the entropy of the p(r | h, s, t), i.e. favors peaky distributions?
2. In the loss function, there is a dispersion regularization term. I think the paper lacks the detail regarding why that is required. What would cause the encoder to predict the same relation across all instances?
3. In the end of section 3, you mention a way of encoding the input as [x_h;x_t]. However, later in Sec 4, you redefine how the input is encoded (e.g. ReLU(Linear(Concat(x_h, x_t, x_eh, x_et)))). I am a little confused regarding the initial definition in Sec 3. Is it being used anywhere?
4. "The observation here indicates an interesting behaviour of NoelA: it seems to adjust its
attention according to the hidden gold relation distribution." —- Why do you think this is the case? How is NoelA adjusting performance wrt the gold relation distribution?
5. Suggestion: It would be informative to plot the C (or q(r'|r)) matrix to visualize if BERT also allocates reasonable scores (if not highest) to the correct relation. Edit (I see the confusion matrix is presented in Appendix D, but I think it could make sense to bring it to the main section of the paper.
6. Paper organization suggestion: I think the paper would be more convincing / readable if the analysis in sec 6.1 and 6.2 regarding Bert matching along with the confusion matrix (in Appendix D) is presented to motivate the modeling decision rather than at the end of the paper. Also, please move the details of bootstrapping to the main section of the paper. I am not sure if an additional page is allowed in the camera-ready. If not, to make up for the additional space, I believe the section 6.4 could be moved to the appendix. Similarly size of table 1 could be shortened (fontsize).
review_title: Review

Meta-review: This paper presents a method for one-shot relation classification. Experimental results demonstrate the effectiveness of the proposed method. The main concern of the reviewers was the limited zero-shot nature of the problem. In spite of that, we think there are enough value in the paper and hence recommend acceptance.
Decision: Accept