One-shot to Weakly-Supervised Relation Classification using Language Models
Year: 2021

Abstract:
Relation classification aims at detecting a particular relation type between two entities in text, whose methods mostly requires annotated data. Data annotation is either a manual process for supervised learning, or automated, using knowledge bases for distant learning. Unfortunately, both annotation methodologies are costly and time-consuming since they depend on intensive human labour for annotation or for knowledge base creation. With recent evidence that language models capture some sort of relational facts as knowledge bases, one-shot relation classification using language models has been proposed via matching a given instance against examples. The only requirement is that each relation type is associated with an exemplar. However, the matching approach often yields incorrect predictions. In this work, we propose NoelA, an auto-encoder using a noisy channel, to improve the accuracy by learning from the matching predictions. NoelA outperforms BERT matching and a bootstrapping baseline on TACRED and reWiki80.

heading: 1. Introduction
text: Relation classification (RC)1 detects the connection type between two entities in text such as place of birth between “Murat Kurnaz” and “Germany”, Figure 1. It is crucial for downstream applications such as knowledge base construction [Ji and Grishman, 2011] and question answering [Xu et al., 2016].
Most work in RC relies on either manually- or automatically-annotated data using knowledge bases (KBs) [Zhang et al., 2017, Mintz et al., 2009, Riedel et al., 2010]. This dependency leads to difficulty in generalising such methods to novel domains where labelled data or knowledge bases are not available. Recent work attempts to address the low resource scenario, namely few-shot relation extraction [Han et al., 2018, Baldini Soares et al., 2019], which requires few examples per relation type during testing. However, they rely on a large available training and validation sets to learn the classification task. Perez et al. [2021] suggests to name this setting, multi-distribution few-shot learning.
∗ This author is currently employed by the Ubiquitous Knowledge Processing (UKP) Lab, Technische Universität Darmstadt. (thytran@ukp.informatik.tu-darmstadt.de)
1Relation classification is Relation extraction with a predefined set of relation types, which assumes a relation hold between the entities.
In contrast, our work evaluates the extreme setting when no separate training and development sets are given. Our setting considers only one example from each relation type, namely relation exemplar. We employ the matching model in Baldini Soares et al. [2019] to compute the similarities between an input sentence and those exemplars. After gathering all the scores, we assign the relation with the highest score to the input sentence, such as country of birth in Figure 1.
The above LM-based one-shot relation predictions are often incorrect. To improve the performance, we propose to learn from these predictions, which we name noisy annotations. We propose NoelA (short for Noisy Channel Auto-encoder), which employs two mechanisms to alleviate the negative impact of noisy labels. Firstly, as entity types have been shown to be helpful for RC [Hancock et al., 2018, Ma et al., 2019, Tran et al., 2020], NoelA reconstructs entity types of the two input entities so that the entity type bias is used when predicting relations. Secondly, we use a noisy channel [Sukhbaatar et al., 2014, Goldberger and Ben-Reuven, 2016, Wang et al., 2019] to explicitly model the prediction noise.
We conducted experiments on two relation classification datasets: TACRED [Zhang et al., 2017] and reWiki80, a modification of Wiki80 [Han et al., 2019]. The two datasets have significantly different characteristics: the relation type distribution (skewed and uniform), the number of relation types (41 and 80), and the domain (news and Wikipedia). We show the performance of different LMs as baselines in order to verify our LM choice. We also demonstrate that bootstrapping [Reed et al., 2014], a traditional approach when supervision is scarce, is not effective. Meanwhile, NoelA outperforms the BERT matching by 9% accuracy on TACRED, and 6% on reWiki80. We conduct an extensive analysis to understand the biases captured in pre-trained LMs and the contributions of each component in NoelA.

heading: 2. Background
text: Relation classification (RC) is the task of assigning a (predefined) relation type to a pair of entities in a sentence. We denote R = {r1, r2, ..., rm} the set of m relation types. Given a sentence s of n words s = (w1, ..., wn), two entities h, t (called head and tail entities respectively), and their corresponding semantic types eh, et, the task is to identify the relation r ∈ R between the two entities. For instance, in Figure 1, h and t are “Murat Kurnaz” and “Germany”. Their entity types are eh = PERSON, et = LOCATION. The relation between them is r = country of birth. Our work focuses on predefined relation types excluding NA, i.e. “no relation”.
A parametric probabilistic relation classifier is given by pr(r|s, h, t; θ) (with parameters θ) which assigns a probability to a relation type r given a sentence s, head and tail entities h, t. Recent RC approaches adopt the diagram s, h, t→ encoder → classifier → r, converting 〈s, h, t〉 to a fixed-size vector representation before applying a classification layer (often a pair of linear-softmax over the relation type set R)
True One-shot Setting In this work, we assume the extreme scenario where only oneshot training set is available, i.e., a list of desired relation types and a single exemplar for each relation is given. For example, for the relation country of birth, its exemplar is “[Obama] was born in [the USA]”. Those exemplars are short and simple, often less than 10 words. Following Perez et al. [2021], we refer this scenario as true one-shot setting to differentiate with previous work [Han et al., 2018], in which a model has access to either separate training or development sets for tuning. Since our work does not aim at creating a new dataset, we use the predefined relation set of the evaluation datasets (TACRED and reWiki80) and manually create an exemplar for each relation type. By creating an exemplar for each relation, we can simulate the process of imagining artificial examples when we are given a set of relation types at hand. The relation types and corresponding exemplars used in our experiments are presented in Appendix C.
We note that our one-shot setting is very challenging even without detecting NA (no relation). Temporally leaving the problem of predicting NA is in line with the few-shot relation classification research. In particular, the few-shot setting initialised by Han et al. [2018] and further research following it [Baldini Soares et al., 2019] also ignore NA.

heading: 3. Matching Using Language Models
text: As shown in recent work, an LM trained on massive raw data (e.g., BERT) can capture some level of semantic similarity. For instance, “[A] is the mother of [B]” is more similar to
“[A] gave birth to [B]” than to “[A] works for [B]”. Therefore, we can use matching similarity computed by the LM to assign similar scores between an unseen sentence and the exemplars. Finally, the relation type assigned to the sentence is the one that has the highest score.
Let f be a function mapping sentence s and two entities h, t to a vector in Rd. We compute the matching score sim(f(s1, h1, t1), f(s2, h2, t2)), where sim is any function that computes the similarity between two vectors, such as dot product. f produces the vector of 〈s, h, t〉 in two steps, depicted in Figure 2a. We first compute the entity representations of h, t (i.e., xh,xt) by taking average of the words (w) that the mention contains, e.g., xh = avgwi∈h(wi). Then, we concatenate the two entity representations to form the relation candidate representation xs,h,t = [xh; xt]. We note that our BERT matching, i.e. when BERT is used, is similar to Baldini Soares et al. [2019]’s BERT with mention pooling.

heading: 4. Noisy Channel Auto-encoder (NoelA)
text: Because the prediction by LMs above is inevitably noisy, we propose NoelA (depicted in Figure 2b) to learn from noisy data.

heading: 4.1 Encoder
text: The encoder maps 〈s, h, t〉 to xs,h,t ∈ Rd. To compute a vector representation (e.g., xh) for each entity mention (e.g., h), we take mean pooling over the entity span from BERT. We then concatenate the two vectors of h and t with their entity types’ embeddings (xeh ,xet ∈ Rde), and apply a linear and a ReLU layers:
xs,h,t = ReLU(Linear(Concat(xh,xt,xeh ,xet))) (1)

heading: 4.2 Decoder
text: Differently from traditional decoders, our decoder does not completely reconstruct the input 〈s, h, t〉. It reconstructs the entity types eh, et of h, t only, and predicts the relation r hidden in the input by computing pr(r|s, h, t).
Relation Classifier After having a vector representation of 〈s, h, t〉, we apply a linear and softmax (over the relation type set R) layers to compute pr(r|s, h, t)
pr(.|s, h, t) = SoftmaxR(Linear(xs,h,t)) (2)
Noisy Channel Although a large-scale pre-trained LM is assumed to passively memorise some relational facts, the annotation using such model results in relatively noisy labels. We thus explicitly model the annotation noise by q(r′|r, s, h, t), the probability of assigning s to r′ given the correct relation type r. This probabilistic function is called “noisy channel” [Goldberger and Ben-Reuven, 2016]. Not knowing the correct r, we marginalise over the relation type set R:
p′r(r ′|s, h, t) = ∑ r∈R q(r′|r, s, h, t)pr(r|s, h, t) (3)
It is often assumed that r′ is independent from 〈s, h, t〉 given r, thus we base q(r′|r, s, h, t) = q(r′|r) on a matrix C ∈ R|R|2 :
q(r′|r) = exp(cr ′r)∑
r′′ exp(cr′′r) (4)
where cij is the entry of C at row i, column j. Initialising q(r′|r) has been shown crucial for learning p(r|s, h, t) [Goldberger and BenReuven, 2016]. We initialise C with a matrix computed by the confusion of choosing relation types by the used LM matching. Formally, let count(r′, r) be the number of times when r′ 6= r appear together in the top-k candidate lists for all sentences, then
cr′r = log count(r′, r)∑ r′′ count(r ′′, r) (5)
This initialisation provides to the learning an idea of how much the LM is confused r with r′. For instance, country of birth and country of death are likely confusing (because in the past an average person often died and was born in the same city/town). On the other hand, country of birth and spouse are easy to distinguish one from the other. In our experiments, we did not fine-tune q(r′|r) and chose k = b|R|/4c.
Entity Type Reconstruction Another way to tolerate the annotation noise is to inject into the model useful biases. Our encoder uses the entity types of h, t to compute the vector representation xs,h,t, because entity types have been shown to be helpful for RC [Hancock et al., 2018, Ma et al., 2019, Tran et al., 2020]. Intuitively, relation types are constrained by entity types, e.g., country of birth is constituted by an object such as person and a location. However, if trained on noisy labels only, the model may not be able to make use of entity types to tolerate the noise. Therefore, we force the model to capture the entity type bias by reconstructing the entity types of h, t. Formally speaking, denoting E the entity type set (e.g., E = {PER,LOC,ORG,MISC}), we compute the reconstruction probability using a linear and a softmax (over E × E) layers:
pe(.|s, h, t) = SoftmaxE×E(Linear(xee)) (6)
where xee = ReLU(Linear(xs,h,t)) ∈ Rdee .

heading: 4.3 Learning
text: Given D a noisy dataset, we train NoelA by minimising the following loss:
L(θ) = LNC(θ) + LETR(θ) + λLDR(θ) (7)
where LNC is the negative log-likelihood of predicting the noisy labels
LNC(θ) = − 1 |D| ∑
〈s,h,t,r′〉∈D
log p′r(r ′|s, h, t; θ); (8)
LETR is entity type reconstruction loss, which is the negative log-likelihood of predicting the entity types
LETR(θ) = − 1 |D| ∑
〈s,h,t,r′〉∈D
log pe(eh, et|s, h, t; θ); (9)
LDR is the dispersion regularisation term proposed by Simon et al. [2019] and λ ∈ R is its coefficient. A motivation behind the use of this regulariser is that relation distributions of the noisy data are peaky (Figure 3). Learning from such peaky distributions may lead the model biased towards frequent predicted relations and result in predicting a subset of the relation types. To prevent this issue, we employ the dispersion regulariser that encourages the model to predict a diverse set of relations. Following Simon et al. [2019], we set λ to 0.01 in all of our experiments.

heading: 5. Experiments
text: Our implementation was developed using the Transformers library [Wolf et al., 2019] and PyTorch [Adam et al., 2017].2 We use accuracy as evaluation metric. 3

heading: 5.1 Settings
text: Datasets We conducted experiments on two English datasets TACRED [Zhang et al., 2017] and reWiki80 whose statistics are shown in Appendix A. TACRED is a widely used dataset for supervised relation extraction, in which we removed the no relation instances. reWiki80 is a rearranged variant of the Wiki80 dataset used in Han et al. [2019], originated from FewRel [Han et al., 2018]. Since the test set of Wiki80 is not provided, we used the development set for testing. We took 20% of the training data as the development set for analysis.4 We tagged the data sets using the Stanford NER tagger [Manning et al., 2014].
The two datasets are significantly different in several aspects. reWiki80 has almost double the relation types than TACRED (80 vs 41). TACRED’s relation distribution is skewed while reWiki80’s is uniform. TACRED’s sentences are from news, and reWiki80’s are from Wikipedia.
For each dataset, we manually created a data-agnostic exemplar for each relation in which head and tail entities were randomly selected and mostly unseen in the two datasets. Considering Wiki80, 51.72% entities in our examplars are seen in the dataset but only 1 pair of entities occur in the training dataset. Regarding TACRED, 28.81% of entities are seen in the dataset but no pairs of entities in the exemplars occur in the training set. We use BERT to generate weak labels on the two training sets, namely Noisy Data. The labels originally given in the datasets are Gold Data.
Pre-trained LMs We examined the base versions of three LMs: BERT [Devlin et al., 2019], GPT2 [Radford et al., 2019], and SpanBERT [Joshi et al., 2020]. The used BERT version is uncased while the used GPT2 and SpanBERT are cased. Note that the BERT matching is similar to the BERT mention pooling model for one-shot RC proposed by Baldini Soares et al. [2019].5
2The source code is available at https://github.com/ttthy/noela. 3Accuracy and F1-score are equivalent without a negative class (no relation). 4This is acceptable since the gold relations are unseen during training and tuning the model. 5We apply mean pooling rather than max pooling because the former outperforms the latter in our preliminary experiments. Although Baldini Soares et al. [2019] show that BERT with entity markers achieves the best performance, it is unclear how the embeddings of entity markers are initialised.
Relation Classification Settings The hyper-parameters of our relation classifiers are given in Appendix A. 6 We used the Adam optimiser [Kingma and Ba, 2014] with a widely used learning rate of 3.10−4, and early stopping based on the accuracy over the exemplars with a patience of 5. We note that most of the hyperparameters were taken from previous work as the setting does not allow us to use the gold annotations of the development sets for fine-tuning. Besides, the performance on the exemplar set is not able to reflect the generalisability of a model.
Models in Comparison We compare NoelA against several baselines. The first two are Random (randomly assigning relation types) and Frequency (choosing the most frequent relation types). The others are three LMs (including GPT2-small [Radford et al., 2019], SpanBERT-base [Joshi et al., 2020], BERT-base [Devlin et al., 2019]) and bootstrap-hard [Reed et al., 2014, a bootstrapping approach;]. The bootstrapping model used in our experiments is proposed by Reed et al. [2014], namely Bootstrap-hard. The idea is to consider the relation type predicted by the current classifier for an instance at each training step as the noisy label. In particular, we employ the encoder and relation classifier from Section 4 (main text) with the loss Lbootstrap−hard(θ). The loss is a combination of LNC(θ) and Lmodel(θ) (Eq. (8) and Eq. (11), respectively), the negative log-likelihood loss of the label predicted by the model with the current θ.
Lbootstrap−hard(θ) = βLnc(θ) + (1− β)Lmodel(θ), (10)
where β is set to 0.8 following Reed et al. [2014] and Lmodel is computed as follows.
Lmodel(θ) = − 1 |D| ∑
〈s,h,t〉∈D r′=argmaxp(.|s,h,t;θ)
log p(r′|s, h, t; θ) (11)
We also tried their “soft” bootstrapping that minimises the entropy of the predicted label probability distribution H(p(.|s, h, t; θ)). However, the entropy regulariser caused the model collapsed. We thus did not include in our comparison.

heading: 5.2 Results
text: The results are shown in Table 1 (further details in Appendix B). In general, the BERT matching yields substantially higher accuracy than the baselines and other two LM matching methods on both datasets. It is worth noting that, the results of BERT in Table 1 are not comparable to those reported in Baldini Soares et al. [2019]. In particular, we take into account all relation types (41 in TACRED and 80 in reWiki), while they consider only 5 or 10 relation types for each testing example (N way K shot setting, §7).
NoelA substantially outperforms bootstrap-hard about 3-6%. Bootstrap-hard only performs on par with BERTwET, our NoelA without the two learning-from-noisy-labels mechanisms and the dispersion regularisation. This suggests that bootstrapping may not work on such noisy data where the seed set is too small, i.e., one example per category.
Removing the components one-by-one reduces the performance of our model. The entity type reconstruction (–ETR) has less impact on the reWiki80 test set, since it only reduces a
6For a fair comparison with the matching models, we did not fine-tune BERT during training in order to emphasise the contribution of our additional mechanisms instead of the large number of trainable parameters.
marginal score (0.69%). However, the contribution of the dispersion regulariser (–DR) is inconsistent on the two datasets.

heading: 6. Analysis

heading: 6.1 Relation Distribution
text: As the two datasets have different relation distributions, we firstly look at them and those yielded by BERT matching (Figure 3). In TACRED, the gold distribution is skewed towards a few relation types such as per:title, org:top members/employees. BERT matching however is in favour of infrequent ones such as org:shareholders, per:charges. In reWiki80, although the gold distribution is uniform, BERT matching’s distribution is multi-modal. This observation shows inappropriate biases of BERT matching, suggesting that one can improve the annotation by injecting inductive bias to BERT matching to make the predicted relation
distribution close to the gold. Furthermore, the difference between the gold distribution and the one yielded by BERT matching by some means explains the under-performance of the bootstrapping baseline as bootstrapping often suffers from semantic drift.

heading: 6.2 Accuracy of BERT Matching
text: We show the accuracy of BERT matching according to relation types in Figure 4. On TACRED, BERT matching performs exceptionally well for per:charges, per:age, but poorly for org:dissolved, org:subsidiaries. Although Figure 4a gives an intuition that the overall accuracy should be substantially higher than 15.46%, it is not the case because most frequent types have low accuracy. This observation again suggests the need for biasing BERT matching towards frequent relation types. On reWiki80, the highest accuracy is for mountain range, and low is for part of, subsidiary, operating system. Because the gold relation distribution is uniform rather than skewed, the overall accuracy of BERT matching on reWiki80 (27.48%) is substantially higher than that on TACRED (15.46%).
BERT matching’s confusion matrices are illustrated in Figure 5. The diagonal line is clearly shown for reWiki80 while it is lighter for TACRED. This explains the matching
performance on TACRED is low, as BERT gets confused by other relations. Generally on both datasets, we observe that BERT matching performs poorly for human-human relations such as parent, mother, sibling, and spouse. The reason is that these relations usually occur in the context of each other. BERT also performs worse on human-location relations such as residence are citizenship that are often hold between the same pair of entities. Meanwhile, place of birth/death are typically the same place in the past, causing the confusion between the two relations.

heading: 6.3 Accuracy of NoelA versus BERT Matching
text: Next, we show the accuracy difference of NoelA in comparison with BERT matching in Figure 6. The improvement of NoelA over BERT matching is mediocre on TACRED but visible on reWiki80. Nevertheless, the overall accuracy gain of NoelA on TACRED (9.33%) is substantial. This is due to the skewness of the gold relation distribution of TACRED (Figure 3a): a slight improvement for highly frequent relation types would lead to a substantial overall accuracy improvement.
The observation here indicates an interesting behaviour of NoelA: it seems to adjust its attention according to the hidden gold relation distribution. On TACRED, NoelA trades off the accuracy loss for some infrequent relation types against the accuracy gain for some frequent ones. On reWiki80, NoelA, on the other hand, pays attention to most relation types since all of them are equally frequent.

heading: 6.4 Impact of Entity Type Reconstruction
text: We examine to what extend entity types can help to predict gold relations. To do so, we measure the mutual information between entity type pairs (ET) and gold relations (R), and between gold relations (R) and gold relations (R). In information theory, given two random variables X,Y , the mutual information I(X,Y ) measures the amount of information in X that tells us about Y and vice versa. Therefore, the more helpful to predict relations (R) entity types (ET) are, the larger I(ET,R) is. Because the maximum value of I(ET,R) is I(R,R), we propose the below normalisation
Î(ET ;R) = I(ET ;R)
I(R;R) ∈ [0, 1] (12)
If entity types are not related to gold relations, I(ET ;R) = 0; thus Î(ET ;R) = 0. Otherwise, if gold relations are determined by entity types, I(ET ;R) = I(R;R), leading to Î(ET ;R) = 1.
For TACRED Î(ET ;R) = 0.81 whereas Î(ET ;R) = 0.33 for reWiki80. This explains why the impact of the entity type construction on reWiki80 is substantially smaller than on TACRED.

heading: 7. Related Work
text: Probing pre-trained LMs Prior work suggests that pretrained LMs capture factual information that they have seen during pretraining. A few studies have been introduced to extract such factual knowledge from LMs including relations between entities [Petroni et al., 2019, Jiang et al., 2020]. These studies define templates for each relation type, which are used in combination with a given subject to predict the corresponding object. While Petroni et al. [2019] manually define such templates, following studies propose few techniques for automating prompt generation and selection [Jiang et al., 2020, Shin et al., 2020, Gao et al., 2021]. A subsequent line of research is to exploit pre-trained LMs for weak supervision, such as Schick and Schütze [2020] 7 and our two baselines (bootstrap-hard and BERTwET, see Table 1). Our full work goes beyond that by employing learning-from-noisy-label techniques, resulting in substantial improvements.
Few-shot relation extraction Few-shot relation extraction have been introduced firstly by Han et al. [2018], following the N way K shot setting. In particular, N unseen relation types and their corresponding K examples are provided during evaluation. This setting assumes access to data from other distributions during training, which allows a model to learn the target task. Moreover, N does not cover the full set of relation types but a subset of it, e.g., N usually equals to 5 or 10. Another setting, requiring less supervision, is the BERT-based one-shot RC, proposed by Baldini Soares et al. [2019]. They evaluate BERT-based relation matching model in N way one-shot setting without using any training data. Their model, however, uses the development set for tuning the hyperparameters. Different from the both settings, our few-shot is true in the sense proposed by Perez et al. [2021]: neither training sets or development sets exist.

heading: 8. Conclusion and Future Work
text: We demonstrated one-shot relation classification using LMs by generating noisy relation data. To reduce the impact of noisy labels, we proposed NoelA (Noisy Channel Auto-encoder) which can learn the latent correct labels by explicitly modeling noise and using entity type bias. NoelA gains a promising 6 and 9% accuracy over BERT on reWiki80 and TACRED, respectively, demonstrating the potential of using weak supervision from LMs. Interestingly, we observed from the analysis of NoelA’s accuracy that our model can adjust towards the latent gold relation distribution. We note that no relation is important for relation classification, hence leave it for future work. We also believe that our NoelA is applicable to few-shot because the noisy channel idea is independent on number of examples per relation. In theory, the more examples we have, the better we can denoise the annotation.
7Because of using an ensemble of LMs for less-noisy annotations, this method requires significant more computation and memory than our NoelA. Creating less-noisy annotations like this is orthogonal to our approach, and left for future work.

heading: Appendix A. Experimental Settings
text: Table 2 shows the statistics of TACRED and reWiki80 datasets. We used exemplars as the development set and stopped the training process based on the accuracy on it. For every model, we conducted five runs with different initialised parameters and computed the average performance. We list the hyper-parameters of NoelA and bootstrap-hard in Table 3. We note that a small number of instances were eliminated when training the models due to max length constraint. The numbers of removed instances in TACRED [Zhang et al., 2017] train/dev/test sets are as follows: 148, 47 and 20 instances, respectively. There is no instance beyond the restricted length in reWiki80 [Han et al., 2019]. Additionally, regarding entity type embeddings, we distinguish the entity types of subject and object, e.g., PERSION-SUBJ and PERSION-OBJ. All experiments were performed on a compute node which has an Intel Skylake CPU and an NVIDIA V100 GPU (16GB GPU RAM).

heading: Appendix B. Detailed Results
text: We also report average accuracy over five runs on TACRED and reWiki80 development and test sets in Table 4 with the average training time (Avg. Runtime; minutes). We note that we do not use the development sets during training or for early stopping, we only use them for analysis.

heading: Appendix C. Exemplars
text: We present all the exemplars used for TACRED and reWiki80 in Table 5 and Table 6, respectively. All exemplars are manually created by one author and partially revised by another author.

Review summaries:
This paper tackles the semi-supervised relation classification problem with only one manually annotated example for each relation. The method is largely based on the BERT matching model (Baldini Soares et al., 2019). The authors propose to (1) also take entity types as input, (2) use BERT matching to get noisy labels on the unlabeled data, (3) take a noisy channel and an entity type reconstruction objective to train the model on the self-labeled data. The setting is new, so the authors compared their method to BERT matching and a previous bootstrap method. The improvement compared to those selected baseline models is significant, but the performance is in general very low and there is a large gap to the supervised model. There is an ablation study showing the effectiveness of the entity type reconstruction and the noisy channel. 

This paper proposes a weakly supervised relation classification model using pre-trained language models (LM). It considers the very challenging setting in which only one example of relation type is available. However, LM matching-based predictions which align a test sentence to a relation exemplar, are usually incorrect. This paper proposes to learn from those noisy labeling. 

This paper considers the problem of one-shot relation classification, in which a model is given a sentence with two entities identified, along with the entity types. The model must predict the label of the relationship, given only unlabeled data and a single labeled example from each possible relationship label. The proposed technique, NoelA, first uses weak supervision by labeling the unlabeled data to train a classifier. Each unlabeled example is embedded with a language model such as BERT and given the label of the nearest neighbor among the labeled examples. Then, the weak labels are used to train a multi-task model. One task head predicts the relationship type. The other is an auto-encoder task that predicts back the entity types of the entities in the sentence. The model also includes dispersion regularization.