One-shot to Weakly-Supervised Relation Classification using Language Models

1 . Introduction . Relation classification ( RC ) 1 detects the connection type between two entities in text such as place of birth between “ Murat Kurnaz ” and “ Germany ” , Figure 1 . It is crucial for downstream applications such as knowledge base construction [ Ji and Grishman , 2011 ] and question answering [ Xu et al. , 2016 ] . Most work in RC relies on either manually- or automatically-annotated data using knowledge bases ( KBs ) [ Zhang et al. , 2017 , Mintz et al. , 2009 , Riedel et al. , 2010 ] . This dependency leads to difficulty in generalising such methods to novel domains where labelled data or knowledge bases are not available . Recent work attempts to address the low resource scenario , namely few-shot relation extraction [ Han et al. , 2018 , Baldini Soares et al. , 2019 ] , which requires few examples per relation type during testing . However , they rely on a large available training and validation sets to learn the classification task . Perez et al . [ 2021 ] suggests to name this setting , multi-distribution few-shot learning . ∗ This author is currently employed by the Ubiquitous Knowledge Processing ( UKP ) Lab , Technische Universität Darmstadt . ( thytran @ ukp.informatik.tu-darmstadt.de ) 1Relation classification is Relation extraction with a predefined set of relation types , which assumes a relation hold between the entities . In contrast , our work evaluates the extreme setting when no separate training and development sets are given . Our setting considers only one example from each relation type , namely relation exemplar . We employ the matching model in Baldini Soares et al . [ 2019 ] to compute the similarities between an input sentence and those exemplars . After gathering all the scores , we assign the relation with the highest score to the input sentence , such as country of birth in Figure 1 . The above LM-based one-shot relation predictions are often incorrect . To improve the performance , we propose to learn from these predictions , which we name noisy annotations . We propose NoelA ( short for Noisy Channel Auto-encoder ) , which employs two mechanisms to alleviate the negative impact of noisy labels . Firstly , as entity types have been shown to be helpful for RC [ Hancock et al. , 2018 , Ma et al. , 2019 , Tran et al. , 2020 ] , NoelA reconstructs entity types of the two input entities so that the entity type bias is used when predicting relations . Secondly , we use a noisy channel [ Sukhbaatar et al. , 2014 , Goldberger and Ben-Reuven , 2016 , Wang et al. , 2019 ] to explicitly model the prediction noise . We conducted experiments on two relation classification datasets : TACRED [ Zhang et al. , 2017 ] and reWiki80 , a modification of Wiki80 [ Han et al. , 2019 ] . The two datasets have significantly different characteristics : the relation type distribution ( skewed and uniform ) , the number of relation types ( 41 and 80 ) , and the domain ( news and Wikipedia ) . We show the performance of different LMs as baselines in order to verify our LM choice . We also demonstrate that bootstrapping [ Reed et al. , 2014 ] , a traditional approach when supervision is scarce , is not effective . Meanwhile , NoelA outperforms the BERT matching by 9 % accuracy on TACRED , and 6 % on reWiki80 . We conduct an extensive analysis to understand the biases captured in pre-trained LMs and the contributions of each component in NoelA . 2 . Background . Relation classification ( RC ) is the task of assigning a ( predefined ) relation type to a pair of entities in a sentence . We denote R = { r1 , r2 , ... , rm } the set of m relation types . Given a sentence s of n words s = ( w1 , ... , wn ) , two entities h , t ( called head and tail entities respectively ) , and their corresponding semantic types eh , et , the task is to identify the relation r ∈ R between the two entities . For instance , in Figure 1 , h and t are “ Murat Kurnaz ” and “ Germany ” . Their entity types are eh = PERSON , et = LOCATION . The relation between them is r = country of birth . Our work focuses on predefined relation types excluding NA , i.e . “ no relation ” . A parametric probabilistic relation classifier is given by pr ( r|s , h , t ; θ ) ( with parameters θ ) which assigns a probability to a relation type r given a sentence s , head and tail entities h , t. Recent RC approaches adopt the diagram s , h , t→ encoder → classifier → r , converting 〈s , h , t〉 to a fixed-size vector representation before applying a classification layer ( often a pair of linear-softmax over the relation type set R ) True One-shot Setting In this work , we assume the extreme scenario where only oneshot training set is available , i.e. , a list of desired relation types and a single exemplar for each relation is given . For example , for the relation country of birth , its exemplar is “ [ Obama ] was born in [ the USA ] ” . Those exemplars are short and simple , often less than 10 words . Following Perez et al . [ 2021 ] , we refer this scenario as true one-shot setting to differentiate with previous work [ Han et al. , 2018 ] , in which a model has access to either separate training or development sets for tuning . Since our work does not aim at creating a new dataset , we use the predefined relation set of the evaluation datasets ( TACRED and reWiki80 ) and manually create an exemplar for each relation type . By creating an exemplar for each relation , we can simulate the process of imagining artificial examples when we are given a set of relation types at hand . The relation types and corresponding exemplars used in our experiments are presented in Appendix C. We note that our one-shot setting is very challenging even without detecting NA ( no relation ) . Temporally leaving the problem of predicting NA is in line with the few-shot relation classification research . In particular , the few-shot setting initialised by Han et al . [ 2018 ] and further research following it [ Baldini Soares et al. , 2019 ] also ignore NA . 3 . Matching Using Language Models . As shown in recent work , an LM trained on massive raw data ( e.g. , BERT ) can capture some level of semantic similarity . For instance , “ [ A ] is the mother of [ B ] ” is more similar to “ [ A ] gave birth to [ B ] ” than to “ [ A ] works for [ B ] ” . Therefore , we can use matching similarity computed by the LM to assign similar scores between an unseen sentence and the exemplars . Finally , the relation type assigned to the sentence is the one that has the highest score . Let f be a function mapping sentence s and two entities h , t to a vector in Rd . We compute the matching score sim ( f ( s1 , h1 , t1 ) , f ( s2 , h2 , t2 ) ) , where sim is any function that computes the similarity between two vectors , such as dot product . f produces the vector of 〈s , h , t〉 in two steps , depicted in Figure 2a . We first compute the entity representations of h , t ( i.e. , xh , xt ) by taking average of the words ( w ) that the mention contains , e.g. , xh = avgwi∈h ( wi ) . Then , we concatenate the two entity representations to form the relation candidate representation xs , h , t = [ xh ; xt ] . We note that our BERT matching , i.e . when BERT is used , is similar to Baldini Soares et al . [ 2019 ] ’ s BERT with mention pooling . 4 . Noisy Channel Auto-encoder ( NoelA ) . Because the prediction by LMs above is inevitably noisy , we propose NoelA ( depicted in Figure 2b ) to learn from noisy data . 4.1 Encoder . The encoder maps 〈s , h , t〉 to xs , h , t ∈ Rd . To compute a vector representation ( e.g. , xh ) for each entity mention ( e.g. , h ) , we take mean pooling over the entity span from BERT . We then concatenate the two vectors of h and t with their entity types ’ embeddings ( xeh , xet ∈ Rde ) , and apply a linear and a ReLU layers : xs , h , t = ReLU ( Linear ( Concat ( xh , xt , xeh , xet ) ) ) ( 1 ) 4.2 Decoder . Differently from traditional decoders , our decoder does not completely reconstruct the input 〈s , h , t〉 . It reconstructs the entity types eh , et of h , t only , and predicts the relation r hidden in the input by computing pr ( r|s , h , t ) . Relation Classifier After having a vector representation of 〈s , h , t〉 , we apply a linear and softmax ( over the relation type set R ) layers to compute pr ( r|s , h , t ) pr ( .|s , h , t ) = SoftmaxR ( Linear ( xs , h , t ) ) ( 2 ) Noisy Channel Although a large-scale pre-trained LM is assumed to passively memorise some relational facts , the annotation using such model results in relatively noisy labels . We thus explicitly model the annotation noise by q ( r′|r , s , h , t ) , the probability of assigning s to r′ given the correct relation type r. This probabilistic function is called “ noisy channel ” [ Goldberger and Ben-Reuven , 2016 ] . Not knowing the correct r , we marginalise over the relation type set R : p′r ( r ′|s , h , t ) = ∑ r∈R q ( r′|r , s , h , t ) pr ( r|s , h , t ) ( 3 ) It is often assumed that r′ is independent from 〈s , h , t〉 given r , thus we base q ( r′|r , s , h , t ) = q ( r′|r ) on a matrix C ∈ R|R|2 : q ( r′|r ) = exp ( cr ′r ) ∑ r′′ exp ( cr′′r ) ( 4 ) where cij is the entry of C at row i , column j. Initialising q ( r′|r ) has been shown crucial for learning p ( r|s , h , t ) [ Goldberger and BenReuven , 2016 ] . We initialise C with a matrix computed by the confusion of choosing relation types by the used LM matching . Formally , let count ( r′ , r ) be the number of times when r′ 6= r appear together in the top-k candidate lists for all sentences , then cr′r = log count ( r′ , r ) ∑ r′′ count ( r ′′ , r ) ( 5 ) This initialisation provides to the learning an idea of how much the LM is confused r with r′ . For instance , country of birth and country of death are likely confusing ( because in the past an average person often died and was born in the same city/town ) . On the other hand , country of birth and spouse are easy to distinguish one from the other . In our experiments , we did not fine-tune q ( r′|r ) and chose k = b|R|/4c . Entity Type Reconstruction Another way to tolerate the annotation noise is to inject into the model useful biases . Our encoder uses the entity types of h , t to compute the vector representation xs , h , t , because entity types have been shown to be helpful for RC [ Hancock et al. , 2018 , Ma et al. , 2019 , Tran et al. , 2020 ] . Intuitively , relation types are constrained by entity types , e.g. , country of birth is constituted by an object such as person and a location . However , if trained on noisy labels only , the model may not be able to make use of entity types to tolerate the noise . Therefore , we force the model to capture the entity type bias by reconstructing the entity types of h , t. Formally speaking , denoting E the entity type set ( e.g. , E = { PER , LOC , ORG , MISC } ) , we compute the reconstruction probability using a linear and a softmax ( over E × E ) layers : pe ( .|s , h , t ) = SoftmaxE×E ( Linear ( xee ) ) ( 6 ) where xee = ReLU ( Linear ( xs , h , t ) ) ∈ Rdee .

Summary: This paper tackles the semi-supervised relation classification problem with only one manually annotated example for each relation. The method is largely based on the BERT matching model (Baldini Soares et al., 2019). The authors propose to (1) also take entity types as input, (2) use BERT matching to get noisy labels on the unlabeled data, (3) take a noisy channel and an entity type reconstruction objective to train the model on the self-labeled data. The setting is new, so the authors compared their method to BERT matching and a previous bootstrap method. The improvement compared to those selected baseline models is significant, but the performance is in general very low and there is a large gap to the supervised model. There is an ablation study showing the effectiveness of the entity type reconstruction and the noisy channel.