One-shot to Weakly-Supervised Relation Classification using Language Models

1 . Introduction . Relation classification ( RC ) 1 detects the connection type between two entities in text such as place of birth between “ Murat Kurnaz ” and “ Germany ” , Figure 1 . It is crucial for downstream applications such as knowledge base construction [ Ji and Grishman , 2011 ] and question answering [ Xu et al. , 2016 ] . Most work in RC relies on either manually- or automatically-annotated data using knowledge bases ( KBs ) [ Zhang et al. , 2017 , Mintz et al. , 2009 , Riedel et al. , 2010 ] . This dependency leads to difficulty in generalising such methods to novel domains where labelled data or knowledge bases are not available . Recent work attempts to address the low resource scenario , namely few-shot relation extraction [ Han et al. , 2018 , Baldini Soares et al. , 2019 ] , which requires few examples per relation type during testing . However , they rely on a large available training and validation sets to learn the classification task . Perez et al . [ 2021 ] suggests to name this setting , multi-distribution few-shot learning . ∗ This author is currently employed by the Ubiquitous Knowledge Processing ( UKP ) Lab , Technische Universität Darmstadt . ( thytran @ ukp.informatik.tu-darmstadt.de ) 1Relation classification is Relation extraction with a predefined set of relation types , which assumes a relation hold between the entities . In contrast , our work evaluates the extreme setting when no separate training and development sets are given . Our setting considers only one example from each relation type , namely relation exemplar . We employ the matching model in Baldini Soares et al . [ 2019 ] to compute the similarities between an input sentence and those exemplars . After gathering all the scores , we assign the relation with the highest score to the input sentence , such as country of birth in Figure 1 . The above LM-based one-shot relation predictions are often incorrect . To improve the performance , we propose to learn from these predictions , which we name noisy annotations . We propose NoelA ( short for Noisy Channel Auto-encoder ) , which employs two mechanisms to alleviate the negative impact of noisy labels . Firstly , as entity types have been shown to be helpful for RC [ Hancock et al. , 2018 , Ma et al. , 2019 , Tran et al. , 2020 ] , NoelA reconstructs entity types of the two input entities so that the entity type bias is used when predicting relations . Secondly , we use a noisy channel [ Sukhbaatar et al. , 2014 , Goldberger and Ben-Reuven , 2016 , Wang et al. , 2019 ] to explicitly model the prediction noise . We conducted experiments on two relation classification datasets : TACRED [ Zhang et al. , 2017 ] and reWiki80 , a modification of Wiki80 [ Han et al. , 2019 ] . The two datasets have significantly different characteristics : the relation type distribution ( skewed and uniform ) , the number of relation types ( 41 and 80 ) , and the domain ( news and Wikipedia ) . We show the performance of different LMs as baselines in order to verify our LM choice . We also demonstrate that bootstrapping [ Reed et al. , 2014 ] , a traditional approach when supervision is scarce , is not effective . Meanwhile , NoelA outperforms the BERT matching by 9 % accuracy on TACRED , and 6 % on reWiki80 . We conduct an extensive analysis to understand the biases captured in pre-trained LMs and the contributions of each component in NoelA . 2 . Background . Relation classification ( RC ) is the task of assigning a ( predefined ) relation type to a pair of entities in a sentence . We denote R = { r1 , r2 , ... , rm } the set of m relation types . Given a sentence s of n words s = ( w1 , ... , wn ) , two entities h , t ( called head and tail entities respectively ) , and their corresponding semantic types eh , et , the task is to identify the relation r ∈ R between the two entities . For instance , in Figure 1 , h and t are “ Murat Kurnaz ” and “ Germany ” . Their entity types are eh = PERSON , et = LOCATION . The relation between them is r = country of birth . Our work focuses on predefined relation types excluding NA , i.e . “ no relation ” . A parametric probabilistic relation classifier is given by pr ( r|s , h , t ; θ ) ( with parameters θ ) which assigns a probability to a relation type r given a sentence s , head and tail entities h , t. Recent RC approaches adopt the diagram s , h , t→ encoder → classifier → r , converting 〈s , h , t〉 to a fixed-size vector representation before applying a classification layer ( often a pair of linear-softmax over the relation type set R ) True One-shot Setting In this work , we assume the extreme scenario where only oneshot training set is available , i.e. , a list of desired relation types and a single exemplar for each relation is given . For example , for the relation country of birth , its exemplar is “ [ Obama ] was born in [ the USA ] ” . Those exemplars are short and simple , often less than 10 words . Following Perez et al . [ 2021 ] , we refer this scenario as true one-shot setting to differentiate with previous work [ Han et al. , 2018 ] , in which a model has access to either separate training or development sets for tuning . Since our work does not aim at creating a new dataset , we use the predefined relation set of the evaluation datasets ( TACRED and reWiki80 ) and manually create an exemplar for each relation type . By creating an exemplar for each relation , we can simulate the process of imagining artificial examples when we are given a set of relation types at hand . The relation types and corresponding exemplars used in our experiments are presented in Appendix C. We note that our one-shot setting is very challenging even without detecting NA ( no relation ) . Temporally leaving the problem of predicting NA is in line with the few-shot relation classification research . In particular , the few-shot setting initialised by Han et al . [ 2018 ] and further research following it [ Baldini Soares et al. , 2019 ] also ignore NA . 3 . Matching Using Language Models . As shown in recent work , an LM trained on massive raw data ( e.g. , BERT ) can capture some level of semantic similarity . For instance , “ [ A ] is the mother of [ B ] ” is more similar to “ [ A ] gave birth to [ B ] ” than to “ [ A ] works for [ B ] ” . Therefore , we can use matching similarity computed by the LM to assign similar scores between an unseen sentence and the exemplars . Finally , the relation type assigned to the sentence is the one that has the highest score . Let f be a function mapping sentence s and two entities h , t to a vector in Rd . We compute the matching score sim ( f ( s1 , h1 , t1 ) , f ( s2 , h2 , t2 ) ) , where sim is any function that computes the similarity between two vectors , such as dot product . f produces the vector of 〈s , h , t〉 in two steps , depicted in Figure 2a . We first compute the entity representations of h , t ( i.e. , xh , xt ) by taking average of the words ( w ) that the mention contains , e.g. , xh = avgwi∈h ( wi ) . Then , we concatenate the two entity representations to form the relation candidate representation xs , h , t = [ xh ; xt ] . We note that our BERT matching , i.e . when BERT is used , is similar to Baldini Soares et al . [ 2019 ] ’ s BERT with mention pooling . 4 . Noisy Channel Auto-encoder ( NoelA ) . Because the prediction by LMs above is inevitably noisy , we propose NoelA ( depicted in Figure 2b ) to learn from noisy data . 4.1 Encoder . The encoder maps 〈s , h , t〉 to xs , h , t ∈ Rd . To compute a vector representation ( e.g. , xh ) for each entity mention ( e.g. , h ) , we take mean pooling over the entity span from BERT . We then concatenate the two vectors of h and t with their entity types ’ embeddings ( xeh , xet ∈ Rde ) , and apply a linear and a ReLU layers : xs , h , t = ReLU ( Linear ( Concat ( xh , xt , xeh , xet ) ) ) ( 1 ) 4.2 Decoder . Differently from traditional decoders , our decoder does not completely reconstruct the input 〈s , h , t〉 . It reconstructs the entity types eh , et of h , t only , and predicts the relation r hidden in the input by computing pr ( r|s , h , t ) . Relation Classifier After having a vector representation of 〈s , h , t〉 , we apply a linear and softmax ( over the relation type set R ) layers to compute pr ( r|s , h , t ) pr ( .|s , h , t ) = SoftmaxR ( Linear ( xs , h , t ) ) ( 2 ) Noisy Channel Although a large-scale pre-trained LM is assumed to passively memorise some relational facts , the annotation using such model results in relatively noisy labels . We thus explicitly model the annotation noise by q ( r′|r , s , h , t ) , the probability of assigning s to r′ given the correct relation type r. This probabilistic function is called “ noisy channel ” [ Goldberger and Ben-Reuven , 2016 ] . Not knowing the correct r , we marginalise over the relation type set R : p′r ( r ′|s , h , t ) = ∑ r∈R q ( r′|r , s , h , t ) pr ( r|s , h , t ) ( 3 ) It is often assumed that r′ is independent from 〈s , h , t〉 given r , thus we base q ( r′|r , s , h , t ) = q ( r′|r ) on a matrix C ∈ R|R|2 : q ( r′|r ) = exp ( cr ′r ) ∑ r′′ exp ( cr′′r ) ( 4 ) where cij is the entry of C at row i , column j. Initialising q ( r′|r ) has been shown crucial for learning p ( r|s , h , t ) [ Goldberger and BenReuven , 2016 ] . We initialise C with a matrix computed by the confusion of choosing relation types by the used LM matching . Formally , let count ( r′ , r ) be the number of times when r′ 6= r appear together in the top-k candidate lists for all sentences , then cr′r = log count ( r′ , r ) ∑ r′′ count ( r ′′ , r ) ( 5 ) This initialisation provides to the learning an idea of how much the LM is confused r with r′ . For instance , country of birth and country of death are likely confusing ( because in the past an average person often died and was born in the same city/town ) . On the other hand , country of birth and spouse are easy to distinguish one from the other . In our experiments , we did not fine-tune q ( r′|r ) and chose k = b|R|/4c . Entity Type Reconstruction Another way to tolerate the annotation noise is to inject into the model useful biases . Our encoder uses the entity types of h , t to compute the vector representation xs , h , t , because entity types have been shown to be helpful for RC [ Hancock et al. , 2018 , Ma et al. , 2019 , Tran et al. , 2020 ] . Intuitively , relation types are constrained by entity types , e.g. , country of birth is constituted by an object such as person and a location . However , if trained on noisy labels only , the model may not be able to make use of entity types to tolerate the noise . Therefore , we force the model to capture the entity type bias by reconstructing the entity types of h , t. Formally speaking , denoting E the entity type set ( e.g. , E = { PER , LOC , ORG , MISC } ) , we compute the reconstruction probability using a linear and a softmax ( over E × E ) layers : pe ( .|s , h , t ) = SoftmaxE×E ( Linear ( xee ) ) ( 6 ) where xee = ReLU ( Linear ( xs , h , t ) ) ∈ Rdee .

Summary: This paper proposes a weakly supervised relation classification model using pre-trained language models (LM). It considers the very challenging setting in which only one example of relation type is available. However, LM matching-based predictions which align a test sentence to a relation exemplar, are usually incorrect. This paper proposes to learn from those noisy labeling.