Prototypical Representation Learning for Relation Extraction

1 INTRODUCTION . Relation extraction aims to predict relations between entities in sentences , which is crucial for understanding the structure of human knowledge and automatically extending knowledge bases ( Cohen & Hirsh , 1994 ; Bordes et al. , 2013 ; Zeng et al. , 2015 ; Schlichtkrull et al. , 2018 ; Shen et al. , 2020 ) . Learning representations for relation extraction is challenging due to the rich forms of expressions in human language , which usually contains fine-grained , complicated correlations between marked entities . Although many works are proposed to learn representations for relations from well-structured knowledge ( Bordes et al. , 2013 ; Lin et al. , 2015 ; Ji et al. , 2015 ) , when we extend the learning source to be unstructured distantly-labeled text ( Mintz et al. , 2009 ) , this task becomes particularly challenging due to spurious correlations and label noise ( Riedel et al. , 2010 ) . This paper aims to learn predictive , interpretable , and robust relation representations from large-scale distantly labeled data . We propose a prototype learning approach , where we impose a prototype for each relation and learn the representations from the semantics of each statement , rather than solely from the noisy distant labels . Statements are defined as sentences expressing relations between two marked entities . As shown in Figure 1 , a prototype is an embedding in the representation space capturing the most essential semantics of different statements for a given relation . These prototypes essentially serve as the center of data representation clusters for different relations and are surrounded by statements expressing the same relation . We learn the relation and prototype representations based on objective functions with clear geometric interpretations . Specifically , our approach assumes prototypes are unit vectors uniformly dispersed in a unit ball , and statement embeddings are centered ∗Equal contribution . †Corresponding author . ‡http : //ruizhang.info -8.703243 38.913616 -11.877513 26.187891 -10.24494 31.439125 -11.185886 28.53963 -6.7643037 31.873375 -9.293796 30.790508 -7.6870537 36.014236 -11.216029 31.59263 -7.4329624 35.181496 -10.611754 35.345486 -12.039416 22.71971 -11.616351 26.774305 -8.657383 35.28101 -13.320659 29.52109 -11.843529 26.295746 -10.6841545 28.803246 -7.6381073 36.70645 -9.564307 31.78187 -7.315561 39.579956 -4.533672 34.883198 -8.275788 33.035698 -7.4689636 35.644993 -11.624744 30.369122 -9.997218 38.49715 -7.7783055 37.59266 -9.765418 38.476517 -9.501085 32.574696 -6.3372855 37.55847 -11.9366 27.271059 -12.867382 36.685673 -10.081516 36.11969 -11.364351 34.99714 -13.114578 37.7766 -12.3460045 36.313652 -10.533733 29.321291 -6.8389997 37.80524 -10.056862 37.98478 -11.23021 29.466564 -8.006758 36.853874 -9.294378 29.327047 -10.339967 35.70266 -10.410015 31.533749 -13.232315 35.18631 -13.210859 29.454363 -7.7240295 37.255066 -10.900574 32.073906 -7.5629783 36.904022 -10.104565 31.449959 -8.409077 33.879704 -4.9887133 35.23039 -9.434023 33.882618 -10.956362 38.222942 -12.509354 31.459927 -6.1397 37.212097 -8.249299 36.092 -8.1636095 38.060524 -13.816341 26.780077 -12.954381 37.67792 -8.914301 38.879467 -11.33116 32.713974 15.663114 12.8621025 17.03712 18.486742 17.551886 14.484679 18.931808 18.930895 15.572551 13.700756 10.558792 17.574535 17.613794 13.805676 17.844099 18.073603 10.654739 18.235968 16.538021 10.523316 17.641438 13.43363 14.079281 12.15079 18.864927 14.856975 15.856341 10.3096 14.077647 16.088354 8.66363 17.863062 13.279723 15.911679 19.156492 14.567751 16.535969 16.739618 17.02585 13.34684 18.596682 11.846535 12.980082 18.898485 16.939898 17.028992 11.612106 16.237764 16.605167 17.61748 17.126232 9.916799 18.854584 16.075922 10.663816 15.945908 15.273596 11.521369 13.061167 16.711267 17.905909 10.257096 12.815981 17.505522 15.421573 16.05703 17.434866 14.227275 14.178516 17.211475 16.826227 12.7907715 8.85155 18.680946 16.42301 15.614987 16.469488 15.520174 17.752037 11.194748 13.268249 11.668395 19.25654 19.09535 17.51284 8.887941 10.873803 14.245057 15.503976 14.112884 12.484748 12.027752 12.8171425 9.299756 10.16367 10.303493 19.28029 17.801203 6.9740515 14.092293 16.590027 14.853376 15.17265 18.155851 18.5615 18.52006 17.618689 14.498615 16.670238 16.880407 18.137753 16.441898 14.629623 19.16981 17.200861 18.588112 17.524498 11.210086 17.630442 18.84577 11.128795 -11.828977 -43.852882 -18.523153 -48.35973 -9.013127 -49.965767 -12.775725 -38.4372 -11.386365 -45.235485 -12.341138 -46.811172 -18.695421 -47.257236 -17.548359 -47.813652 -16.86908 -47.66306 -9.285677 -48.134 -15.553001 -47.495358 -14.435563 -47.35719 -15.938199 -44.728043 -10.96651 -50.96271 -9.333371 -47.840088 -8.612561 -50.579475 -9.66584 -45.757843 -14.620562 -36.031902 -13.302847 -47.38101 -12.612855 -44.99101 -11.566471 -48.70189 -14.438774 -45.744644 -13.297511 -51.752514 -12.276336 -48.390713 -8.143216 -49.363167 -14.794755 -45.36748 -9.096069 -49.784893 -9.096887 -48.303566 -11.601617 -40.474926 -8.760579 -48.474392 -8.8828 -51.211826 -14.739173 -13.053048 -14.749782 -44.379173 -13.720553 -47.30534 -13.980023 -38.663837 -16.943655 -45.568115 -7.284974 -47.792404 -13.98673 -38.59544 -15.000847 -42.772976 -11.206605 -39.627068 -14.621026 -37.741215 -5.989932 -49.660408 -7.066158 -50.164024 -13.359581 -41.030807 -11.044324 -48.147877 -10.257005 -41.0706 -8.1137085 -49.74866 -11.536943 -44.25597 -10.309574 -49.382477 -16.535332 -48.246834 -16.57508 -44.80348 -14.411162 -48.468327 -14.510945 -36.572926 -14.163412 -45.338158 -19.041832 -47.55801 -14.49797 -46.043823 -16.699863 -49.536724 -15.524808 -47.48149 -8.445343 -48.45605 -11.151114 -46.65739 -14.092925 -49.13657 -8.212766 -49.940895 -11.3532095 -46.101093 -11.26969 -45.414417 -7.291685 -49.634018 -15.727305 -43.962696 -14.725657 -36.18008 -8.530978 -48.039 -12.263783 -46.4457 -13.111706 -45.93163 -16.514885 -47.25092 -17.156609 -46.77544 10.73023 -60.08112 -8.636485 -48.777756 -15.849547 -46.368256 -14.301521 -36.94852 -13.472748 -47.12356 9.924656 -54.517498 -12.016547 -49.535435 -12.705193 -43.919346 -13.171877 -39.083767 -13.118616 -42.864147 -14.06309 -49.1256 -6.9750905 -51.342743 -10.250488 -47.86121 -6.3860655 -50.226997 -13.797769 -46.049805 -9.539331 -43.63843 -7.4702673 -50.618774 -14.247917 -38.567375 -15.689215 -35.869644 -11.770379 -46.01179 -7.974457 -48.025314 -15.115464 -41.50516 -17.751368 -47.27474 -11.07022 -47.56136 -12.364049 -45.718647 -7.192023 -51.772503 -16.014822 -36.732063 -11.9974165 -42.06392 -9.965427 -51.854122 -13.210664 -38.62058 -8.270923 -49.581097 -11.441111 -49.41309 -8.452865 -13.024228 -14.365504 -41.821922 -15.149126 -40.793488 -13.114061 -43.343094 -15.477967 -42.245445 -16.184212 -41.677425 -15.117426 -47.537777 -11.991396 -41.70241 -12.122491 -44.353127 -14.115153 -36.14243 -13.586221 -45.992226 -13.601769 -44.940388 -10.696865 -45.62354 -9.2235775 -48.957237 -8.406905 -50.76499 -18.666452 -47.53441 -7.9554815 -46.99664 -10.015536 -52.052444 -14.992561 -37.97665 -14.920621 -48.478596 at the end of their corresponding prototype vectors on the surface of the ball . We propose statementstatement and prototype-statement objectives to ensure the intra-class compactness and inter-class separability . Unlike conventional cross-entropy loss that only uses instance-level supervision ( which could be noisy ) , our objectives exploit the interactions among all statements , leading to a predictively powerful encoder with more interpretable and robust representations . We further explore these properties of learned representations with extensive experiments . We apply our approach using a pretraining fine-tuning paradigm . We first pretrain a relation encoder with prototypes from a large-scale distantly labeled dataset , then fine-tune it on the target dataset with different relational learning settings . We further propose a probing dataset , FuzzyRED dataset ( Section 4.3 ) , to verify if our method can capture the underlying semantics of statements . Experiments demonstrate the predictive performance , robustness , and interpretability of our method . For predictive performance , we show that our model outperforms existing state-of-the art methods on supervised , few-shot , and zero-shot settings ( Section 4.4 and 4.2 ) . For robustness , we show how the model generalize to zero-shot setting ( Section 4.2 ) and how the prototypes regularize the decision boundary ( Section 4.5 ) . For interpretability , we visualize the learned embeddings and their corresponding prototypes ( Section 4.4 ) and show the cluster clearly follow the geometric structure that the objective functions impose . The source code of the paper will be released at https : //github.com/ Alibaba-NLP/ProtoRE . 2 RELATED WORK . Relation learning could be mainly divided into three categories . The logic-based methods reason relations via symbolic logic rules , with adopting probabilistic graphical models or inductive logic systems to learn and infer logic rules orienting to relations ( Cohen & Hirsh , 1994 ; Wang et al. , 2015 ; Yang et al. , 2017 ; Chen et al. , 2018 ; Kazemi & Poole , 2018 ; Qu & Tang , 2019 ) . The graph-based methods encode entities and relations into low-dimensional continues spaces to capture structure features of KBs . ( Nickel et al. , 2011 ; Bordes et al. , 2013 ; Yang et al. , 2015 ; Wang et al. , 2014 ; Nickel et al. , 2016 ; Ji et al. , 20 ; Lin et al. , 15 ; Trouillon et al. , 2016 ; Sun et al. , 2019 ; Balažević et al. , 2019 ) . The text-based methods have been widely explored recently , focusing on extracting semantic features from text to learn relations . The conventional methods to learn relations from the text are mainly supervised models , like statistical supervised models ( Zelenko et al. , 2003 ; GuoDong et al. , 2005 ; Mooney & Bunescu , 2006 ) . As deep neural networks have gained much attention then , a series of neural supervised models have been proposed ( Liu et al. , 2013 ; Zeng et al. , 2014 ; Xu et al. , 2015 ; Santos et al. , 2015 ; Zhang & Wang , 2015 ; Verga et al. , 2016 ; Verga & McCallum , 2016 ; Li et al. , 2019 ; Distiawan et al. , 2019 ; Ding et al. , 2019 ) . To address the issue of the insufficiency of annotated data , distant supervision has been applied to automatically generate a dataset with heuristic rules ( Mintz et al. , 2009 ) . Accompanying with auto-labeled data , massive noise will be introduced into models . Accordingly , various denoising methods have been explored to reduce noise effects for distant supervision ( Zeng et al. , 2015 ; Lin et al. , 2016 ; Jiang et al. , 2016 ; Han et al. , 2018a ; Wu et al. , 2017 ; Qin et al. , 2018a ; b ; Feng et al. , 2 2018 ) . Further efforts pay attention to learning relations from the text in various specific scenarios , such as zero-few-shot scenarios ( Levy et al. , 2017 ; Han et al. , 2018b ; Gao et al. , 2019 ; Soares et al. , 2019 ; Ye & Ling , 2019 ) and open-domain scenarios ( Banko et al. , 2007 ; Fader et al. , 2011 ; Mausam et al. , 2012 ; Del Corro & Gemulla , 2013 ; Angeli et al. , 2015 ; Stanovsky & Dagan , 2016 ; Mausam , 2016 ; Cui et al. , 2018 ; Shinyama & Sekine , 2006 ; Elsahar et al. , 2017 ; Wu et al. , 2019 ) . However , up to now , there are few strategies of relation learning that could be adapted to any relation learning tasks . Soares et al . ( 2019 ) introduces a preliminary method that learns relation representations from unstructured corpus . And the assumption that two same entities must express the same relation is imprecise , because it does not consider the semantics of contextual information . This paper abstracts the task from the text as a metric and prototype learning problem and proposes a interpretable method for relation representation learning . Apart from the differences in methods and task scenarios , row-less universal schema ( Verga & McCallum , 2016 ) has a similar spirit with our method , where relation type embeddings guide clustering of statements for entity pair representations and embeddings of entity pairs are regarded as an aggregation of relations . Although the intuition that assigns a representative prototype for each class is similar to some previous studies like Prototypical Networks ( Snell et al. , 2017 ) ( ProtoNet ) , there exist some essential distinctions between our approach and the ProtoNet . The ProtoNet computes the prototype of each class as the average of the embeddings of all the instance embeddings , that could be regarded as a non-linear version of the nearest class mean approach ( Mensink et al. , 2013 ) . The idea of mean-ofclass prototypes could also be traced to earlier studies in machine learning ( Graf et al. , 2009 ) and cognitive modeling and psychology ( Reed , 1972 ; Rosch et al. , 1976 ) . In our method , prototypes and relation encoder are collaboratively and dynamically trained by three objectives , which divides the high-dimensional feature space into disjoint manifolds . The ProtoNet performs instance-level classification to update the parameters , which is not robust for noisy labels . Our method carries out a novel prototype-level classification to effectively regularize the semantic information . The prototype-level classification reduces the distortion caused by noisy labels of the decision boundary fitted by NNs . Moreover , the ProtoNet is designed for few-zero-shot learning and our methods are more like a semi-supervised pre-training approach that could be applied to supervised , few-shot , and transfer learning scenarios . Furthermore , the ProtoNet does not contain the metric learning among instances , and our method simultaneously optimizes the prototype-statement and statement-statement metrics . We also make a geometry explanation of our method ( Section 3.1 ) .

Summary: This paper presents a pre-training method for encoders (i.e., BERT) of relation extraction, leveraging distant supervision data. The main idea is to introduce a prototype embedding for each relation in the distantly generated data. The loss function to pre-train the encoders involve several terms, aiming to exploit the intra-class compactness (e.g., the contrastive loss for statement pairs) and the inter-class separability (i.e., constraining the distances between the prototype for one relation and the embeddings of the statements for the same and different relations). The paper incorporates a prototype-level classification loss. Compared to few-shot learning models (e.g., prototypical networks), the paper claims that prototype-based losses can help the model to better address the noisy label issue of distantly generated data due to the exploitation of prototype and instance interactions. The pre-trained encoder is then applied on datasets for few-shot, supervised, and fuzzy relation extraction. The proposed method achieves competitive performance with other methods. Some analysis are also conducted to demonstrate the effectiveness of the proposed model.