SlowGuess/NAACL_2021_OCR huggingface dataset:
Chih-Yao Chen
Program of Data Science
r09946001@ntu.edu.tw
chengte@ncku.edu.tw
While relation extraction is an essential task in knowledge acquisition and representation, and new-generated relations are common in the real world, less effort is made to predict unseen relations that cannot be observed at the training stage. In this paper, we formulate the zero-shot relation extraction problem by incorporating the text description of seen and unseen relations. We propose a novel multi-task learning model, zero-shot BERT (ZS-BERT), to directly predict unseen relations without handcrafted attribute labeling and multiple pairwise classifications. Given training instances consisting of input sentences and the descriptions of their relations, ZS-BERT learns two functions that project sentences and relation descriptions into an embedding space by jointly minimizing the distances between them and classifying seen relations. By generating the embeddings of unseen relations and new-coming sentences based on such two functions, we use nearest neighbor search to obtain the prediction of unseen relations. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing methods by at least  $13.54\%$  improvement on F1 score.
Relation extraction is an important task in the natural language processing field, which aims to infer the semantic relation between a pair of entities within a given sentence. There are many applications based on relation extraction, such as extending knowledge bases (KB) (Lin et al., 2015) and improving question answering task (Xu et al., 2016). Existing approaches to this task usually require large-scale labeled data. However, the labeling cost is a considerable difficulty. Some recent studies generate labeled data based on distant supervision (Mintz et al., 2009; Ji et al., 2017). Nevertheless, when putting the relation extraction task in the wild, existing supervised models cannot
well recognize the relations of instances that are extremely rare or even never covered by the training data. That said, in the real-world setting, we should not presume the relations/classes of new-coming sentences are always included in the training data. Thus it is crucial to invent new models to predict new classes that are not defined or observed beforehand. Such a task is referred as zero-shot learning (ZSL) (Norouzi et al., 2013; Lampert et al., 2014; Ba et al., 2015; Kodirov et al., 2017). The idea of ZSL is to connect seen and the unseen classes by finding an intermediate semantic representation. Unlike the common way to train a supervised model, seen and unseen classes are disjoint at training and testing stages. Hence, ZSL models need to generate transferable knowledge between them. With a model for ZSL relation extraction, we will be allowed to extract unobserved relations, and to deal with new relations resulting from the birth of new entities.
Existing studies on ZSL relation extraction are few and face some challenges. First, while the typical study (Levy et al., 2017) cannot perform zero-shot relation classification without putting more human effort on it, as they solve this problem via pre-defining question templates. However, it is infeasible and impractical to manually create templates of new-coming unseen relations under the zero-shot setting. We would expect a model that can produce accurate zero-shot prediction without the effort of hand-crafted labeling. In this work, we take advantage of the description of relations, which are usually publicly available, to achieve the goal. Second, although there exists studies that also utilize the accessibility of the relation descriptions (Obamuyide and Vlachos, 2018), they simply treat zero-shot prediction as the text entailment task and only output a binary label that indicates whether the entities in the input sentence can be depicted by a given relation description. Such problem formulation requires the impractical execution
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3470-3479 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: An example for elaborating our ZS-BERT.
Input Sentence	Relation	Relation Description
Training	X1: In 1997, Dennis Crouch and Hester put together a western swing band called "The Time Jumper"	y3: member of (seen)	D3: organization, musical group, or club to which the subject belongs
X2: He had roles in two 2008 films: the sci-fi film "Jumper" and the World War II drama "Defiance"	y7: main subject (seen)	D7: primary topic of a work
Testing	Z1: During the Philippine–American War, Mark Twain wrote a short pacifist story titled "The War Prayer"	y9: author (unseen, ground truth)	D9: main creator(s) of a written work
of multiple classifications over all relation descriptions, and cannot make relations comparable with each other.
This paper presents a novel model, Zero-shot BERT (ZS-BERT), to perform zero-shot learning for relation extraction to cope with the challenges mentioned above. ZS-BERT takes two model inputs. One is the input sentence containing the pair of target entities, and the other is the relation description, i.e., text describing the relation of two target entities. The model output is the attribute vector<sup>1</sup> depicting the relation. The attribute vector can be considered as a semantic representation of the relation, and will be used to generate the final prediction of unseen relations. We think a better utilization of relation descriptions by representation learning is more cost-effective than collecting tons of instances with labeled relations. Therefore, an essential benefit of ZS-BERT is free from heavy-cost crowdsourcing or annotation, i.e., annotating what kind of attribute does a class have, which is commonly used in zero-shot learning problem (Lu et al., 2018; Lampert et al., 2009).
Figure 1 depicts the overview of the proposed ZS-BERT, which consists of five steps. Each training instance is a pair of input sentence  $X_{i}$  and its corresponding relation's description  $D_{j}$ . First, we learn a projection function  $f$  that projects the input sentence  $X_{i}$  to its corresponding attribute vector, i.e., sentence embedding. Second, we learn another mapping function  $g$  that encodes the relation description  $D_{j}$  as into its corresponding attribute vector, which is the semantic representation of  $D_{j}$ . Third, given the training instance  $(X_{i},D_{j})$ , we train ZS-BERT by minimizing the distance be
tween attribute vectors  $f(X_{i})$  and  $g(D_{j})$  in the embedding space. Fourth, with the learned  $g(D_{l})$ , we are allowed to project the unseen relation's description  $D_{l}$  into the embedding space so that unseen classes can be as separate as possible for zero-shot prediction. Last, given a new input sentence  $Z_{k}$ , we can use its attributed vector  $f(Z_{k})$  to find the nearest neighbor in the embedding space as the final prediction. In short, the main idea of ZS-BERT is to learn the representations of relations based on their descriptions, and to align the representations with input sentences, at the training stage. In addition, we exploit the learned alignment projection functions  $f$  and  $g$  to generate the prediction of unseen relations for the new sentence so that the zero-shot relation extraction can be achieved. Our contributions can be summarized as below.
- Conceptually, we formulate the zero-shot relation extraction problem by leveraging text descriptions of seen and unseen relations. To the best of our knowledge, we are the first attempt to directly predict unseen relation under the zero-shot setting via learning the representations from relation descriptions.
- Technically, we propose a novel deep learning-based model, ZS-BERT $^2$ , to tackle the zero-shot relation extraction task. ZS-BERT learns the projection functions to align the input sentence with its relation in the embedding space, and thus is capable of predicting relations that were not seen during the training stage.
- Empirically, experiments conducted on two well-known datasets exhibit that ZS-BERT can significantly outperform state-of-the-art methods for predicting unseen relations under the ZSL setting. We also show that ZS-BERT can be quickly adapted and generalized to few-shot learning when a small fraction of labeled data for unseen relations is available.
BERT-based Relation Extraction. Contextual representation of words is effective for NLP tasks. BERT (Devlin et al., 2019) is a pre-training language model that learns useful contextual word representations. BERT can be moderately adopted
<sup>1</sup>The terms, "attribute vector", "embedding", and "representation", are used interchangeably throughout this paper.
$^{2}$ Code and implementation details can be accessed via: https://github.com/dinobby/ZS-BERT.
for supervised or few-shot relation extraction. R-BERT (Wu and He, 2019) utilize BERT to generate contextualized word representation, along with entities' information to perform supervised relation extraction and have shown promising result. BERT-PAIR (Gao et al., 2019) makes use of the pre-train BERT sentence classification model for few-shot relation extraction. By pairing each query sentence with all sentences in the support set, they can get the similarity between sentences by pretrained BERT, and accordingly classify new classes with a handful of instances. These models aim to solve the general relation extraction task, which are more or less having ground truth, rather than having it under the zero-shot setting.
Zero-shot Relation Extraction. Relevant studies on zero-shot relation extraction are limited. To the best of our knowledge, there are two most similar papers, which consider zero-shot relation extraction as two different tasks. Levy et al. (2017) treat zero-shot relation extraction as a question answering task. They manually define 10 question templates to represent relations, and generate the prediction by training a reading comprehension model to answer which relation satisfies the given sentence and question. However, it is required to have human efforts on defining question templates for unseen relations so that ZSL can be performed. Such annotation by domain knowledge is unfeasible in the wild when more unseen relations come. On the contrary, the data requirement of ZS-BERT is relatively lightweight. For each relation, we only need one description that could express the semantic meaning. The descriptions of relations are easier to be collected as we may access them from open resources. Under such circumstances, we may be free from putting additional effort to the annotation.
Obamuyide and Vlachos (2018) formulate ZSL relation extraction as a textual entailment task, which requires the model to predict whether the input sentence containing two entities matches the description of a given relation. They use Enhanced Sequential Inference Model (ESIM) (Chen et al., 2016) and Conditioned Inference Model (CIM) (Roektäschel et al., 2015) as their entailment methods. By pairing each input sentence with every relation description, they train the models to answer whether the paired texts are contradiction or entailment. This allows the model to inference on input sentence and unseen relation description pair, thus is able to predict unseen relation accordingly.
Let  $Y_{s} = \{y_{s}^{1},\dots,y_{s}^{n}\}$  and  $Y_{u} = \{y_{u}^{1},\dots,y_{u}^{m}\}$  denote the sets of seen and unseen relation labels, respectively, in which  $n = |Y_s|$  and  $m = |Y_u|$  are the numbers of relations in two sets. Such two sets are disjoint, i.e.,  $Y_{s}\cap Y_{u} = \emptyset$ . For each relation label in seen and unseen sets, we denote the corresponding attribute vector as  $a_{s}^{i}\in \mathbb{R}^{n\times d}$  and  $a_{u}^{i}\in \mathbb{R}^{m\times d}$ , respectively. Given the training set with  $N$  samples, consisting of input sentence  $X_{i}$ , entities  $e_{i1}$  and  $e_{i2}$ , and the description  $D_{i}$  of the corresponding seen relation  $y_{s}^{j}$ , denoted as  $\{S_i = (X_i,e_{i1},e_{i2},D_i,y_s^j)\}_{i = 1}^N$ . Our goal is to train a zero-shot relation extraction model  $\mathcal{M}$ , i.e.,  $\mathcal{M}(S_i)\to y_s^i\in Y_s$ , based on the training set such that using  $\mathcal{M}$  to predict the unseen relation  $y_{u}^{k}$  of a testing instance  $S^{\prime}$ , i.e.,  $\mathcal{M}(S^{\prime})\to y_u^j\in Y_u$ , can achieve as better as possible performance.
We train the model  $\mathcal{M}$  so that the semantics between input sentence and relation description can be aligned. We learn  $\mathcal{M}$  by minimizing the distance between two embedding vectors  $f(X_{i})$  and  $g(D_{i})$ , where learnable functions  $f$  and  $g$  project  $X_{i}$  and  $D_{i}$  into the embedding space, respectively. When new unseen relation  $y_{u}^{j}$  and its description is in hand, we can project the description of  $y_{u}^{j}$  to the embedding space by function  $g$ . When testing, new instance  $S^{\prime} = (Z_{j},e_{j1},e_{j2},D_{j})$  is input, in which  $Z_{i}$  denotes new sentence containing entities  $e_{j1}$  and  $e_{j2}$ , we project  $Z_{i}$  to the embedding space by our learned function  $f$ , and find the nearest neighboring unseen relation  $y_{u}^{j}$ , where  $Z_{i}$  and  $y_{u}^{i}$  are both unknown at the training stage.
4 The Proposed ZS-BERT Model
We give an overview of our ZS-BERT in Figure 2. The input sentence  $X_{i}$  is tokenized and sent into the upper-part ZS-BERT encoder to obtain contextual representation. We specifically extract the representation of [CLS],  $H_{0}$ , and two entities' representations  $H_{e}^{1}, H_{e}^{2}$ , and then concatenate them to derive sentence embeddings  $\hat{a}_{s}^{i}$ , by a fully-connected layer and activation operation. In the bottom part, we use Sentence-BERT (Reimers and Gurevych, 2019) to obtain attribute vector  $a_{s}^{i}$  for seen relations by encoding the corresponding description of relation  $D_{i}$ . We train ZS-BERT under a multitask learning structure. One task is to minimize the distance between attribute vector  $a_{s}^{i}$  and sentence embedding  $\hat{a}_{s}^{i}$ . The other is to classify the seen relation  $y_{s}^{j}$  at the training stage, in which a softmax
Figure 2: The overall architecture of our model.
layer that accepts relation embedding is used to produce the relation classification probability. At the testing stage, by obtaining the embeddings of new-coming sentences and unseen relations, we use  $\hat{a}_s^i$  and nearest neighbor search to obtain the prediction of unseen relations.
4.1 Learning Relation Attribute Vectors
For each seen and unseen relation, we learn its representation that depicts the corresponding semantic attributes based on relation description  $D_{i}$ . Most relations are well-defined and their descriptions are accessible from online open resources such as Wikidata<sup>3</sup>. We feed relation description  $D_{i}$  into a pre-trained Sentence-BERT encoder (Reimers and Gurevych, 2019) to generate the sentence-level representation as the attribute vector  $a^{i}$  of relations. This procedure is shown in the bottom part of Figure 2. The ground truth relation of the example is publisher, along with its description Organization or person responsible for publishing books, games or software. We feed only the relation description to the Sentence-BERT in order to get the attribute vector. That said, we consider the derived Sentence-BERT to be a projection function  $g$  that transforms the relation description  $D_{i}$  into  $a^{i}$ . Note that the relation attribute vectors produced by Sentence-BERT are fixed during model training.
4.2 Input Sentence Encoder
We utilize BERT (Devlin et al., 2019) to generate the contextual representation of each token. We first tokenize the input sentences  $X_{i}$  with Word-Piece tokenization (Sennrich et al., 2016). Two special tokens [CLS] and [SEP] are appended to the first and last positions, respectively. Since the entity itself does matter in relation extraction, we use an entity marker vector, consisting of all zeros except the indices that entities appear in a sentence, to indicate the positions of entities  $e_{i1}$  and  $e_{i2}$ . Let  $H_{0}$  be the hidden state of the first special token [CLS]. We use a tanh activation function, together with a fully connected layer, to derive the representation vector  $H_0^\prime$ , given by:  $H_0^\prime = W_0[\tanh (H_0)] + b_0$ , where  $W_{0}$  and  $b_{0}$  are learnable parameters for weights and biases. We obtain the hidden state vectors of two entities,  $H_{e}^{1}$  and  $H_{e}^{2}$ , by averaging their respective tokens' hidden state vectors. The entity can be recognized via simple element-wise multiplication between entity marker vector and token hidden vector. Specifically, if an entity  $e$  consists of multiple tokens and the indices range from  $q$  to  $r$ , we average the hidden state vectors, and also add an activation operation with a fully connected layer to generate its representation of that entity, given by:  $H_{e}^{c} = W_{e}\Bigl [\tanh \left(\frac{1}{r - q + 1}\sum_{t = q}^{r}H_{t}\right)\Bigr ] + b_{e}$ , where  $c = 1,2$ . Note that the representations of two entities  $H_{e}^{c}(c = 1,2)$  in the sentence shares the same parameters  $W_{e}$  and  $b_{e}$ . Then we learn the attribute vector  $\hat{a}_s^i$  by concatenating  $H_0^{\prime}, H_e^1$ , and  $H_{e}^{2}$ , followed by a hidden layer, given by:
$$
\hat {a} _ {s} ^ {i} = W _ {1} \left(\tanh  \left(\left[ H _ {0} ^ {\prime} \oplus H _ {e} ^ {1} \oplus H _ {e} ^ {2} \right]\right)\right) + b _ {1}, \tag {1}
$$
where  $W_{1}$  and  $b_{1}$  are learnable parameters, the dimensionality of  $\hat{a}_s^i$  is  $d$ , and  $\oplus$  is the concatenation operator.
The training of our ZS-BERT model consists of two objectives. The first is to minimize the distance between input sentence embedding  $a_{s}^{i}$  and the corresponding relation attribute vector  $\hat{a}_{s}^{i}$  (i.e., positive pairs), meanwhile to ensure embedding pairs between input sentence embedding and mismatched relation (i.e., negative pairs) to be farther away from each other. The black arrow connecting  $a_{s}^{i}$  and  $\hat{a}_{s}^{i}$  in Figure 2 is a visualization to indicate that we take both  $a_{s}^{i}$  and  $\hat{a}_{s}^{i}$  into consideration to achieve this goal. This is also reflected in the first term of our proposed loss function introduced
<sup>3</sup>https://www.wikipidata.org
Table 1: Datasets. "avg. len." is average sentence len.
#instances	#entities	#relations	avg. len.
Wiki-KB	1,518,444	306,720	354	23.82
Wiki-ZSL	94,383	77,623	113	24.85
FewRel	56,000	72,954	80	24.95
below. The second objective is to maximize the accuracy of relation classification based on seen relations using cross entropy loss. We transform the relation embedding, along with a softmax layer, to generate a  $n$ -dimensional  $(n = |Y_s|)$  classification probability distribution over seen relations:  $p(y_s | X_i, \theta) = \text{softmax}(W^*(\tanh(\hat{a}_s^i)) + b^*)$ , where  $y_s \in Y_s$  is the seen relation,  $\theta$  is the model parameter,  $W^* \in \mathbb{R}^{n \times h}$ ,  $h$  is the dimension of hidden layer, and  $b^* \in \mathbb{R}^n$ . Note that we do not use the probability distribution but the input sentence embedding  $\hat{a}_s^i$  produced intermediately for predicting unseen relations under zero-shot settings.
The objective function of ZS-BERT is as follows:
$$
\begin{array}{l} L = (1 - \alpha) \sum_ {i} ^ {N} \max  \left(0, \gamma - a _ {s} ^ {i} \cdot \hat {a} _ {s} ^ {i} + \max  _ {i \neq j} \left(a _ {s} ^ {i} \cdot \hat {a} _ {s} ^ {j}\right)\right) \\ - \alpha \sum_ {i} ^ {N} y _ {s} ^ {i} \log \left(\hat {y _ {s} ^ {i}}\right), \tag {2} \\ \end{array}
$$
where  $N$  is the number of samples,  $a_{s}^{i}$  is the relation attribute vector, and  $\hat{a}_s^i$  is the input sentence embedding. The first term in Eq. (2) sets a margin  $\gamma >0$  such that the inner product of the positive pair (i.e.,  $a_{s}^{i}\cdot \hat{a}_{s}^{i})$  must be higher than the maximum of the negative one (i.e.,  $\max_{i\neq j}(a_s^i\cdot \hat{a}_s^j))$  for more than a pre-decided threshold  $\gamma$ . With the introduction of  $\gamma$ , the loss will be increased owing to the difference between the positive and the closest negative pairs. This design of loss function can be viewed as ranking the correct relation attribute higher than the closest incorrect one. In addition,  $\gamma$  is also utilized to avoid the embedding space from collapsing. If we consider only minimizing the distance of positive pair using loss like Mean Squared Error, the optimization may lead to the result that every vector in the embedding space is too close to one another. We will examine how different  $\gamma$  values affect the performance in the experiment. To maintain low computational complexity, we consider only those mismatched relations within a batch as the negative samples  $j$ . The second term in Eq. (2) is a commonly used
cross entropy loss, which decreases as the prediction  $\hat{y}_s^i$  is correctly classified. Such a multi-task structure is expected to refine the input sentence embeddings and simultaneously bring high prediction accuracy of seen relations.
4.4 Generating Zero-Shot Prediction
With the trained model, when the descriptions of new relations are in hand, we can generate their attribute vectors  $a_{u}^{j}$ . As the new input sentence  $Z_{i}$  arrives, we can also produce its sentence embedding  $\hat{a}_{u}^{i}$  via:  $\hat{a}_{u}^{i} = W_{1}(\tanh ([H_{0}^{\prime}\oplus H_{e}^{1}\oplus H_{e}^{2}])) + b_{1}$ , where  $W_{1}$  and  $b_{1}$  are learned parameters. The prediction on unseen relations can be achieved by the nearest neighbor search. For the input sentence embedding  $\hat{a}_{u}^{i}$ , we find the nearest attribute vector  $a_{u}^{j}$  and consider the corresponding relation as the predicted unseen relation. This can be depicted by:  $C(Z_{i}) = \mathrm{argmin}_{j}dist(\hat{a}_{u}^{i},a_{u}^{j})$ , where function  $C$  returns the predicted relation of new input sentence  $Z_{i}$ ,  $a_{u}^{j}$  is the  $j$ -th attribute vector among all unseen relations in the embedding space,  $\hat{a}_{u}^{i}$  is the new input sentence embedding, and  $dist$  is a distance computing function. Here negative inner product is used as  $dist$  since we aim to consider the nearest neighboring relation as the predicted outcome.
5.1 Evaluation Settings
Datasets. Two datasets are employed, Wiki-ZSL and FewRel (Han et al., 2018). Wiki-ZSL is originated from Wiki-KB (Sorokin and Gurevych, 2017), and is generated with distant supervision. That said, in Wiki-ZSL, entities are extracted from complete articles in Wikipedia, and are linked to the Wikidata knowledge base so that their relations can be obtained. Since 395,976 instances (about  $26\%$  of the total data) do not contain relations in the original Wiki-KB data, we neglect instances with relation "none". To ensure having sufficient data instances for each relation in zero-shot learning, we further filter out the relations that appear fewer than 300 times. Eventually, we can have yields Wiki-ZSL, a subset of Wiki-KB.
On the other hand, FewRel (Han et al., 2018) is compiled by a similar way to collect entity-relation triplet with sentences, but had been further filtered by crowd workers. This ensures the data quality and class balance. Although FewRel is originally proposed for few-shot learning, it is also suitable for zero-shot learning as long as the relation labels within training and testing data are disjoint. The statistics of Wiki-KB, Wiki-ZSL and FewRel datasets are shown in Table 1.