SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhenzhen Li $^{1}$ , Yuyang Zhang, Jian-Yun Nie $^{2}$ , Dongsheng Li $^{1}$
$^{1}$  National University of Defense Technology, Changsha, China
$^{2}$  DIRO, Université de Montréal, Montreal, Canada
{lizhenzhen14, dsli}@nudt.edu.cn, Nie@iro.umontreal.ca
Few-shot relation classification is difficult because the few instances available may not represent well the relation patterns. Some existing approaches explored extra information such as relation definition, in addition to the instances, to learn a better relation representation. However, the encoding of the extra information has been performed independently from the labeled instances. In this paper, we propose to learn a prototype encoder from relation definition text in a way that is useful for relation instance classification. To this end, we use a joint training approach to train both a prototype encoder from definition and an instance encoder. Extensive experiments on several datasets demonstrate the effectiveness and usefulness of our prototype encoder from definition text, enabling us to outperform state-of-the-art approaches.
Relation classification (RC) aims to determine the relation expressed between two entities in a sentence. Typical approaches to RC train a classification model or a prototype from labeled sentences, which are intended to represent the typical relation patterns including various syntactic, semantic and contextual features. In practical situations, we may have only a few labeled examples, which are insufficient for training a good classification model. In some cases, the labeled examples may also be atypical. For example, if a relation is exemplified by sentences describing Obama's presidency in the US, the resulting prototype would be largely tuned toward a political context, which would be difficult to apply outside the political context. Even for a human annotator, with only a relation ID and some examples, it would be difficult to learn to classify a relation correctly. This is exactly the same problem faced by a few-shot RC system: the best guess based on the limited labeled examples may fail.
If, however, the human annotator is told that the relation is named "position held", then he/she would have a better understanding of the relation to better generalize the examples to other instances. In this case, the human annotator indeed exploits the prior knowledge about the relation (from its name). If more information such as a description of the relation is available, then the annotator would do an even better job. The definition of a relation, being it the name or the description, is important to help the human annotator understand the relation. It is the same for an automatic RC model.
Several previous studies have explored using such extra knowledge about the relation (Qu et al., 2020; Dong et al., 2020; Zhang et al., 2021). For example, relation names have been used for better initializing the relation prototype before being trained with labeled instances (Dong et al., 2020). However, Dong et al. only considered a limited number of relations in the training data, making the few-shot RC model prone to over-fitting. Inspired by these studies and human behaviors, in this paper, we leverage relation definitions to build high-level prototype representations of the relations.
A second problem we consider in this paper is the ability to construct prototypes for new relations, i.e. zero-shot learning. In this case, if the new relation has a description, then one would be able to apply the same prototype encoder learned on the other relations to the new relation. In other words, the mechanism of building relation prototypes from their definitions could be generalized and transferred to a new relation. Some previous work (Qu et al., 2020) has considered prototype transfer based on a knowledge graph which provides relationships between relations. It is assumed that a relation's prototype can be partly transferred to a related relation in the graph. While this could be a possible way to generate a more reasonable prototype for a relation, we believe that relation definitions provide a better basis for constructing a
Findings of the Association for Computational Linguistics: NAACL 2022, pages 454 - 464 July 10-15, 2022 ©2022 Association for Computational Linguistics
Relation name	Description	Labeled data
language of work	language associated with creative work, such as books, shows, songs, or websites	Nokta ("dot" in Turkish) was a leading Turkish weekly political news magazine All Things Must Pass is a triple album by English musician George Harrison It was performed in French by French singer France Gall
position held	subject currently or formerly holds the object position or public office	Goebbels succeeded him as Chancellor of Germany It is named after Justus, Archbishop of Canterbury from 624-627 He represented Central Lancashire as a Member of the European Parliament (MEP)
Table 1: Example of relation definitions (relation name and description) and weakly labeled data for relation P407 and P39 in Wikidata, both of which are used for prototypical representation learning.
relation prototype.
We propose to learn a general mapping function from relation definitions to their prototypes. The advantages of the approach come from the fact that the definition text expresses the intrinsic semantics of relation, which is not explicitly covered by labeled instances. For example, as shown in Table 1, the name of the relation "language of work" provides a general idea of the relation. The description further specifies that it is the language used for "creative work". Such information is not explicitly expressed, but only hinted, in the corresponding labeled examples. The definition and labeled instances provide different but complementary information about the relation, therefore can be combined to improve RC.
Mapping a relation definition to a prototype vector could be done naively using a pre-trained language model such as BERT (Devlin et al., 2019), but the resulting prototype may not be the most useful for relation instance classification. We believe that a good prototype encoded from a definition should be the one that helps RC. Therefore, we propose to train a prototype encoder together with RC of some examples. To tackle the problem of limited labeled examples, inspired by the RC-oriented pre-training work (Baldini Soares et al., 2019), we use abundant distantly labeled data to help train a prototype representation. The pre-training will generate the relation prototype vector that can best classify the weakly labeled data.
To integrate the prototype representation learned from relation definitions with the given limited hand-labeled instances, we adapt the Bayesian meta-learning approach (Qu et al., 2020) to learn a posterior distribution of the prototype vectors of relations based on both the initial prototype representations and the labeled instances. This process helps adapt the prototype to the labeled instances.
We test our approach on two few-shot RC datasets. It outperforms previous competitive models that apply pre-trained instance encoder or rela
tion definition text. We also show that the encoder can be easily generalized to new relations in zero-shot RC setting. Experimental results demonstrate the effectiveness and generalization ability of our pre-trained prototype encoder.
Our main contribution in this paper is twofold: 1. We propose a new relation prototype construction method from relation definition; 2. We experimentally show that the approach is effective in few-shot RC and can be generalized to new relations in zero-shot RC.
RC aims to predict whether a sentence (instance)  $\mathbf{x}$  expresses a pre-defined relation between two given entities  $(e_1, e_2)$ . Neural RC models usually contain an instance encoder that encodes the relation expressed by an instance into a dense vector and a classification layer to classify the dense vector to the relation which has the most similar prototype vector. In this work, we leverage the definition text denoted as  $\mathbf{y}$  to help learn the relation prototype.
The prototype encoder is trained to help the classification of relation instances. Some training instances are required. Inspired by the RC-oriented pre-training work (Baldini Soares et al., 2019; Peng et al., 2020), we leverage a large set of distantly labeled data. The distant labeling (Mintz et al., 2009) is done as follows: given a known relation containing a pair of entities, any sentence mentioning the same entity pair is labeled by the relation.
Let us denote the set of distantly labeled instances as  $\mathbb{D}:\{(x_i,e_{1i},e_{2i},r_i)\}_{i = 1}^z$  and a set of definition texts for all relations in  $\mathbb{D}$  denote as  $\mathbb{T}:\{(y_t,r_t)\}_{t = 1}^n$ . The instance encoder (instance relation encoder)  $\mathrm{RelEnc}_{\phi}$ , parameterized by  $\phi$ , produces an instance embedding  $\mathbf{s}\in \mathbb{R}^d$  by  $\mathbf{s} = \mathrm{RelEnc}_{\phi}(\mathbf{x},e_1,e_2)$ , where  $d$  is the vector dimension of prototype representation. The prototype encoder  $\mathrm{ProtoEnc}_{\theta}$ , parameterized by  $\theta$ , produces a set of definition representations  $\{\mathbf{v}_t\}_{t = 1}^n$ , where  $\mathbf{v}_t\in \mathbb{R}^d$  is produced by  $\mathbf{v}_t = \mathrm{ProtoEnc}_{\theta}(\mathbf{y}_t)$ . The
Figure 1: Overview of the prototype encoder pretraining framework. On the right side, two modules encode different text separately: the prototype encoder encodes the definition text as prototypes; the instance encoder encodes the possible relation expressed by an instance as the instance embedding. On the left side, a multi-instance learning method takes the representations from the two encoders for relation classification.
training goal is to map definition representations to relation prototypes useful for classifying instances.
In this section, we first explain how a relation prototype is learned with the help of a set of distantly labeled instances. Then we describe how the learned prototype is further enhanced by the limited labeled instances in a Bayesian meta-learning framework.
3.1 Pre-training Framework
Pre-training a general prototype encoder involves two processes: generalizing contextual features of distantly labeled data to prototypes and mapping the relation definitions to their prototypes. As instances and the definition text describe relations in different forms, we adopt two pre-trained language models based on BERT (Devlin et al., 2019), as the backbones to encode them separately. The utilization of two distinct encoders is motivated by the different natures of the inputs: Relation definition provides a high-level, conceptual description of the relation, while a relation instance provides a concrete example about relation expression. From the former, the encoder would be asked to capture the general concepts, while from the latter, features relating to the syntactic/semantic pattern or the context can emerge. To cope with the large quantity of relations and the noisy labeling problem of distantly labeled data, we propose a simple yet
effective design of instance encoder, prototype encoder and their joint training process, as illustrated in Figure 1. The joint training process learns the mapping function of prototype encoder by directly regarding definition representations as relation prototypes for classification. We elaborate the three components of the pre-training framework in the following parts.
3.1.1 Instance Encoder
We use the entity marker strategy to extract the instance relation following previous investigation about architectures of instance encoder (Baldini Soares et al., 2019). An example is shown in Figure 1. Four special entity markers - [E1], [/E1], [E2], [/E2] are used to delimit entity1 and entity2. Then the hidden vectors of [E1] and [E2] at the last layer are concatenated to represent the relation expressed by the instance. Thus the hidden representation of an instance  $\mathbf{x}_i$  is obtained as  $H_{i} = \mathrm{RelEnc}_{\phi}(\mathbf{x}_{i},e_{1i},e_{2i}) = <$ $\mathbf{h}_{e_1}|\mathbf{h}_{e_2}>$ , where  $H_{i}\in \mathbb{R}^{2d}$  and  $d$  is the size of the hidden representation space of the pre-trained language model. The final instance embedding is transformed into a  $d$ -dimensional representation by a linear layer as  $\mathbf{s}_i = H_i^T W_t$ , where parameters  $W_{t}\in \mathbb{R}^{2d\times d}$ .
3.1.2 Prototype Encoder
Each relation from KBs typically has a name and a description text. We use [SEP] to separate the relation name and description, and add the special token [CLS] at the beginning, another [SEP] at
<sup>1</sup>We also tested with the same encoder, but obtained poor results.
the end. Figure 1 shows an example. After tokenizing the input text, we feed the input tokens into TinyBERT and use the hidden vector of [CLS] (of dimension  $d$ ) of the last layer as the definition representation. We adopt TinyBERT (Jiao et al., 2020) as the prototype encoder for its high efficiency. We also experimented the alternative with  $\mathrm{BERT}_{\mathrm{BASE}}$  to update relation prototypes within a mini batch. However, this alternative converges much more slowly and at a higher final training loss than using TinyBERT.
3.1.3 Joint Training with MIL
Multi-instance learning (MIL) has been widely used to alleviate the noisy labeling problem (Ji et al., 2017; Alt et al., 2019) in distantly labeled data. It regards a set of instances containing the same entity pair as a bag and assign the bag with one relation label. Then relation classification is relaxed from sentence level to bag level. By selecting the most reliable instance or assigning different attention weights among the instances of the bag (Lin et al., 2016), the impact of wrong labels is reduced.
In this work, we use the relation definition to guide the instance attention learning among the bag. We assume that instances that are semantically closer to the relation definition are more likely to express such a relation. Let us denote a bag sample from the noisy dataset  $\mathbb{D}$  as  $(B_{k},r_{k},e_{1,k},e_{2,k})$ , where  $B_{k} = \{\mathbf{x}_{1},\mathbf{x}_{2},\dots ,\mathbf{x}_{m}\}$  is a set of instances containing the entity pair  $(e_{1,k},e_{2,k})$ , which are encoded as  $\{\mathbf{s}_i\}_{i = 1}^m$  (also called 'instance embeddings') using RelEnc $\phi$ . The bag representation  $\mathbf{b}_k$  is computed by aggregating all instances according to the selective attention weights as follows:
$$
\mathbf {b} _ {k} = \sum_ {i = 1} ^ {m} a _ {i} \mathbf {s} _ {i}, a _ {i} = \frac {\exp (\mathbf {s} _ {i} ^ {T} \mathbf {v} _ {k})}{\sum_ {j = 1} ^ {m} \exp (\mathbf {s} _ {j} ^ {T} \mathbf {v} _ {k})} \quad (1)
$$
where the attention weight  $a_{i}$  represents the confidence score of instance  $\mathbf{x}_i$  expressing relation  $r_k$ .  $a_{i}$  is calculated according to the similarity of instance embedding  $\mathbf{s}_i$  and the definition representation  $\mathbf{v}_k$ .
We use dot product to compute the similarity of  $\mathbf{b}_k$  and candidate relation prototypes  $\{\mathbf{v}_i\}_{i=1}^n$ . Then the bag-level prediction probability for relation  $r_k$  is computed as follows:
$$
p (r _ {k} | B _ {k}, W _ {t}, \phi , \theta) = \frac {\exp (\mathbf {b} _ {k} ^ {T} \mathbf {v} _ {k})}{\sum_ {j = 1} ^ {n} \exp (\mathbf {b} _ {k} ^ {T} \mathbf {v} _ {j})}. (2)
$$
Standard cross entropy is used to compute RC loss. We also add an auxiliary loss about language mod
eling over training instances to avoid catastrophic forgetting. We follow the same setting as previous work (Devlin et al., 2019; Baldini Soares et al., 2019) to compute the masked language modeling loss  $(\mathcal{L}_{\mathrm{MLM}})$ . The final loss is defined as Eq. 3,
$$
\mathcal {L} = \alpha * \mathcal {L} _ {\mathrm {M L M}} + \mathcal {L} _ {\mathrm {R C}}, \tag {3}
$$
where  $\alpha$  controls the importance of the language modeling loss, set as 0.5 by default. We update the parameters of  $\{\phi, \theta, W_t\}$  for both encoders at each training iteration to minimize the final loss.
3.2 Application to Downstream RC
After the pre-training phase, we use the limited labeled instances to further enhance the prototype representation.
We adopt the Bayesian meta-learning approach proposed in (Qu et al., 2020), which models the uncertainty of prototype vectors by regarding them as random variables and learning the probability distribution for each relation. It could effectively learn the posterior distribution of the relation prototypes by combining the prior knowledge and a few labeled instances (i.e. the support set). In the model of (Qu et al., 2020), the prior of prototype vectors is derived from the structural relationship of different relations in a knowledge graph. The assumption is that a relation (a node in the graph) can gain some information from its neighbors. Relation representations are derived by applying graph neural network. In our case, prior relation prototypes are obtained from relation definition texts. A big advantage is that we do not require a relation be included in the knowledge graph to be able to handle it. A new relation can be handled if we have a definition of it, which is often the case (or the definition can be easily created) in practice. We will show later that our prototype representation performs better than that of (Qu et al., 2020).
The Bayesian meta-learning framework works as follows. For each sample in few-shot learning, given the candidate relations  $C$ , we denote their labels and textual definitions as  $\mathbf{r}_C$  and  $\mathbf{y}_C$ ; given a few supporting instances  $S$ , we denote their sentences and relation labels as  $\mathbf{X}_S$  and  $\mathbf{r}_S$ , where each relation  $r_s \in \mathbf{r}_C$ . In Bayesian statistics, we infer the posterior distribution of relation prototypes as follows. Considering context variables  $\{\mathbf{X}_S, \theta, \mathbf{y}_C\}$ , we formulate a Bayesian formula for  $p(\mathbf{v}_C | \mathbf{r}_S)$  in Eq. 5. Eq. 6 could be obtained when we assume that  $\mathbf{r}_S$  is uniformly distributed. Since
$\theta, \mathbf{y}_C$  are independent to  $\mathbf{r}_S$  and  $\mathbf{X}_S$  is independent to  $\mathbf{v}_C$ , we could finally get Eq. 7.
$$
\begin{array}{l} p \left(\mathbf {v} _ {C} \mid \mathbf {r} _ {S}; \mathbf {X} _ {S}, \theta , \mathbf {y} _ {C}\right) (4) \\ = \frac {p (\mathbf {r} _ {S} | \mathbf {v} _ {C} ; \mathbf {X} _ {S} , \theta , \mathbf {y} _ {C}) p (\mathbf {v} _ {C} | ; \mathbf {X} _ {S} , \theta , \mathbf {y} _ {C})}{p (\mathbf {r} _ {S} | ; \mathbf {X} _ {S} , \theta , \mathbf {y} _ {C})} (5) \\ \propto p (\mathbf {r} _ {S} | \mathbf {v} _ {C}; \mathbf {X} _ {S}, \theta , \mathbf {y} _ {C}) p (\mathbf {v} _ {C} |; \mathbf {X} _ {S}, \theta , \mathbf {y} _ {C}) (6) \\ \propto p \left(\mathbf {r} _ {S} \mid \mathbf {v} _ {C}, \mathbf {X} _ {S}\right) p \left(\mathbf {v} _ {C} \mid \theta , \mathbf {y} _ {C}\right) (7) \\ \end{array}
$$
Therefore, the posterior distribution of relation prototypes could be factorized as the likelihood of supporting instances and the prior knowledge of relation prototypes, which is derived from the definition text and pre-trained prototype encoder.  $p(\mathbf{v}_C|\theta ,\mathbf{y}_C)$  is the prior distribution for relation prototypes and each relation is assumed to follow a Gaussian distribution independently (i.e.,  $\mathcal{N}(\mathbf{v}_c|\mathrm{ProtoEnc}_{\theta}(\mathbf{y}_c),I))$  .  $p(\mathbf{r}_S|\mathbf{X}_S,\mathbf{v}_C)$  is the likelihood of supporting instances computed by the softmax function, where dot product is used to compute the similarity of instance embeddings and the final relation prototypes.
Following the implementation of (Qu et al., 2020), we sample multiple prototypes for estimating the posterior distribution and each sampling is obtained via multiple stochastic updates. The optimization process is end-to-end and further details can be found in the paper (Qu et al., 2020).
Given a query instance  $\mathbf{X}_q$  and a list of candidate relations  $C$  whose prototype vectors are denoted as  $\mathbf{v}_C$ , the relation distribution of the query instance over candidate relations can be computed by a softmax function as follows:
$$
p \left(r _ {q} \mid \mathbf {X} _ {q}, \mathbf {v} _ {C}\right) = \frac {\exp \left(\mathcal {E} \left(\mathbf {X} _ {q}\right) \cdot \mathbf {v} _ {q}\right)}{\sum_ {c \in C} \exp \left(\mathcal {E} \left(\mathbf {X} _ {q}\right) \cdot \mathbf {v} _ {c}\right)}, \tag {8}
$$
where  $\mathcal{E}$  is the instance encoder (like BERT or RelEnc $_{\phi}$ ).
Pre-training details To run experiments with limited computation resources, we use  $\mathrm{BERT}_{\mathrm{BASE}}^2$  as the backbone of instance encoder and a four-layer TinyBERT  $^3$  for the prototype encoder. We obtain a large-scale distantly-labeled dataset by processing the largest available alignments - T-REx (Elsahar et al., 2018), which align the documents from Wikipedia and triplets from Wikidata.
We removed the relations that do not have the textual definition, i.e., all the relations in the distantly labeled data have the textual definition. After removing the repetitive instances, the final dataset contains 636 relations and  $\sim 8\mathrm{M}$  instances. We get the relation name and descriptions from Wikidata.
The hyper-parameters we used during the pretraining process are: batch size is 96; the number of training epochs is 3; optimizer is Adam with a learning rate of 1e-4, which decays by 0.8 per epoch. Our pre-training takes about 68 hours on four V100 GPUs.
4.1 Few-Shot Relation Learning
Dataset and evaluation metrics We adopt two few-shot RC datasets: FewRel 1.0 (Han et al., 2018) and FewRel 2.0 (Gao et al., 2019b). FewRel 1.0 contains 100 relations split into training, validation and test sets with respectively 64, 16 and 20 relations without overlapping. Each relation has 700 instance sentences from Wikipedia. FewRel 2.0 is constructed to evaluate models for domain adaption challenge and its validation data and test data are from biomedical domain.
The typical N-way K-shot setting means each evaluation episode will sample N relations, each of which has K labeled instances, and some query instances. The models are asked to classify query instances into the sampled N relations given the  $\mathrm{N}\times \mathrm{K}$  labeled data. Accuracy is used to evaluate the classification performance. Note that we exclude entity pairs in test set of FewRel 1.0 that appear in the pre-training dataset to avoid data leakage.
Baselines We choose the following representative and strong baselines for comparison. (1) few-shot learning methods relying only on given training instances: ProtoNet (Snell et al., 2017), Pair (Han et al., 2018); (2) methods that integrate extra information: REGRAB (Qu et al., 2020) uses structural relationship between different relations in KBs, MIML (Dong et al., 2020) uses class semantic information from relation names; (3) RC-oriented pre-training methods that provide new instance encoders: MTB (Baldini Soares et al., 2019) constructs sentence pairs as training samples based on entity linking techniques, COL (Ding et al., 2021) uses relation prototypes for regularization assuming they are uniformly dispersed in a unit ball. CP (Peng et al., 2020) conducts contrastive pre-training over distantly labeled data. CP achieves state-of-the-art performance. For fair com
$^{2}$ https://huggingface.co/bert-base-uncased  
 $^{3}$ https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT
Method	5-way 1-shot	5-way 5-shot	10-way 1-shot	10-way 5-shot
ProtoNet†	80.68	89.60	71.48	82.89
Pair†	88.32	93.22	80.63	87.02
REGRAB†	90.30	94.25	84.09	89.93
MIML†	92.55	96.03	87.47	93.22
MTB†	89.09	95.32	82.17	91.73
COL†	92.51	95.88	86.39	92.76
BERT-EM	88.12	95.55	83.44	91.19
CP-RI	93.03	96.10	88.69	93.09
REGRAB+Proto	90.72	94.87	84.44	90.43
REGRAB+Rel+Proto	93.20	96.50	87.32	92.80
Rel+Proto	96.69	97.52	93.43	94.64
parison, we re-run CP model with our pre-training dataset and denote the model as CP-RI. We also implement a baseline BERT-EM that is optimized by the cross-entropy loss on every instance during pre-training and uses exemplar comparison for few-shot classification (Baldini Soares et al., 2019).