SlowGuess/NAACL_2022_OCR huggingface dataset:
Shusen Wang, Bosen Zhang, Yajing Xu*, Yanan Wu, Bo Xiao
{shusenw,zhangbosen,xyj,yanan.wu,xiaobo}@bupt.edu.cn
Zero-shot relation extraction aims to identify novel relations which cannot be observed at the training stage. However, it still faces some challenges since the unseen relations of instances are similar or the input sentences have similar entities, the unseen relation representations from different categories tend to overlap and lead to errors. In this paper, we propose a novel Relation Contrastive Learning framework (RCL) to mitigate above two types of similar problems: Similar Relations and Similar Entities. By jointly optimizing a contrastive instance loss with a relation classification loss on seen relations, RCL can learn subtle difference between instances and achieve better separation between different relation categories in the representation space simultaneously. Especially in contrastive instance learning, the dropout noise as data augmentation is adopted to amplify the semantic difference between similar instances without breaking relation representation, so as to promote model to learn more effective representations. Experiments conducted on two well-known datasets show that RCL can significantly outperform previous state-of-the-art methods. Moreover, if the seen relations are insufficient, RCL can also obtain comparable results with the model trained on the full training set, showing the robustness of our approach<sup>1</sup>
Relation extraction is a fundamental problem in natural language processing, which aims to identify the semantic relation between a pair of entities mentioned in the text. Recent progress in supervised relation extraction has achieved great successes (Zeng et al., 2014; Zhou et al., 2016; Soares et al., 2019), but these approaches usually require large-scale labeled data. While in practice, human annotation is time-consuming and labor-intensive. To alleviate the human annotation efforts in relation
Figure 1: Top: Overview of the proposed RCL at the training and test stage.  $f(\cdot)$  is a learnable projection function that projects the input sentence  $X_{i}$  to its corresponding relation representation  $f(X_{i})$ .  $f(\hat{X}_i)$  is the augmented view of  $f(X_{i})$  and  $Z$  is the whole test set. Bottom: Four examples at the test stage and their corresponding relation representations are shown in the right of Top. The entities are marked in orange.
Error Type	Input Sentence	Ground Truth	Baseline Pred	RCL Pred(Ours)
Similar Relation	Z1: In 1978, he replaced Thomas Erdelyi in the Ramones, assuming the name Marky Ramone.	member_of	member_of	member_of
Z2: The Doctor tries to restore the universe with the help of River and the alternative universe versions of his companions Amy Pond(Karen Gillan) and Rory Williams(Arthur Darvill).	part_of	member_of	part_of
Similar Entity	Z3: In May 2015, Vienna hosted the Eurovision Song Contest following Austria's victory in the 2014 contest.	follows	follows	follows
Z4: Thus, the song was succeeded as Romanian repre-sentative at the 2002 Contest by Monica Anghel & Marcel Pavel with "Tell Me Why".	part_of	follows	part_of
extraction, some recent studies use distant supervision to generate labeled data for training (Mintz et al., 2009; Lin et al., 2016). However, in the real-world setting, the relations of instances are not always included in the training data, and existing supervised methods cannot well recognize unobserved relations due to weak generalization ability.
To address the aforementioned limitations, zero-shot relation extraction has been proposed to extract relational facts where the target relations cannot be observed at the training stage. The challenge of zero-shot relation extraction models is how to learn effective representations based on seen relations at the training stage and well generalize to unseen relations at the test stage. Two studies (Levy et al., 2017; Obamuyide and Vlachos, 2018) treat zero-shot relation extraction as a different task (i.e., question answering and textual entailment), but they both need human annotation auxiliary information for input, i.e., pre-defining question templates and relation descriptions. ZS-BERT (Chen
*Yajing Xu is the corresponding author.
<sup>1</sup>https://github.com/ShusenWang/NAACL2022-RCL
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2456 - 2468
and Li, 2021) predicts unseen relations with attribute representation learning. Despite promising improvements on directly predicting unseen relations, ZS-BERT still makes wrong predictions due to similar relations or similar entities. The same problem arises in supervised methods under the zero-shot settings.
As shown in Figure 1, there are two types of similar errors: Similar Relations and Similar Entities. For similar relations (see  $Z_{1}$  and  $Z_{2}$ ), existing methods predict wrongly results because the unseen relations possess similar semantics and data points belong to two relations in the representation space are overlapped. For similar entities (i.e., 2014 contest and 2002 Contest), since entities are the context of relation and relation representations are derived from entities, the relation representations containing similar entities are close (see  $f(Z_{3})$  and  $f(Z_{4})$ ) and baselines wrongly consider  $f(Z_{4})$  belongs to follows in the representation space, even if two unseen relations are not related. Recently, Instance-wise Contrastive Learning (Instance-CL) (He et al., 2020; Chen et al., 2020; Yan et al., 2021; Gao et al., 2021; Zhang et al., 2021) has achieved remarkable success in representation learning. Instance-CL is used to learn an effective representation by pulling together the instances from the same class, while pushing apart instances from different classes. Inspired by Instance-CL, we attempt to use Instance-CL on seen relations to learn the difference between similar relations and the divergence of relation representations derived from similar entities.
In this paper, we propose a novel Relation Contrastive Learning framework (RCL) to solve the above-mentioned problems. Figure 1 depicts the overview of the proposed model, which consists of four steps: (i) The input for RCL is a batch of sentences containing the pair of target entities and each sentence is sent into input sentence encoder to generate the contextual sentence embeddings $^2$ . (ii) Taking the sentence embeddings as input, relation augmentation layer is designed to obtain the relation representations  $f(X_{i})$  and their corresponding augmented views  $f(\hat{X}_i)$ . (iii) By jointly optimizing a contrastive loss and a relation classification loss on seen relations, RCL can learn subtle difference between instances and achieve better separation between relations in the representation space simultaneously to obtain an effective projection function
$f$ . (iv) With the learned  $f$ , the whole test set  $Z$  can be projected for unseen relation representations in the representation space and zero-shot prediction is performed on unseen relation representations by K-Means.
To summarize, the major contributions of our work are as follows: (i) We propose a novel framework based on contrastive learning for zero-shot relation extraction. It effectively mitigates two types of similar problems: similar relations and similar entities by learning representations jointly optimized with contrastive loss and classification loss. (ii) We explore various data augmentation strategies in relation augmentation to minimize semantic impact for contrastive instance learning and experimental results show dropout noise as minimal data augmentation can help RCL learn the difference between similar instances better. (iii) We conduct experiments on two well-known datasets. Experimental results show that RCL can advance state-of-the-art performance by a large margin. Besides, even if the number of seen relations is insufficient, RCL can also achieve comparable results with the model trained on the full training set.
Relation Extraction. Relation extraction aims at extracting relation between entities within a given sentence. Many relation extraction methods (Qian et al., 2008; Zeng et al., 2014; Zhou et al., 2016) are supervised model. Recently, some studies focus on pre-training language model (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020) because of its powerful capability of semantic representation. Wu and He (2019) propose R-BERT that uses BERT to extract relation features and incorporates entity information to perform relation extraction. Soares et al. (2019) propose a relation representation learning method based on BERT and have shown promising results. However, these models require labeled data. Unsupervised relation extraction (Yu et al., 2017; Saha and Mausam, 2018; Stanovsky et al., 2018) can discover semantic relation feature from data without human annotations. One representative work is Open relation extraction. Wu et al. (2019) propose a novel model to learn a similarity metric of relations from labeled data, and identify unseen relations by transferring knowledge learned from seen relations. While OpenRE method can identify novel relation without annotations and external resources, it cannot effectively
2The words, "embeddings", and "representations", are used interchangeably throughout this paper.
Figure 2: Architecture of the RCL, which consists of three components, and the details are introduced in Section 3. Note that relation augmentation layer contains data augmentation and a concat layer.
discard irrelevant information and severely suffers from the instability.
Zero-shot Relation Extraction. Zero-shot relation extraction aims to identify novel relation without training instances. Existing zero-shot relation extraction methods are few and most rely on human annotation auxiliary information for input. Levy et al. (2017) reduce zero-shot relation extraction to a question answering task. They use 10 predefining question templates to represent relations, and then train a reading comprehension model to infer which relation satisfies the given sentence and question. Obamuyide and Vlachos (2018) treat zero-shot relation extraction as a textual entailment task, which requires the model to input descriptions of relations. They train a textual entailment model to predict whether the input sentence containing two entities matches the description of a given relation, identifying novel relations by generalizing from the descriptions of seen relations at the training stage to those of unseen relations at test time. Chen and Li (2021) propose ZS-BERT to tackle zero-shot relation extraction task with attribute representation learning. ZS-BERT learns the representations of relations based on their descriptions during the training time, and generates the prediction of unseen relation for new sentence by nearest neighbor search. However, ZS-BERT suffers from similar relation error and similar entity error, and it needs human annotation auxiliary information for input, i.e., relation descriptions. In this paper, we do not require any human annotation auxiliary information for input.
Contrastive Learning. In the field of image and natural language processing, many recent successes are inspired by contrastive learning (He et al., 2020; Chen et al., 2020; Yan et al., 2021; Gao et al., 2021).
Contrastive learning regards the input data and corresponding augmented views as an independent class. The goal of contrastive learning is to pull together representations from the same class, while keeping representations from different classes away. Therefore, the representations learned from contrastive learning are better separated and good for clustering. Gao et al. (2021) propose a novel sentence embeddings learning framework based on contrastive learning to produce superior sentence embeddings and show that dropout is an effective data augmentation. SCCL (Zhang et al., 2021) jointly optimizes a contrastive loss and a clustering loss to disperse overlap categories in the representation space. Inspired by contrastive learning, we leverage contrastive learning to help the model learn an effective representation.
As illustrated in Figure 2, the proposed model RCL consists of three components: input sentence encoder, contrastive learning module and relation classification module. Given a batch of sentences containing two entities, the sentence representations are generated by input sentence encoder and then are sent to relation classification module and contrastive learning module. For contrastive learning module, the relation representations and their corresponding augmented views generated by a relation augmentation layer are used to perform contrastive instance learning to learn the difference between instances. For relation classification module, the relation representations generated by the concat layer are used to identify seen relations to achieve better separation between relations. We train RCL under a multi-task learning structure
with contrastive learning module and relation classification module to learn effective representations for unseen relations. At the test stage, we obtain the unseen relation representations by the input sentence encoder and concat layer, and then send them into K-Means to predict the unseen relations.
3.2 Input Sentence Encoder
Input Sentence Encoder aims to generate the contextual representation of each token. In this work, we assume entities contained in the sentence have been recognized before input. For a sentence  $X_{i} = \left[x_{i}^{1},..,x_{i}^{L}\right]$  where two entities  $e_1$  and  $e_2$  are mentioned, we use the ENTITY MARKERS (Soares et al., 2019) to augment  $X_{i}$  to better extract relation features from context. Specifically, we introduce four special tokens to mark the beginning and the end of each entity mentioned in the sentence. The input token sequence  $X_{i}$  for input sentence encoder is as follows:
$$
\begin{array}{l} X _ {i} = \left[ x _ {i} ^ {1}, \dots , <   e _ {1} >, x _ {i} ^ {k}, \dots , x _ {i} ^ {l - 1}, <   / e _ {1} >, \right. \\ \left. \dots , <   e _ {2} >, x _ {i} ^ {p}, \dots , x _ {i} ^ {z - 1}, <   / e _ {2} >, \dots , x _ {i} ^ {L} \right] \tag {1} \\ \end{array}
$$
where  $< e_1 >, < /e_1 >, < e_2 >, < /e_2 >$  are four special tokens to mark the beginning and the end of each entity mentioned in the sentence,  $L$  is the length of sentence. Then we use BERT (Devlin et al., 2019) to obtain the sentence embeddings  $\pmb{h}_i \in \mathbb{R}^{L \times d}$ :
$$
\begin{array}{l} \mathbf {h} _ {\mathbf {i}} = \left[ \mathbf {h} _ {\mathbf {i}} ^ {\mathbf {1}}, \dots , \mathbf {h} _ {\mathbf {i}} ^ {<   \mathbf {e} _ {1} >}, \dots , \mathbf {h} _ {\mathbf {i}} ^ {<   / \mathbf {e} _ {1} >}, \right. \tag {2} \\ \ldots , \mathbf {h} _ {\mathbf {i}} ^ {<   \mathbf {e} _ {2} >}, \ldots , \mathbf {h} _ {\mathbf {i}} ^ {<   / \mathbf {e} _ {2} >}, \ldots , \mathbf {h} _ {\mathbf {i}} ^ {\mathbf {L}} ] \\ \end{array}
$$
where  $d$  is the hidden dimension.
3.3 Contrastive Learning Module
Contrastive Learning Module aims at learning the difference between a batch of instances to better represent relations.
Contrastive Instance Learning. After we obtained  $H = \{h_1, \dots, h_N\}$  from  $N$  input sentences using input sentence encoder, relation augmentation layer is used to generate relation representations and their augmented views. More specifically, the relation augmentation layer consists of data augmentation and a concat layer. For each sentence embeddings  $h_i$ , a transformation  $T(\cdot)$  is applied to generate its augmented view:  $\hat{h}_i = T(h_i)$  where  $\hat{h}_i \in \mathbb{R}^{L \times d}$ .
After obtaining sentence embeddings  $h_i$  and its augmentation  $\hat{h}_i$ , we obtain relation representa
tions and its augmentation by a concat layer. Specifically, we use the token embeddings corresponding to  $< e_1 >, < e_2 >$  positions as the entity representation and concatenate them to derive a contextualized relation representation and its augmented view  $\pmb{r}_i, \hat{\pmb{r}}_i \in \mathbb{R}^{2 \cdot d}$ :
$$
\begin{array}{l} \mathbf {r} _ {\mathrm {i}} = \mathbf {h} _ {\mathrm {i}} ^ {<   \mathbf {e} _ {1} >} \oplus \mathbf {h} _ {\mathrm {i}} ^ {<   \mathbf {e} _ {2} >} \\ \hat {\mathbf {r}} _ {\hat {\mathbf {i}}} = \hat {\mathbf {h}} _ {\hat {\mathbf {i}}} ^ {<   \mathbf {e} _ {1} >} \oplus \hat {\mathbf {h}} _ {\hat {\mathbf {i}}} ^ {<   \mathbf {e} _ {2} >} \tag {3} \\ \end{array}
$$
where  $\oplus$  is the concatenation operator and  $r_i, \hat{r}_i$  are both fixed-length vector.
To better learn effective relation representations, we optimize a contrastive objective, which dispenses different relation of instances apart while implicitly bringing the same relation of instances together. Let  $\mathbf{R} = \{r_1, \dots, r_N\}$  and  $\hat{\mathbf{R}} = \{\hat{r}_1, \dots, \hat{r}_N\}$  denote a mini-batch of relation representations and its augmented views respectively. We regard  $(r_i, \hat{r}_i)$  as a positive pair and other  $N - 1$  augmented views as negative instances. For a minibatch with  $N$  pairs, we follow the contrastive framework in SimCSE (Gao et al., 2021) and take a cross-entropy objective with in-batch negatives (Chen et al., 2017). The training objective for  $(r_i, \hat{r}_i)$  is:
$$
\ell_ {i} ^ {c l} = - \log \frac {e ^ {\operatorname {s i m} \left(\mathbf {r} _ {\mathbf {i}} , \hat {\mathbf {r}} _ {\mathbf {i}}\right) / \tau}}{\sum_ {j = 1} ^ {N} e ^ {\operatorname {s i m} \left(\mathbf {r} _ {\mathbf {i}} , \hat {\mathbf {r}} _ {\mathbf {j}}\right) / \tau}} \tag {4}
$$
where  $\pmb{r}_i\in R,\hat{\pmb{r}}_i\in \hat{R}$  , sim  $(\mathbf{r_1},\mathbf{r_2})$  is the cosine similarity  $\frac{\mathbf{r}_1^\top\mathbf{r}_2}{\|\mathbf{r}_1\|\cdot\|\mathbf{r}_2\|}$  , and  $\tau$  is a temperature hyperparameter.
Data Augmentation Strategies. To amplify the semantic difference between similar instances without breaking the semantic of relation representations, we explore five different data augmentations  $T(\cdot)$  for contrastive instance learning, including feature cutoff (Shen et al., 2020), random mask (Hinton et al., 2012), dropout (Gao et al., 2021), composition of dropout and feature cutoff and composition of dropout and random mask.
Feature cutoff is a simple and efficient data augmentation strategy to introduce minimal semantic impact for relation instances. Specifically, we randomly erase some feature dimensions in the sentence embeddings produced by input sentence encoder.
Random mask is proved its effectiveness as an augmentation strategy (Yan et al., 2021). In our experiments, we randomly drop elements in the sentence embeddings by a specific probability and sets their values to zero.
Dropout has been shown its effectiveness as minimal data augmentation by SimCSE (Gao et al., 2021). Thus, similar to SimCSE, we augment sentence embeddings by feeding the same input sentence to BERT again.
Composition of augmentations is an effective strategy in image domain (Chen et al., 2020). Based on dropout, we explore two strategies of composition of data augmentations. Composition of dropout and feature cutoff is a strategy that we first use dropout to obtain augmented view and then send it into feature cutoff to obtain the final augmented view. Similarly, composition of dropout and random mask is a strategy that dropout first and then random mask. We present the experimental results of these strategies and analyze their effects for contrastive learning in Section 4.4.
3.4 Relation Classification Module
Relation Classification Module aims to identify seen relations. With sentence embeddings  $h_i$  from input sentence encoder, we obtain relation representation  $r_i$  by the concat layer, following the way same as Equation (3). Let  $n$  denotes the number of seen relations and  $Y_s$  denotes the set of seen relations. By transforming the relation representation  $r_i$ , along with a softmax layer, we generate the n-dimensional classification probability distribution of the i-th sample over seen relations:
$$
p \left(y _ {i} \mid X _ {i}, \theta\right) = \operatorname {s o f t m a x} (W (\tanh  \left(\mathbf {r} _ {\mathbf {i}}\right)) + b) \tag {5}
$$
where  $X_{i}$  is the i-th input sentence containing two entities,  $y_{i} \in Y_{s}$  is the seen relation,  $\theta$  is the model parameter,  $W \in \mathbb{R}^{n \times 2 \cdot d}$ , and  $b \in \mathbb{R}^n$ . Note that we use the relation representation  $\pmb{r}_{i}$  produced intermediately for predicting unseen relations under the zero-shot settings instead of the probability distribution. For each data point  $X_{i}$ , we use cross-entropy to calculate classification loss:
$$
\ell_ {i} ^ {r c} = \operatorname {C r o s s E n t r o p y} (p \left(y _ {i} \mid X _ {i}, \theta\right), \hat {y} _ {i}) \tag {6}
$$
where  $\hat{y}_i$  is the ground-truth label of the i-th sample.
3.5 Train and Test
At the training stage, We train the model with two objectives under the multi-task learning structure. The first is to minimize the distance between the relation representation and its augmented view, while keeping the relation representation distant from other augmented relation representations in a minibatch. The second objective is to bring high prediction accuracy of seen relations. For a mini-batch of
input sentences, the training objective of RCL is as follows:
$$
\mathcal {L} _ {\mathrm {C L}} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell_ {i} ^ {c l}, \mathcal {L} _ {\mathrm {R C}} = - \frac {1}{N} \sum_ {i = 1} ^ {N} \ell_ {i} ^ {r c} \tag {7}
$$
$$
\mathcal {L} _ {\text {j o i n t}} = \mathcal {L} _ {\mathrm {R C}} + \alpha \mathcal {L} _ {\mathrm {C L}}
$$
where  $N$  is the number of input sentences,  $\alpha$  is a hyper-parameter to balance two objectives.
At the test stage, we send the new-coming sentences into the input sentence encoder and concat layer to generate unseen relation representations, and the prediction on unseen relations can be achieved by K-Means.
Two datasets are used to evaluate our model: SemEval2010 Task8 (Hendrickx et al., 2010) and FewRel (Han et al., 2018). SemEval2010 Task8 has been widely used in relation extraction task, which contains 9 relations and an Other relation. There are 10,717 instances in the dataset and the number of instances of each relation is not equal. Each relation has direction in the dataset, but in our experiments, we do not consider the direction of 9 relations and not use the Other relation. For each relation, we combine the instances of training set with instances of test set to obtain overall instances of each relation. FewRel is a public dataset based on Wikipedia, and it contains 80 types of relations, each with 700 instances. Although FewRel is widely used in few-shot learning setting, it is also suitable for zero-shot learning as long as we disjoint the relation labels within training and test data. The statistics of the two datasets are shown in Appendix A.
4.2 Evaluation Settings
Zero-shot Learning Settings. Let  $m$  denotes the number of unseen relations, and  $Y_{u}$  denotes the set of unseen relations. We randomly select  $m$  relations as unseen relations and the rest of relations  $n$  as seen relations. Note that  $Y_{s} \cap Y_{u} = \emptyset$ . Then we split the whole dataset into training and test data. The training data only contains the instances of seen relations, in contrast to test data only with the instances of unseen relations. We repeat experiments 5 times on SemEval2010 Task8 and FewRel, and then report the average results. As for implementation details for RCL, we implement
our model based on Transformers package<sup>3</sup> (Wolf et al., 2020). And we use an Adam optimizer (Kingma and Ba, 2014), in which the learning rate is 5e-5. Please refer to the Appendix B for more implementation details.