SlowGuess/NAACL_2021_OCR huggingface dataset:
Fenia Christopoulou<sup>1</sup>, Makoto Miwa<sup>2,3</sup>, Sophia Ananiadou<sup>1</sup>
$^{1}$ National Centre for Text Mining,
Department of Computer Science, The University of Manchester, United Kingdom
$^{2}$ Toyota Technological Institute, Nagoya, 468-8511, Japan
<sup>3</sup>Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology, Japan
{efstathia.christopoulou, sophia.ananiadou}@manchester.ac.uk
makoto-miwa@toyota-ti.ac.jp
We propose a multi-task, probabilistic approach to facilitate distantly supervised relation extraction by bringing closer the representations of sentences that contain the same Knowledge Base pairs. To achieve this, we bias the latent space of sentences via a Variational Autoencoder (VAE) that is trained jointly with a relation classifier. The latent code guides the pair representations and influences sentence reconstruction. Experimental results on two datasets created via distant supervision indicate that multi-task learning results in performance benefits. Additional exploration of employing Knowledge Base priors into the VAE reveals that the sentence space can be shifted towards that of the Knowledge Base, offering interpretability and further improving results<sup>1</sup>.
Distant supervision (DS) is a setting where information from existing, structured knowledge, such as Knowledge Bases (KB), is exploited to automatically annotate raw data. For the task of relation extraction, this setting was popularised by Mintz et al. (2009). Sentences containing a pair of interest were annotated as positive instances of a relation, if and only if the pair was found to share this relation in the KB. However, due to the strictness of this assumption, relaxations were proposed, such as the at-least-one assumption introduced by Riedel et al. (2010): Instead of assuming that all sentences in which a known related pair appears express the relationship, we assume that at least one of these sentences (namely a bag of sentences) expresses the relationship. Figure 1 shows example bags for two entity pairs.
Figure 1: Example of the bag-level setting in distantly supervised relation extraction and the main idea of our approach. Sentences are adapted from the NYT10 dataset (Riedel et al., 2010).
The usefulness of distantly supervised relation extraction (DSRE) is reflected in facilitating automatic data annotation, as well as the usage of such data to train models for KB population (Ji and Grishman, 2011). However, DSRE suffers from noisy instances, long-tail relations and unbalanced bag sizes. Typical noise reduction methods have focused on using attention (Lin et al., 2016; Ye and Ling, 2019) or reinforcement learning (Qin et al., 2018b; Wu et al., 2019). For long-tail relations, relation type hierarchies and entity descriptors have been proposed (She et al., 2018; Zhang et al., 2019; Hu et al., 2019), while the limited bag size is usually tackled through incorporation of external data (Beltagy et al., 2019), information from KBs (Vashishth et al., 2018) or pre-trained language models (Alt et al., 2019). Our goal is not to investigate noise reduction, since it has already been widely addressed. Instead, we aim to propose a more general framework that can be easily combined with existing noise reduction methods or pre-trained language models.
Methods that combine information from Knowledge Bases in the form of pre-trained Knowledge Graph (KG) embeddings have been particularly effective in DSRE. This is expected since they capture broad associations between entities,
Source code is available at https://github.com/fenchri/dsre-vae
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 11-26 June 6-11, 2021. ©2021 Association for Computational Linguistics
thus assisting the detection of facts. Existing approaches either encourage explicit agreement between sentence- and KB-level classification decisions (Weston et al., 2013; Xu and Barbosa, 2019), minimise the distance between KB pairs and sentence embeddings (Wang et al., 2018) or directly incorporate KB embeddings into the training process in the form of attention queries (Han et al., 2018; She et al., 2018; Hu et al., 2019). Although these signals are beneficial, direct usage of KB embeddings into the model often requires explicit KB representations of entities and relations, leading to poor generalisation to unseen examples. In addition, forcing decisions between KB and text to be the same makes the connection between context-agnostic (from the KB) and context-aware (from sentences) pairs rigid, as they often express different things.
Variational Autoencoders (VAEs) (Kingma and Welling, 2013) are latent variable encoder-decoder models that parameterise posterior distributions using neural networks. As such, they learn an effective latent space which can be easily manipulated. Sentence reconstruction via encoder-decoder networks helps sentence expressivity by learning semantic or syntactic similarities in the sentence space. On the other hand, signals from a KB can assist detection of factual relations. We aim to combine these two using a VAE together with a bag-level relation classifier. We then either force each sentence's latent code to be close to the Normal distribution (Bowman et al., 2016), or to a prior distribution obtained from KB embeddings. This latent code is employed into sentence representations for classification and is responsible for sentence reconstruction. As it is influenced by the prior we essentially inject signals from the KB to the target task. In addition, sentence reconstruction learns to preserve elements that are useful for the bag relation. To the best of our knowledge, this is the first attempt to combine a VAE with a bag-level classifier for DSRE.
Finally, there are methods for DSRE that follow a rather flawed evaluation setting, where several test pairs are included in the training set. Under this setting, the generalisability of such methods can be exaggerated. We test these approaches under data without overlaps and find that their performance is severely deprecated. With this comparison, we aim to promote evaluation on the amended version of existing DSRE data that can prevent memori
sation of test pair relations. Our contributions are threefold:
- Propose a multi-task learning setting for DSRE. Our results suggest that combination of both bag classification and bag reconstruction improves the target task.
- Propose a probabilistic model to make the space of sentence representations resemble that of a KB, promoting interpretability.
- Compare existing approaches on data without train-test pair overlaps to enforce fairer comparison between models.
2.1 Task Description
In DSRE, the bag setting is typically adopted. A model's input is a pair of named entities  $e_1, e_2$  (mapped to a Knowledge Base), and a bag of sentences  $B = \{s_1, s_2, \ldots, s_n\}$ , where the pair occurs, retrieved from a raw corpus. The goal of the task is to identify the relation(s), from a predefined set  $R$ , that the two entities share, based on the sentences in the bag  $B$ . Since each pair can share multiple relations at the same time, the task is considered a multi-label classification problem.
2.2 Overall Framework
Our proposed approach is illustrated in Figure 2. The main goal is to create a joint learning setting where a bag of sentences is encoded and reconstructed and, at the same time, the bag representation is used to predict relation(s) shared between two given entities. The architecture receives as input a bag of sentences for a given pair and outputs (i) predicted relations for the pair and (ii) the reconstructed sentences in the bag. The two outputs are produced by two branches: the left branch, corresponding to bag classification and the right branch, corresponding to bag reconstruction. Both branches start from a shared encoder and they communicate via the latent code of a VAE that is responsible for the information used in the representation and reconstruction of each sentence in the bag. Naturally, both branches have an effect on one another during training.
2.3 Bag Reconstruction
Autoencoders (Rumelhart et al., 1986) are encoder-decoder neural networks that are trained in an unsupervised manner, i.e., to reconstruct their input
Figure 2: Schematic of the model architecture.
(e.g. a sentence). They learn an informative representation of the input into a dense and smaller feature vector, namely the latent code. This intermediate representation is then used to fully reconstruct the original input. Variational Autoencoders (VAE) (Kingma and Welling, 2013) offer better generalisation capabilities compared to the former by sampling the features of the latent code from a prior distribution that we assume to be similar to the distribution of the data.
2.3.1 Encoder
We form the input of the network similarly to previous work. Each sentence in the input bag is transformed into a sequence of vectors. Words and positions are mapped into real-valued vectors via word embedding  $\mathbf{E}^{(w)}$  and position embedding layers  $\mathbf{E}^{(p)}$ , similarly to Lin et al. (2016). The concatenation of word  $(\mathbf{w})$  and position  $(\mathbf{p})$  embeddings  $\mathbf{x}_t = [\mathbf{w}_t; \mathbf{p}_t^{(e_1)}; \mathbf{p}_t^{(e_2)}]$  forms the representation of each word in the input sentence. A Bidirectional Long-Short Term Memory (BiLSTM) network (Hochreiter and Schmidhuber, 1997) acts as the encoder, producing contextualised representations for each word.
The representations of the left-to-right and right-to-left passes of the BiLSTM are summed to produce the output representation of each word  $t$ ,  $\mathbf{o}_t = \overrightarrow{\mathbf{o}_t} + \overleftarrow{\mathbf{o}_t}$ , as well as the representations of the last hidden  $\mathbf{h} = \overrightarrow{\mathbf{h}} + \overleftarrow{\mathbf{h}}$  and cell states  $\mathbf{c} = \overrightarrow{\mathbf{c}} + \overleftarrow{\mathbf{c}}$  of the input sentence. We use the last hidden and cell states of each sentence  $s$  to construct the pa
rameters of a posterior distribution  $q_{\phi}(\mathbf{z}|\mathbf{s})$  using two linear layers,
$$
\begin{array}{l} \boldsymbol {\mu} = \mathbf {W} _ {\mu} [ \mathbf {h}; \mathbf {c} ] + \mathbf {b} _ {\mu}, \\ \boldsymbol {\sigma} ^ {2} = \mathbf {W} _ {\sigma} [ \mathbf {h}; \mathbf {c} ] + \mathbf {b} _ {\sigma}, \\ \end{array}
$$
where  $\mu$  and  $\sigma^2$  are the parameters of a multivariate Gaussian, representing the feature space of the sentence. This distribution is approximated via a latent code  $\mathbf{z}$ , using the reparameterisation trick (Kingma and Welling, 2013) to enable back-propagation, as follows:
$$
\mathbf {z} = \boldsymbol {\mu} + \boldsymbol {\sigma} \odot \epsilon , \text {w h e r e} \epsilon \sim \mathcal {N} (\mathbf {0}, \mathbf {I}). \tag {2}
$$
This trick essentially forms the posterior as a function of the normal distribution.
2.3.2 Decoder
The decoder network is a uni-directional LSTM network, that reconstructs each sentence in the input bag. The input is formed in two steps. Firstly, the latent code  $\mathbf{z}$  is given as the initial hidden state of the decoder  $\mathbf{h}_0^{\prime}$  via a linear layer transformation. Secondly, the same latent code is concatenated with the representation of each word  $\mathbf{w}_t$  in the input sequence of the decoder.
$$
\mathbf {h} _ {0} ^ {\prime} = \mathbf {W} \mathbf {z} + \mathbf {b}, \quad \mathbf {x} _ {t} ^ {\prime} = [ \mathbf {w} _ {t}; \mathbf {z} ], \tag {3}
$$
A percentage of words in the decoder's input is randomly replaced by the UNK word to force the decoder to rely on the latent code for word prediction, similar to Bowman et al. (2016).
2.3.3 Learning
The optimisation objective of the VAE, namely Evidence Lower BOund (ELBO), is the combination of two losses. The first is the reconstruction loss that corresponds to the cross entropy between the actual sentence  $s$  and its reconstruction  $\hat{s}$ . The second is the Kullback-Leibler divergence  $(D_{\mathrm{KL}})$  between a prior distribution  $p_{\theta}(\mathbf{z})$ , which the latent code is assumed to follow, and the posterior  $q_{\phi}(\mathbf{z}|\mathbf{h})$ , which the decoder produces,
$$
\begin{array}{l} L _ {\mathrm {E L B O}} = \mathbb {E} _ {z \sim q _ {\phi} (z | h)} [ \log (p _ {\theta} (\mathbf {h} | \mathbf {z})) ] \\ - D _ {\mathrm {K L}} \left(q _ {\phi} (\mathbf {z} | \mathbf {h}) | | p _ {\theta} (\mathbf {z})\right) \tag {4} \\ \end{array}
$$
The first loss is responsible for the accurate reconstruction of each word in the input, while the second acts as a regularisation term that encourages the posterior of each sentence to be close to
the prior. Typically, an additional parameter  $\beta$  is introduced in front of the  $D_{\mathrm{KL}}$  to overcome KL vanishing, a phenomenon where the posterior collapses to the prior and the VAE essentially behaves as a standard autoencoder (Bowman et al., 2016).
2.4 Bag Classification
Moving on to the left branch of Figure 2, in order to represent a bag we first need to represent each sentence inside it. We realise this using information produced by the VAE as follows.
2.4.1 Sentence Representation
Given the contextualised output of the encoder  $\mathbf{o}$ , we construct entity representations  $\mathbf{e}_1$  and  $\mathbf{e}_2$  for a given pair in a sentence by averaging the word representations included in each entity. A sentence representation  $\mathbf{s}$  is formed as follows:
$$
\mathbf {e} _ {i} = \frac {1}{| e _ {i} |} \sum_ {k \in e _ {i}} \mathbf {o} _ {k}, \quad \mathbf {s} = \mathbf {W} _ {v} [ \mathbf {z}; \mathbf {e} _ {1}; \mathbf {e} _ {2} ], \tag {5}
$$
where  $|e_i|$  corresponds to the number of words inside the mention span of entity  $e_i$  and  $\mathbf{z}$  is the latent code of the sentence that was produced by the VAE, as described in Equation (2).
2.4.2 Bag Representation
In order to form a unified bag representation  $B$  for a pair, we adopt the popular selective attention approach introduced by Lin et al. (2016). In particular, we first map relations into real-valued vectors, via a relation embedding layer  $\mathbf{E}^{(r)}$ . Each relation embedding is then used as a query over the sentences in the bag, resulting in  $|R|$  bag representations for each pair,
$$
a _ {r} ^ {\left(s _ {i}\right)} = \frac {\exp \left(\mathbf {s} _ {i} ^ {\top} \mathbf {r}\right)}{\sum_ {j \in B} \exp \left(\mathbf {s} _ {j} ^ {\top} \mathbf {r}\right)}, \quad \mathbf {B} _ {r} = \sum_ {i = 1} ^ {| B |} a _ {r} ^ {\left(s _ {i}\right)} \mathbf {s} _ {i}, \tag {6}
$$
where  $\mathbf{r}$  is the embedding associated with relation  $r$ ,  $\mathbf{s}_i$  is the representation of sentence  $s_i \in B$ ,  $a_r^{(s_i)}$  is the weight of sentence  $s_i$  with relation  $r$  and  $\mathbf{B}_r$  is the final bag representation for relation  $r$ .
During classification, we select the probability of predicting a relation category  $r$ , using the bag representation that was constructed when the respective relation embedding  $\mathbf{r}$  was the query. Binary cross entropy loss is applied on the resulting predictions,
$$
\begin{array}{l} p (r = 1 | B) = \sigma \left(\mathbf {W} _ {c} \mathbf {B} _ {r} + \mathbf {b} _ {c}\right), \\ L _ {\mathrm {B C E}} = - \sum_ {r} y _ {r} \log p (r | B) \tag {7} \\ + (1 - y _ {r}) \log (1 - p (r | B)), \\ \end{array}
$$
where  $\mathbf{W}_c$  and  $\mathbf{b}_c$  are learned parameters of the classifier,  $\sigma$  is the sigmoid activation function,  $p(r|B)$  is the probability associated with relation  $r$  given a bag  $B$  and  $y_{r}$  is the ground truth for this relation with possible values 1 or 0.
2.5 Knowledge Base Priors
In the scenario where no KB information is incorporated into the model, we simply assume that the prior distribution of the latent code  $p_{\theta}(\mathbf{z})$  is a standard Gaussian with zero mean and identity covariance  $\mathcal{N}(\mathbf{0},\mathbf{I})$ .
To integrate information about the nature of triples into the bag-level classifier, we create KB-guided priors as an alternative to the standard Gaussian. In particular, we train a link prediction model, such as TransE (Bordes et al., 2013), on a subset of the Knowledge Graph that was used to originally create the dataset. Using the link prediction model, we obtain entity embeddings for the subset KB. A KB-guided prior can thus be constructed for each pair, as another Gaussian distribution with mean value equal to the KB pair representation and covariance as the identity matrix,
$$
p _ {\theta} (\mathbf {z}) \sim \mathcal {N} \left(\boldsymbol {\mu} _ {\mathrm {K B}}, \mathbf {I}\right), \text {w i t h} \boldsymbol {\mu} _ {\mathrm {K B}} = \mathbf {e} _ {h} - \mathbf {e} _ {t}, \tag {8}
$$
where  $\mathbf{e}_h$  and  $\mathbf{e}_t$  are the vectors for entities  $e_{\mathrm{head}}$  and  $e_{\mathrm{tail}}$  as resulted from training a link prediction algorithm on a KB.
The link prediction algorithm is trained to make representations of pairs expressing the same relations to be close in space. Hence, by using KB priors we try to force the distribution of sentences in a bag to follow the distribution of the pair in the KB. If one of the pair entities does not exist in the KB subset, the mean vector of the pair's prior will be zero, resulting in a standard Gaussian prior. Finally, KB priors are only used during training. Consequently, the model does not use any direct KB information during inference.
2.6 Training Objective
We train jointly bag classification and sentence reconstruction. The final optimisation objective
is formed as,
$$
L = \lambda L _ {\mathrm {B C E}} + (1 - \lambda) L _ {\mathrm {E L B O}}, \tag {9}
$$
where  $\lambda$  corresponds to a weight in [0,1]. We weigh the classification loss more than the ELBO to allow the model to better fit the target task.
We experiment with the following two datasets: NYT10. The widely used New York Times dataset (Riedel et al., 2010) contains 53 relation categories including a negative relation (NA) indicating no relation between two entities. We use the version of the data provided by the OpenNRE framework (Han et al., 2019), which removes overlapping pairs between train and test data. The dataset statistics are shown in Table 1. Additional information can be found in Appendix A.1.
For the choice of the Knowledge Base, we use a subset of Freebase $^2$  that includes 3 million entities with the most connections, similar to Xu and Barbosa (2019). For all pairs appearing in the test set of NYT10 (both positive and negative), we remove all links in the subset of Freebase to ensure that we will not memorise any relations between them (Weston et al., 2013). The resulting KB contains approximately 24 million triples.
WIKIDISTANT. The WikiDistant dataset is almost double the size of the NYT10 and contains 454 target relation categories, including the negative relation. It was recently introduced by Han et al. (2020) as a cleaner and more well structured bag-level dataset compared to NYT10, with fewer negative instances.
For the Knowledge Base, we use the version of Wikidata<sup>3</sup> provided by Wang et al. (2019b) (in particular the transductive split<sup>4</sup>), containing approximately 5 million entities. Similarly to Freebase, we remove all links between pairs in the test set from the resulting KB, which contains approximately 20 million triples after pruning.
Following prior work, we consider the Precision-Recall Area Under the Curve (AUC) as the primary
Dataset	Split	Instances	Bags	NA (%)
NYT10	Train	469,290	252,044	93.4
Val.	53,321	28,109	93.5
# Relations: 53	Test	172,448	96,678	97.9
WIKIDISTANT	Train	1,050,246	575,620	64.8
Val.	29,145	14,748	70.6
# Relations: 454	Test	28,897	15,509	72.0
Table 1: Datasets statistics. 'NA' corresponds to the 'no relation' category.
metric for both datasets. We additionally report Precision at  $N$  (P@N), that measures the percentage of correct classifications for the top  $N$  most confident predictions.
To obtain the KB priors, we train TransE on the subsets of Freebase and Wikidata using the implementation of the DGL-KE toolkit (Zheng et al., 2020) for 500K steps and a dimensionality equal to the dimension of the latent code. The main model was implemented with PyTorch (Paszke et al., 2019). We use the Adam (Kingma and Ba, 2014) optimiser with learning rate 0.001. KL logistic annealing is incorporated only in the case where the prior is the Normal distribution to avoid KL vanishing (Bowman et al., 2016). Early stopping is used to determine the best epoch based on the AUC score on the validation set. Words in the vocabulary are initialised with pre-trained, 50-dimensional GloVe embeddings (Pennington et al., 2014).
We limit the vocabulary size to the top 40K and 50K most frequent words for NYT10 and WIKIDISTANT, respectively. To enable fast training, we use Adaptive Softmax (Grave et al., 2017). The maximum sentence length is restricted to 50 for NYT10 and 30 words for WIKIDISTANT. Each bag in the training set is allowed to contain maximum 500 sentences selected randomly. For prediction on the validation and test sets, all sentences (with full length) are used.
In this work we compare with various models applied on the NYT10 dataset: PCNN-ATT (Lin et al., 2016) is one of the first neural models that uses a PCNN encoder and selective attention over the instances in a bag, similar to our approach. RESIDE (Vashisth et al., 2018), utilises syntactic, entity and relation type information as additional input to the network to assist classification. JOINT
2https://developers.google.com/freebase
<sup>3</sup>https://www.wikidata.org/
<sup>4</sup>https://deepgraphlearning.github.io/
project.wikidata5m
Method	Encoder	NYT 520K	NYT 570K
AUC (%)	P@N (%)	AUC (%)	P@N (%)
100	200	300	100	200	300
Baseline	34.94	74.0	67.5	67.0	43.59	84.0	77.0	75.3
+ pθ(z) ~ N(0, I)	BiLSTM	38.59	74.0	74.5	71.6	44.64	80.0	76.0	75.6
+ pθ(z) ~ N(μKB, I)	42.89	83.0	75.5	73.0	45.52	81.0	77.5	73.6
PCNN-ATT (Lin et al., 2016)	PCNN	32.66	71.0	67.5	62.6	36.25	76.0	72.5	64.0
JOINT NRE (Han et al., 2018)	CNN	30.62	60.0	57.0	55.3	40.15	75.8	-	68.0
RESIDE (Vashishth et al., 2018)	BiGRU	35.80	80.0	69.0	65.3	41.60	84.0	78.5	75.6
INTRA-INTER BAG (Ye and Ling, 2019)	PCNN	34.41	82.0	74.0	69.0	42.20	91.8	84.0	78.7
DISTRE (Alt et al., 2019)	GPT-2	42.20	68.0	67.0	65.3	-	-	-	-
Table 2: Performance comparison between different methods on the NYT10 test set for the two different versions of the dataset. Results in the 520K column are re-runs of existing implementations, except for DISTRE. Results on the 570K column are taken from the respective publications.
Method	AUC (%)	P@N (%)
100	200	300
Baseline	28.54	94.0	93.0	88.3
+ pθ(z) ~ N(0, I)	30.59	96.0	93.5	89.3
+ pθ(z) ~ N(μKB, I)	29.54	92.0	89.0	90.0
PCNN-ATT (Han et al., 2020)	22.20	-	-	-
w/o non KB-prior pairs (72% of training pairs preserved)
Baseline	26.16	88.0	85.0	82.6
+ pθ(z) ~ N(0, I)	27.46	90.0	88.0	84.6
+ pθ(z) ~ N(μKB, I)	28.38	94.0	95.0	89.3
Table 3: Performance comparison on the WIKIDISTANT test set.
NRE (Han et al., 2018) jointly trains a textual relation extraction component and a link prediction component by sharing attention query vectors among the two. INTRA-INTER BAG (Ye and Ling, 2019) applies two attention mechanisms inside and across bags to enforce similarity between bags that share the same relations. DISTRE (Alt et al., 2019) uses a pre-trained Transformer model, instead of a recurrent or convolutional encoder, fine-tuned on the NYT10 dataset.
We report results on both the filtered data (520K) that do not contain train-test pair overlaps, as well as the non-filtered version (570K) to better compare with prior work. With the exception of DISTRE, all prior approaches were originally applied on the 570K version. Hence, performance of prior work on the 520K version corresponds to re-runs of existing implementations (via their open-source code). For the non-filtered version, results are taken from the respective publications.