Venue: EMNLP 2017

# Deep Residual Learning for Weakly-Supervised Relation Extraction

 Yi Yao Huang

Department of Electrical Engineering

National Taiwan University

Taipei, Taiwan

b02901042@ntu.edu.tw William Yang Wang

Department of Computer Science

University of California, Santa Barbara

Santa Barbara, CA 93106 USA

william@cs.ucsb.edu

###### Abstract

Deep residual learning (ResNet) He et al. (2016) is a new method for training very deep neural networks using identity mapping for shortcut connections. ResNet has won the ImageNet ILSVRC 2015 classification task, and achieved state-of-the-art performances in many computer vision tasks. However, the effect of residual learning on noisy natural language processing tasks is still not well understood. In this paper, we design a novel convolutional neural network (CNN) with residual learning, and investigate its impacts on the task of distantly supervised noisy relation extraction. In contradictory to popular beliefs that ResNet only works well for very deep networks, we found that even with 9 layers of CNNs, using identity mapping could significantly improve the performance for distantly-supervised relation extraction.

## 1 Introduction

Relation extraction is the task of predicting attributes and relations for entities in a sentence Zelenko et al. (2003); Bunescu and Mooney (2005); GuoDong et al. (2005). For example, given a sentence _"**Barack Obama** was born in **Honolulu**, Hawaii.**", a relation classifier aims at predicting the relation of _"bornInCity"_. Relation extraction is the key component for building relation knowledge graphs, and it is of crucial significance to natural language processing applications such as structured search, sentiment analysis, question answering, and summarization.

A major issue for relation extraction is the lack of labeled training data. In recent years, distant supervision Mintz et al. (2009); Hoffmann et al. (2011); Surdeanu et al. (2012) emerges as the most popular method for relation extraction-- it uses knowledge base facts to select a set of noisy instances from unlabeled data. Among all the machine learning approaches for distant supervision, the recently proposed Convolutional Neural Networks (CNNs) model Zeng et al. (2014) achieved the state-of-the-art performance. Following their success, Zeng et al. (2015) proposed a piece-wise max-pooling strategy to improve the CNNs. Various attention strategies Lin et al. (2016); Shen and Huang (2016) for CNNs are also proposed, obtaining impressive results. However, most of these neural relation extraction models are relatively shallow CNNs--typically only one convolutional layer and one fully connected layer were involved, and it was not clear whether deeper models could have benefits on distilling signals from noisy inputs in this task.

In this paper, we investigate the effects of training deeper CNNs for distantly-supervised relation extraction. More specifically, we designed a convolutional neural network based on residual learning He et al. (2016)--we show how one can incorporate word embeddings and position embeddings into a deep residual network, while feeding identity feedback to convolutional layers for this noisy relation prediction task. Empirically, we evaluate on the NYT-Freebase dataset Riedel et al. (2010), and demonstrate the state-of-the-art performance using deep CNNs with identify mapping and shortcuts. In contrast to popular beliefs in vision that deep residual network only works for very deep CNNs, we show that even with a moderately deep CNNs, there are substantial improvements over vanilla CNNs for relation extraction. Our contributions are three-fold:

* We are the first to consider deeper convolutional neural networks for weakly-supervised

relation extraction using residual learning;
* We show that our deep residual network model outperforms CNNs by a large margin empirically, obtaining state-of-the-art performances;
* Our identity mapping with shortcut feedback approach can be easily applicable to any variants of CNNs for relation extraction.

## 2 Deep Residual Networks for Relation Extraction

In this section, we describe a novel deep residual learning architecture for distantly supervised relation extraction. Figure 1 describes the architecture of our model.

### Vector Representation

Let \({\textbf{x}_{i}}\) be the \(i\)-th word in the sentence and _e1_, _e2_ be the two corresponding entities. Each word will access two embedding look-up tables to get the word embedding \({\textbf{WF}_{i}}\) and the position embedding \({\textbf{PF}_{i}}\). Then, we concatenate the two embeddings and denote each word as a vector of \({\textbf{v}_{i}}=[{\textbf{WF}_{i}},{\textbf{PF}_{i}}]\).

#### 2.1.1 Word Embeddings

Each representation \({\textbf{v}_{i}}\) corresponding to \({\textbf{x}_{i}}\) is a real-valued vector. All of the vectors are encoded in an embeddings matrix \({\textbf{V}_{w}}\in\mathbb{R}^{d_{w}\times|V|}\) where \(V\) is a fixed-sized vocabulary.

#### 2.1.2 Position Embeddings

In relation classification, we focus on finding a relation for entity pairs. Following Zeng et al. (2014), a PF is the combination of the relative distances of the current word to the first entity \(e_{1}\) and the second entity \(e_{2}\). For instance, in the sentence "_Steve_Johbs is the founder of Apple._", the relative distances from _founder_ to \(e_{1}\) (_Steve_Job_) and \(e_{2}\) are 3 and -2, respectively. We then transform the relative distances into real valued vectors by looking up one randomly initialized position embedding matrices \({\textbf{V}_{p}}\in\mathbb{R}^{d_{p}\times|P||}\) where P is fixed-sized distance set. It should be noted that if a word is too far from entities, it may be not related to the relation. Therefore, we choose maximum value \(e_{max}\) and minimum value \(e_{min}\) for the relative distance.

In the example shown in Figure 1, it is assumed that \(d_{w}\) is 4 and \(d_{p}\) is 1. There are two position embeddings: one for \(e_{1}\), the other for \(e_{2}\). Finally, we concatenate the word embeddings and position embeddings of all words and denote a sentence of length \(n\) (padded where necessary) as a vector

\[{\textbf{v}}={\textbf{v}_{1}}\oplus{\textbf{v}_{2}}\oplus...\oplus{\textbf{ v}_{n}}\]

where \(\oplus\) is the concatenation operator and \({\textbf{v}_{i}}\in\mathbb{R}^{d}\) (\(d=d_{w}+d_{p}\times 2\)).

### Convolution

Let \({\textbf{v}_{i:i+j}}\) refer to the concatenation of words \({\textbf{v}_{i}},{\textbf{v}_{i+1}},...,{\textbf{v}_{i+j}}\). A convolution operation involves a _filter_\({\textbf{w}}\in\mathbb{R}^{hd}\), which is applied to a window of \(h\) words to produce a new feature. A feature \(c_{i}\) is generated from a window of word \({\textbf{v}_{i:i+h-1}}\) by

\[c_{i}= f({\textbf{w}}\cdot{\textbf{x}_{i:i+h-1}}+b)\]

Here \(b\in\mathbb{R}\) is a bias term and \(f\) is a non-linear function. This filter is applied to each possible window of words from \({\textbf{v}_{1}}\) to \({\textbf{v}_{n}}\) to produce _feature_\({\textbf{c}}=[c_{1},c_{2},...,c_{n-h+1}]\) with \({\textbf{c}}\in\mathbb{R}^{s}(s=n-h+1)\).

### Residual Convolution Block

Residual learning connects low-level to high-level representations directly, and tackles the vanishing gradient problem in deep networks. In our model, we design the residual convolution block by applying shortcut connections. Each residual convolutional block is a sequence of two convolutional layers, each one followed by an ReLU activation. The kernel size of all convolutions is \(h\), with padding such that the new feature will have the same size as the original one. Here are two convolutional _filter_\({\textbf{w}_{1}}\), \({\textbf{w}_{2}}\in\mathbb{R}^{h\times 1}\). For the first convolutional layer:

\[\tilde{c}_{i}= f({\textbf{w}_{1}}\cdot c_{i:i+h-1}+b_{1})\]

For the second convolutional layer:

\[\dot{c}_{i}= f({\textbf{w}_{2}}\cdot\tilde{c}_{i:i+h-1}+b_{2})\]

Here \(b_{1}\), \(b_{2}\) are bias terms. For the residual learning operation:

\[{\textbf{c}}={\textbf{c}}+\hat{{\textbf{c}}}\]

Conveniently, the notation of **c** on the left is changed to define as the output vectors of the block. This operation is performed by a shortcut connection and element-wise addition. This block will be multiply concatenated in our architecture.

### Max Pooling, Softmax Output

We then apply a max-pooling operation over the _feature_ and take the maximum value \(\hat{c}=max\{{\textbf{c}}\}\) 

We have described the process by which _one_ feature is extracted from _one_ filter. Take all features into one high level extracted feature \({\bf z}=[\hat{c}_{1},\hat{c}_{2},...,\hat{c}_{m}]\)(note that here we have \(m\) filters). Then, these features are passed to a fully connected softmax layer whose output is the probability distribution over relations. Instead of using \(y={\bf w}\cdot{\bf z}+b\) for output unit \(y\) in forward propagation, dropout uses \(y={\bf w}\cdot({\bf z}\circ{\bf r})+b\) where \(\circ\) is the element-wise multiplication operation and \({\bf r}\in\mathbb{R}^{m}\) is a 'masking' vector of Bernoulli random variables with probability \(p\) of being 1. In the test procedure, the learned weight vectors are scaled by \(p\) such that \(\hat{\mathbf{w}}=p{\bf w}\) and used (without dropout) to score unseen instances.

## 3 Experiments

### Experimental Settings

In this paper, we use the word embeddings released by Lin et al. (2016) which are trained on the NYT-Freebase corpus Riedel et al. (2010). We fine tune our model using validation on the training data. The word embedding is of size 50. The input text is padded to a fixed size of 100. Training is performed with tensorflow adam optimizer, using a mini-batch of size 64, an initial learning rate of 0.001. We initialize our convolutional layers following Glorot and Bengio (2010). The implementation is done using Tensorflow 0.11. All experiments are performed on a single NVidia Titan X (Pascal) GPU. In Table 1 we show all parameters used in the experiments.

We experiment with several state-of-the-art baselines and variants of our model.

* **CNN-B**: Our implementation of the CNN baseline Zeng et al. (2014) which contains one convolutional layer, and one fully connected layer.
* **CNN+ATT**: CNN-B with attention over instance learning Lin et al. (2016).
* **PCNN+ATT**: Piecewise CNN-B with attention over instance learning Lin et al. (2016).
* **CNN**: Our CNN model which includes one convolutional layer and three fully connected layers.
* **CNN-x**: Deeper CNN model which has x convolutional layers. For example, CNN-9 is a model constructed with 9 convolutional layers (1 + 4 residual cnn block without identity shortcut) and three fully connected layers.
* **ResCNN-x**: Our proposed CNN-x model with residual identity shortcuts.

We evaluate our models on the widely used NYT freebase larger dataset Riedel et al. (2010). Note that ImageNet dataset used by the original ResNet paper He et al. (2016) has 1.28 million training instances. NYT freebase dataset includes 522K training sentences, which is the largest dataset in relation extraction, and it is the only suitable dataset to train deeper CNNs.

### NYT-Freebase Dataset Performance

The advantage of this dataset is that there are 522,611 sentences in training data and 172,448 sentences in testing data and this size can support

Figure 1: The architecture of ResCNN used for relation extraction.

 

us to train a deep network. Similar to previous work [26, 10], we evaluate our model using the held-out evaluation. We report both the aggregate curves precision/recall curves and Precision@N (P@N).

In Figure 2, we compare the proposed ResCNN model with various CNNs. First, CNNs with multiple fully-connected layers obtained very good results, which is a novel finding. Second, the results also suggest that deeper CNNs with residual learning help extracting signals from noisy distant supervision data. We observe that overfitting happened when we try to add more layers and the performance of CNN-9 is much worse than CNN. We find that ResNet can solve this problem and ResCNN-9 obtains better performance as compared to CNN-B and CNN and dominates the precision/recall curve overall.

We show the effect of depth in residual networks in Figure 3. We observe that ResCNN-5 is worse than CNN-5 because the ResNet does not work well for shallow CNNs, and this is consistent with the original ResNet paper. As we increase the network depth, we see that CNN-9 does overfit to the training data. With residual learning, both ResCNN-9 and ResCNN-13 provide significant improvements over CNN-5 and ResCNN-5 models. In contradictory to popular beliefs that ResNet only works well for very deep networks, we found that even with 9 layers of CNNs, using identity mapping could significantly improve the performance learning in a noisy input setting.

The intuition of ResNet help this task in two aspect. First, if the lower, middle, and higher levels learn hidden lexical, syntactic, and semantic representations respectively, sometimes it helps to bypass the syntax to connect lexical and semantic space directly. Second, ResNet tackles the vanishing gradient problem which will decrease the effect of noise in distant supervision data.

In Table 2, we compare the performance of our models to state-of-the-art baselines. We show that our ResCNN-9 outperforms all models that do not select training instances. And even without piecewise max-pooling and instance-based attention, our model is on par with the PCNN+ATT model.

For the more practical evaluation, we compare the results for precision@N where N is small (1, 5, 10, 20, 50) in Table 3. We observe that our ResCNN-9 model dominates the performance when we predict the relation in the range of higher probability. ResNet helps CNNs to focus on the highly possible candidate and mitigate the noise effect of distant supervision. We believe that residual connections actually can be seen as a form of renormalizing the gradients, which prevents the model from overfitting to the noisy distant supervision data.

In our distant-supervised relation extraction experience, we have two important observations: (1) We get significant improvements with CNNs adding multiple fully-connected layers. (2) Resid

\begin{table}
\begin{tabular}{|l||c|c|c|c|} \hline P@N(\%) & 100 & 200 & 300 & Mean \\ \hline CNN+ATT & 76.2 & 68.6 & 59.8 & 68.2 \\ \hline PCN+ATT & **76.2** & **73.1** & **67.4** & **72.2** \\ \hline \hline CNN-B & 41.0 & 40.0 & 41.0 & 40.7 \\ \hline CNN & 64.0 & 61.0 & 55.3 & 60.1 \\ \hline CNN-5 & 64.0 & 58.5 & 54.3 & 58.9 \\ \hline ResCNN-5 & 57.0 & 57.0 & 54.3 & 56.1 \\ \hline CNN-9 & 56.0 & 54.0 & 49.7 & 53.2 \\ \hline ResCNN-9 & **79.0** & **69.0** & **61.0** & **69.7** \\ \hline ResCNN-13 & 76.0 & 65.0 & 60.3 & 67.1 \\ \hline \end{tabular}
\end{table}
Table 2: P@N for relation extraction with different models. Top: models that select training data. Bottom: models without selective attention.

Figure 3: Varying the depths of our models.

Figure 2: Comparing ResCNN to different CNNs.

 

ual learning could significantly improve the performance for deeper CNNs.

## 4 Conclusion

In this paper, we introduce a deep residual learning method for distantly-supervised relation extraction. We show that deeper convolutional models help distill signals from noisy inputs. With shortcut connections and identify mapping, the performances are significantly improved. These results aligned with a recent study Conneau et al. (2017), suggesting that deeper CNNs do have positive effects on noisy NLP problems.

## References

* Bunescu and Mooney (2005) Razvan Bunescu and Raymond J Mooney. 2005. Subsequence kernels for relation extraction. In _NIPS_, pages 171-178.
* Conneau et al. (2017) Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. 2017. Very deep convolutional networks for natural language processing. _EACL 2017_.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _International Conference on Artificial Intelligence and Statistics_, pages 249-256.
* GuoDong et al. (2005) Zhou GuoDong, Su Jian, Zhang Jie, and Zhang Min. 2005. Exploring various knowledge in relation extraction. In _Proceedings of the 43rd annual meeting on association for computational linguistics_, pages 427-434. Association for Computational Linguistics.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 770-778.
* Hoffmann et al. (2011) Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies Volume 1_, pages 541-550. Association for Computational Linguistics.
* Lin et al. (2016) Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In _ACL 2016_.
* Mintz et al. (2009) Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2_, pages 1003-1011. Association for Computational Linguistics.
* Riedel et al. (2010) Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In _Machine Learning and Knowledge Discovery in Databases_, pages 148-163. Springer.
* Shen and Huang (2016)Yatian Shen and Xuanjing Huang. 2016. Attention-based convolutional neural network for semantic relation extraction. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_.
* Surdeanu et al. (2012) Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance multi-label learning for relation extraction. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_, pages 455-465. Association for Computational Linguistics.
* Zelenko et al. (2003) Dmitry Zelenko, Chinatsu Aoue, and Anthony Richardella. 2003. Kernel methods for relation extraction. _Journal of machine learning research_, 3(Feb):1083-1106.
* Zeng et al. (2015) Daojian Zeng, Kang Liu, Yuko Chen, and Jun Zhao. 2015. Distant supervision for relation extraction via piecewise convolutional neural networks. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal_, pages 17-21.
* Zeng et al. (2014) Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In _COLING_, pages 2335-2344.

\begin{table}
\begin{tabular}{|l||c|c|c|c|} \hline P@N(\%) & 1 & 5 & 10 & 20 & 50 \\ \hline PCNN+ATT & **1** & 0.8 & **0.9** & 0.75 & 0.7 \\ \hline ResCNN-9 & **1** & **1** & **0.9** & **0.9** & **0.88** \\ \hline \end{tabular}
\end{table}
Table 3: P@N for relation extraction with different models where N is small. We get the result of PCNN+ATT using their public source code.