# A Survey of Information Extraction Based on Deep Learning
## 2 Entity Relationship Extraction Based on Deep Learning
### Supervised Learning--Pipeline Extraction
#### 2.1.2 RNNs  
Compared with statistics-based methods, CNN-based methods have shown good progress, but their ability to extract temporal features is weak, especially when the \"distance\" between two entities is relatively large, which limits their performance. Circulating neural networks are considered the most suitable model for temporal feature extraction. In network training, a traditional RNN is prone to gradient disappearance and gradient explosion problems; therefore, RNNs have difficulty dealing with long-term dependence in practice [22]. The LSTM architecture is a special RNN that solves this problem. LSTM models perform better for longer sequences than ordinary RNNs. To improve the performance of these models in handling multiple entities in long text and sentences, [PERSON] et al. proposed a relationship-classified neural network structure based on bi-directional long- short-term memory networks. The model implemented a concat-attention mechanism to capture the most important context words in sentences, a piecewise attention mechanism to improve its performance in processing long sentences, and a tensor-based entity description to overcome the problem of performance declines when multiple entities appear in a sentence [23]. A recurrent neural network with multiple semantic heterogeneous embeddings was built in a self-training framework, as proposed by [PERSON] et al. The framework uses labelled, unlabelled, and social media data to model relational context using a bi-directional recurrent neural network with scalability and versatility. Compared with the SVM model utilizing complex features, the RNN-based time-relation extracted self-training framework performed well on the original features, modelled the sentence structure well, and was highly scalable and generalizable [24].  
At present, most relationship extraction methods extract the relationships reflected by a single entity pair in a certain sentence; however, in long text, prior methods focused on the representation of entire sentences while ignoring the loss of information. The lack of analysis of words and syntactic characteristics leads to poor sentence performance, especially in Chinese relationship extraction. [PERSON] et al. constructed a bi-directional gate recursion unit (BiGRU) network relationship extraction model based on character- and sentence-level attention mechanisms to extract the relationship between diseases, symptoms, and tests [25]. An end-to-end relationship extraction method based on a bi-directional gated recursive unit (BiGRU) neural network and a dual attention mechanism was proposed by [PERSON] et al. The model is designed to focus on words with a decisive influence on sentence-relation extraction and capture relational semantic and directional words. To improve performance, it generates word vectors using a pre-trained FastText model and dynamically adjusts word vectors via FastText according to the context [11]. A large number of relational facts are expressed in multiple sentences. Complex inter  
Figure 3: The architecture of the dual CNN, includes an input layer, a convolutional layer, a pooling layer, a merging layer, and a fully connected layer [20].  
relationships often exist between multiple entities in a document. Some relationships can become coherent only on a global scale. [PERSON] et al. proposed a global-level relationship extractor model, GREG (as shown in Figure 4), which contained two modules, including a context-aware long-and short-term memory (LSTM) relationship extraction module and a knowledge graph constructor module for generating a knowledge graph from a given document. An unsupervised version of this method is planned in the future, which would be much closer to full automation [26].