# A Text-Generated Method to Joint Extraction of Entities and Relations
## 3 Materials and Methods
### Model Description
#### 3.2.2 Decoder  
The decoder is aimed to generate tokens consisting of a layer of unidirectional LSTM.  
\[o_{t},s_{t}=cell([x_{t-1},h_{t}^{*}],s_{t-1}) \tag{2}\]  
In Equation (2), \(cell\) is an LSTM unit, during training \(x_{t-1}\) represents the embedding of the previous word in the target output sequence; in the test phase, it represents the embedding of the word generated by the model at the previous step, and \(s_{t-1}\) represents the decoding state at time \(t-1\). At the same time, we use the attention mechanism to calculate the weight of the hidden vectors in the encoder at the current decoding time, the attention distribution can be viewed as a probability distribution over the source words. The greater the attention weight, the greater the influence on the word generated at the current decoding time. In addition, \(h_{t}^{*}\) represents the weighted sum of the encoder hidden states based on attention weight, i.e., context vector. We use the attention calculation method of [14] to obtain the context vector:  
\[\begin{split} e_{i}^{t}&=v^{T}tanh(W_{h}h_{i}+W_{s }s_{t}+b_{attn})\\ a^{t}&=softmax(e^{t})\\ h_{t}^{*}&=\sum_{i}a_{i}^{t}h_{i}\end{split} \tag{3}\]  
where \(v\), \(W_{h}\), \(W_{s}\), and \(b_{attn}\) in Equation (3) are learnable parameters, \(h_{i}\) represents the hidden state vector of the encoder at time \(i\).  
\(a^{t}\) is the influence weight in attention which is also a probability distribution. When the model wants to 'copy' a word from the original text, the word with the largest weight value will be selected as the predicted word. Therefore, we also call \(a^{t}\) 'pointer'.  
Then the context vector is concatenated with the decoder state \(s_{t}\) and fed through linear layers to produce the vocabulary distribution \(P_{vocab}\):  
\[P_{vocab}=softmax(W_{v}[s_{t},h_{t}^{*}]+b_{v}) \tag{4}\]  
where \(W_{v}\) and \(b_{v}\) are learnable parameters, \(P_{vocab}\) is a probability distribution over all words in the predefined vocabulary.  
To make the model have the ability to copy words from the source text, and to retain the ability to select words through the predefined vocabulary, we calculate a generation probability \(p_{gen}\in[0,1]\) at each decoding step, refer to [16]:  
\[p_{gen}=\sigma(w_{h}^{T}*h_{t}^{*}+w_{s}^{T}s_{t}+w_{x}^{T}x_{t}+b_{gen}) \tag{5}\]  
where \(w_{h}\), \(w_{s}\), \(w_{x}\) and \(b_{gen}\) are learnable parameters and \(\sigma\) is the sigmoid function, \(p_{gen}\) is calculated by context vector \(h_{t}^{*}\), decoder state \(s_{t}\), and decoder input \(x_{t}\). \(p_{gen}\) is aimed to select the word output at the current decoding time, with a greater probability of copying from the source text or more likely to be generated from the predefined vocabulary.  
And now we get the final probability distribution:  
\[P(w)=p_{gen}P_{vocab}(w)+(1-p_{gen})\sum_{i}a_{i}^{t} \tag{6}\]  
where \(a^{t}\) represents the attention weight on the hidden states of the encoder. We select the word with the greatest probability as the predicted word of the current step. In the test phase, the embedding of this word will be sent to the next decoding step.  
For entities, the model will tend to copy from the source text, so that for entities that do not appear in the predefined vocabulary (unseen entities), the model also has the ability to correctly identify; for relational expressions, the model is more tend to generate from the vocabulary.