# Information Extraction of Cybersecurity Concepts: An LSTM Approach
## 3 Models
### Background  
LSTMs are a type of RNNs that have the ability to detect and learn patterns in a sequence of input data. Sequences of data can be stock market time series, natural language text or voice, genomes, etc. RNNs combines the current input (e.g., current word in the text) with the knowledge learned from the previous input (e.g., previous words in the text). While RNNs perform well with short sequence, they suffer from an issue called vanishing or exploding grading issue when the processed sequence becomes too long. When the input becomes long, RNNs become difficult to train especially when the number of model parameters becomes large. In this case, the training is very unlikely to converge.  
The LSTM architecture was introduced by [PERSON] et al. [24] in 1997 to address the problem of long-term dependency learning. The model introduced the concept of a memory cell, shown in Figure 2, which preserves long-term dependency state over time in the memory. Since then, several variants have been proposed. LSTMs and RNNs in general have been applied to various NLP tasks including advanced tasks such as recognizing textual entailment (RTE) [25]. An LSTM cell, at each step t, is defined as a collection of vectors \(R_{d}\) (where \(d\) is the memory dimension of the LSTM). These vectors are defined as follows: Input gate \(i_{t}\), forget gate \(f_{t}\), output gate \(o_{t}\), memory cell \(C_{t}\), next memory state \(C_{t+1}\), and a hidden state \(h_{t}\). The entries of the vectors \(i_{t}\), \(f_{t}\), and \(o_{t}\) are in the range [0, 1]. The transition from a state to the next is defined by the following equations:  
\[\begin{array}{ll}i_{t}=&\sigma(W_{i}x_{t}+\ U_{i}h_{t-1}+\ b_{i})\\ f_{t}=&\sigma\big{(}W_{f}x_{t}+\ U_{f}h_{t-1}+\ b_{f}\big{)}\\ o_{t}=&\sigma(W_{o}x_{t}+\ U_{o}h_{t-1}+\ b_{o})\end{array}\qquad\qquad\begin{array}{ll}C_{t}= \ tanh(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\ c_{t}=i_{t}\big{\circ}\ \big{\rangle}\ \widehat{C}_{t}+f_{t}\big{\circ}\ \alpha_{t-1}\\ h_{t}=o_{t}\big{\circ}\ \big{\rangle}\ \tanh(c_{t}).\end{array}\]  
Here, \(x_{t}\) is the input of the current time step, the symbol \(\sigma\) represents the logistic sigmoid function, and \(\Theta\) represents element-wise multiplication. The forget gate \(f_{t}\) calculates the amount of the previous cell state that should be forgotten. The input gate \(i_{t}\) calculates the amount of information passed as input to each unit, and the output gate \(o_{t}\) calculates the amount of the exposure of the internal memory state. The hidden state vector \(h_{t}\) is therefore the partial view of the state of the unit's internal memory cell. The model learns the representation of information over time because the values of the different gating variables vary for each vector element in \(R_{d}\).  
Word embeddings also contributed to the encouraging results of LSTMs in the NLP field [26]. They were introduced by [PERSON] et al. [27] and represented a significant improvement over one-hot encoding vectors, the traditional way of representing text [16]. Their first advantage is that they are low-dimensional vectors of usually 200-300 length as opposed to one-hot vectors, which tend to be as long as the number of words in the vocabulary of the text. The second advantage is that they capture the semantic relationship between words that had a big impact on NLP tasks such as NER. For example, when the value of a vector representing the word 'King' is subtracted from the 'Queen' vector, we get a result that is equal to the difference between the vectors representing the words 'Man' and 'Woman'. Another characteristic of word embedding is word clustering. Words that are from the same family are grouped in separate cluster in the word embeddings space. For instance, the company names 'Microsoft' and 'Oracle' appear in the same cluster whereas the product names 'WebLogic' and 'Eclipse' appear in another cluster.