These two words often appear nearby malicious file information and are then extracted as contextual keywords in Section SECREF14 . The token “cr.sh” is then correctly identified as a token of malicious file information by the use of the contextual features.

Analysis of Contextual Features	The proposed model provides an intuitive way to inspect the contextual information of each given token. As described in Section SECREF14 , we initialize the contextual features of each given token using the automatically extracted contextual keywords and jointly learn them during the process of training with the whole ANN model. To prove the effectiveness of the contextual features, we visualize the learned weights martix of each contextual keyword of contextual feature and show several examples in Fig. FIGREF28 . Each row of the matrix in each plot indicates the weights of contextual keywords for the given tokens. From this we see which contextual keyword were considered more important to represent the contextual information of the given token. We can see from the matrix in Fig. FIGREF28 that, for the token “spearphshing”, which is an email-spoofing attack method, the contextual keyword “email” has the largest weight. For the malware “SunOrcal”, which drops several malicious executable files, contextual keywords “droppper” and “dropper” have larger weights than other contextual keywords such as “ascii”, “port” and “type”.

Training the Proposed Model with Bilingual Data	Even though security articles are written in different languages, most of the IOCs are written in English, and are described in a similar pattern. Therefore, using multilingual corpora could be a solution for addressing the lack of annotated data, and the performance of the proposed model is expected to be improved by extending the training set. To examine the hypothesis, we ran a number of additional experiments using both the English dataset and Chinese dataset, both of which are described in Section SECREF21 and are not parallel data or comparable data. As pre-trained word embeddings for the bilingual training dataset, we applied a cross-lingual word embedding obtained by the work of Duong el al  , where the English-Chinese cross-lingual dictionary is obtained by simply translating all the English words from English dataset to Chinese and Chinese words from Chinese dataset to English using Google translation. As contextual feature vector, we concatenate the contextual feature vector obtained from English dataset with the contextual feature vector obtained from Chinese dataset. Then we merge the English training set and the Chinese training set into one set and train the proposed model with the merged bilingual training set.

Conclusions	To conclude, in this paper, we newly introduce a multi-head self-attention module and contextual features to the neural based sequence labelling model, which significantly improved the performance in the task of IOC identification. Based on the evaluation results of our experiments, our proposed model is proved effective on both the English test set and the Chinese test set. We further evaluated the proposed model by training the proposed model using both the English training set and the Chinese training set and compared it with models that are trained with only one training set, where the model trained with the merged bilngual training set performs better. One of our future works is to integrate the contextual embeddings from the bidirectional language model into our proposed model. The pretrained neural language models are proved effective in the sequence labelling models  ,  ,  . It is expected to improve the performance of the proposed model by integrating both the contextual features and contextual embeddings into the neural sequence labelling model.

Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling	Indicators of Compromise (IOCs) are artifacts observed on a network or in an operating system that can be utilized to indicate a computer intrusion and detect cyber-attacks in an early stage. Thus, they exert an important role in the field of cybersecurity. However, state-of-the-art IOCs detection systems rely heavily on hand-crafted features with expert knowledge of cybersecurity, and require large-scale manually annotated corpora to train an IOC classifier. In this paper, we propose using an end-to-end neural-based sequence labelling model to identify IOCs automatically from cybersecurity articles without expert knowledge of cybersecurity. By using a multi-head self-attention module and contextual features, we find that the proposed model is capable of gathering contextual information from texts of cybersecurity articles and performs better in the task of IOC identification. Experiments show that the proposed model outperforms other sequence labelling models, achieving the average F1-score of 89.0% on English cybersecurity article test set, and approximately the average F1-score of 81.8% on Chinese test set.

In this paper, we further introduce a multi-head self-attention module and contextual features to the ANN model so that the proposed model can perform better in gathering the contextual information from the unstructured text for the task of IOCs identification. Based on the results of our experiments, our proposed approach achieves an average precision of 93.1% and the recall of 85.2% on English cybersecurity article test set, and an average precision of 82.9% and recall of 80.7% on Chinese test set. We further evaluate the proposed model by training the model using both the English dataset and Chinese dataset, which even achieves better performance.

The set of contextual keywords INLINEFORM6 are automatically extracted from the annotated texts, where each contextual keyword INLINEFORM7 ( INLINEFORM8 ) satisfies the following conditions:  INLINEFORM0 , where INLINEFORM1 is the set of manually annotated IOCs and INLINEFORM2 is a the lower bound of the frequency.  INLINEFORM0 is not a punctuation or stopword. Note that we extract contextual keywords only from manually annotated data (e.g., training set), while we compute the contextual feature vector in all of the unlabeled data. According to this definition, it is obvious that the dimension of the contextual feature vector is as the same as the number of extracted contextual keywords. The size of window INLINEFORM0 and the lower bound of frequency INLINEFORM1 are then tuned by the validation set.

Contextual Features	IOCs in cybersecurity articles are often described in a predictable way: being connected to a set of contextual keywords  ,  . For example, a human user can infer that the word “ntdll.exe” is the name of a malicious file on the basis of the words “download” and “compromised” from the text shown in Fig. FIGREF1 . By analyzing the whole corpus, it is interesting that malicious file names tends to co-occur with words such as "download", "malware", "malicious", etc. In this work, we consider words that can indicate the characteristics of the neighbor words as contextual keywords and develop an approach to generate features from the automatically extracted contextual keywords. Taking the above into account, we introduce the contextual feature vector INLINEFORM0 for a given input token INLINEFORM1 , where the INLINEFORM2 element of INLINEFORM3 is defined as follows: DISPLAYFORM0   INLINEFORM0 is the frequency of token INLINEFORM1 in the whole corpus, while INLINEFORM2 is the frequency of contextual keyword INLINEFORM3 from the windowed portions of the texts centering on the token INLINEFORM4 in the whole corpus and INLINEFORM5 is the size of window.

For non-IOC token “socket”, contextual keywords “gateway” and “port” yield larger weights than other keywords because "socket" tends to co-occur with “gateway” and “port”. We further calculate the average weight of each contextual keyword and show the top 10 and bottom 10 largest weighted contextual keywords in TABLE TABREF29 . From this we see that contextual keywords such as, “hash” and “filename”, which tends to co-occur with malicious filenames, have the largest weights for IOCs, while the contextual keywords such as “ascii”, “password” have the largest weights for non-IOCs. Here, it is interesting to find that contextual keyword “dropped” and “droppper”, which tend to co-occur with malicious file information and malwares, yield large weights for IOCs but small weights for non-IOCs. The proposed ANN model benefits from the differences of contextual information between IOCs and non-IOCs that is represented by the contextual features, and thus, achieves better performance than the previous works.

Furthermore, we introduce some features to defined IOCs to improve the performance of the proposed model on a very small amount of training data. Here, we define two types of features, i.e., spelling features and contextual features, and map each token INLINEFORM0 ( INLINEFORM1 ) to a feature vector INLINEFORM2 , where INLINEFORM3 is the spelling feature vector and INLINEFORM4 is the contextual feature vector. Note that the values of features are then jointly learned during the process of training. In Section SECREF3 , we will explain the features in more detail. As shown in Fig. FIGREF2 , the vector INLINEFORM0 ( INLINEFORM1 ) is a concatenation of the INLINEFORM2 , INLINEFORM3 and INLINEFORM4 . Each vector INLINEFORM5 is then given to a feed-forward neural network with one hidden layer, which outputs the corresponding probability vector INLINEFORM6 .

Spelling Features	Since the IOCs tend to follow fixed patterns, we predefined several regular expressions and spelling rules to identify IOCs. For example, to identify a URL, we defined a regular expression INLINEFORM0 and set the value of the URL feature to 1 when the input token matches the regular expression. However, such expressions and spelling rules could introduce false positives, i.e., tokens that have the same spelling patterns as IOCs but are not malicious. In this work, we further introduce the contextual features as described next.

Results	As shown in TABLE TABREF24 , we report the micro average of precision, recall and F1-score for all 11 types of labels for a baseline as well as the proposed model. As the baseline, we simply judge the input token as IOCs on the basis of the spelling features described in  . As presented in TABLE TABREF24 , the score obtained by the proposed model is clearly higher than the baseline. Here, as described in Section SECREF14 , the sizes of window and lower bounds of frequency for selecting contextual keywords are tuned as 4 and 7 throughout the evaluation of English dataset, and tuned as 3 and 4 throughout the evaluation of Chinese dataset. The number of extracted contextual keywords from the English dataset is 1,328, and from the Chinese dataset is 331. Furthermore, we quantitatively compare our study with other typical works of sequence labelling, i.e., the work of Huang et al.  , the work of Lample et al.  and the work of Rei et al.  . Huang et al.  proposed a bidirectional LSTM model with a CRF layer, including hand-crafted features specialized for the task of sequence labelling.

proposed using a sequence labelling model based on the bidirectional long short-term memory (LSTM)  for the task of named entity recognition (NER). Chiu et al.  and Lample et al.  proposed integrating LSTM encoders with character embedding and the neural sequence labelling model to achieve a remarkable performance on the task of NER as well as part-of-speech (POS) tagging. Besides, Dernoncourt et al.  and Jiang et al.  proposed applying the neural sequence labelling model to the task of de-identification of medical records. Among the previous studies of the neural sequence labelling task, Zhou el al.  firstly propose using an end-to-end neural sequence labelling model to fully automate the process of IOCs identification. Their model is on the basis of an artificial neural networks (ANN) with bidirectional LSTM and CRF. However, their newly introduced spelling features bring a more extraction of false positives, i.e., tokens that are similar to IOCs but not malicious.

A number of systems are proposed to help discover and gather malicious information and IOCs from various types of data sources  ,  ,  ,  ,  ,  . However, most of those systems consist of several components that identify IOCs by using human-crafted features that heavily rely on specific language knowledge such as dependency structure, and they often have to be pre-defined by experts in the field of the cybersecurity. Furthermore, they need a large amount of annotated data used as the training data to train an IOC classifier. Those training data are frequently difficult to be crowed-sourced, because non-experts can hardly distinguish IOCs from those non-malicious IPs or URLs. Thus, it is a time-consuming and laborious task to construct such systems for different languages. In this work, we consider the task of collecting IOCs from cybersecurity articles as a task of sequence labelling of natural language processing (NLP). By applying a sequence labelling model, each token in an unstructured input text is assigned with a label, and tokens assigned with IOC labels are then collected as IOCs. Recently, sequence labelling models have been utilized in many NLP tasks. Huang et al.

Datasets	For English dataset, we crawl 687 cybersecurity articles from a collection of advanced persistent threats (APT) reports which are published from 2008 to 2018. All of these cybersecurity articles are used to train the English word embedding. Afterwards, we randomly select 370 articles, and manually annotate the IOCs contained in the articles. Among the selected articles, we randomly select 70 articles as the validation set and 70 articles as the test set; the remaining articles are used for training. For Chinese dataset, we crawl 5,427 cybersecurity articles online from 35 cybersecurity blogs which are published from 2001 to 2018. All of these cybersecurity articles are used to train the Chinese word embedding. Afterwards, we randomly select 607 articles, and manually annotate the IOCs contained in the articles. Among the selected articles, we randomly select 122 articles as the validation set and 122 articles as the test set; the remaining articles are used for training. TABLE TABREF20 shows statistics of the datasets. The output labels are annotated with the BIO (which stands for “Begin”, “Inside” and “Outside”) scheme.

TABLE TABREF26 compares several examples of correct IOC extraction produced by the proposed model with one by the work of Lample et al.  . In the first example, the model of Lample et al.  fails to identify the malicious URL “http://www7.chrome-up.date/0m5EE”, because the token only appears in the test set and consists of several parts that are uncommon for URLs, such as “www7” and “date”, and thus both the token embedding and the character embedding lack proper information to represent the token as a malicious URL. The proposed model correctly identifies the URL, where the token is defined as a URL by spelling features and is then identified as a malicious URL by the use of the context information. In the second example, the model of Lample et al.  fails to identify token “cr.sh” of the input Chinese text as a malicious file name, while the token is assigned with a correct label by the proposed model. It is mainly because that the token “cr.sh” is defined as a token of file information by spelling features and tends to co-occur with words, “”(download) and “”(mining software).

When using the Chinese test set, the proposed model trained with the English training set and the Chinese training set obtained a obviously higher F1-scores than the model trained with only the Chinese training set for almost all the types of labels. It is interesting to find that types of labels “e-mail address”, “attack method”, “attacker”, which lack of instances in Chinese training set, show the biggest improvement by using the model trained with the bilingual training set.

Introduction	Indicators of Compromise (IOCs) are forensic artifacts that are used as signs when a system has been compromised by an attacker or infected with a particular piece of malware. To be specific, IOCs are composed of some combinations of virus signatures, IPs, URLs or domain names of botnets, MD5 hashes of attack files, etc. They are frequently described in cybersecurity articles, many of which are written in unstructured text, describing attack tactics, technique and procedures. For example, a snippet from a cybersecurity article is shown in Fig. FIGREF1 . From the text , token “INST.exe” is the name of an executable file of a malicious software, and the file “ntdll.exe” downloaded by “INST.exe” is a malicious file as well. Obviously, these kinds of IOCs can be then utilized for early detection of future attack attempts by using intrusion detection systems and antivirus software, and thus, they exert an important role in the field of cybersecurity. However, with the rapid evolvement of cyber threats, the IOC data are produced at a high volume and velocity every day, which makes it increasingly hard for human to gather and manage them.

Usage of Features	The feature vector for an input token is the concatenation of the token spelling feature vector and the contextaul feature vector. Here, to elucidate the best usage of the feature vector, we evaluate the feature vector by concatenating it at different locations in the proposed model, i.e., the input of the token LSTM layer ( INLINEFORM0 ), the hidden state of the token LSTM ( INLINEFORM1 ), and the output of token LSTM ( INLINEFORM2 ). Among them, to concatenate the feature vector with the LSTM hidden state vector and the sentence vector of attention in the token LSTM layer, as shown in Section SECREF4 , achieved the best performance. We speculate that the features played an important role in the task of IOCs identification and feature vectors near the output layer were able to improve the performance more significantly than those at other locations.

Features	We extract a vector of features for each tokens of input sequences. In this section, we present each feature category in detail.

, which is defined as follows: DISPLAYFORM0  That is to say, we first compute the INLINEFORM0 as a hidden representation of the hidden states of Bi-LSTM INLINEFORM1 for INLINEFORM2 input token, where INLINEFORM3 is obtained by concatenating the INLINEFORM4 hidden states of forward and backward LSTM, i.e., INLINEFORM5 . Then, we measure the importance of the INLINEFORM6 token with a trainable vector INLINEFORM7 and get a normalized importance weight INLINEFORM8 through a softmax function. After that, the sentence vector INLINEFORM9 is computed as a weighted sum of INLINEFORM10 ( INLINEFORM11 ). Here, weight matrix INLINEFORM12 , bias INLINEFORM13 and vector INLINEFORM14 are randomly initialized and jointly learned during the training process. Note that each input sentence merely has one sentence vector INLINEFORM15 as its weighted representation, and INLINEFORM16 is then used as a part of the INLINEFORM17 output of attention-based Bi-LSTM module, where INLINEFORM18 ( INLINEFORM19 ).

For multi-head self-attention module, we employ a stack of 6 multi-head self attention layer, each of which has 4 head and dimension of each head is set to 64. (b) All of the ANN’s parameters are initialized with a uniform distribution ranging from -1 to 1. (c) We train our model with a fixed learning rate of 0.005. The minimum number of epochs for training is set as 30. After the first 30 epochs had been trained, we compute the average F1-score of the validation set by the use of the currently produced model after every epoch had been trained, and stop the training process when the average F1-score of validation set fails to increase during the last ten epochs. We train our model for, if we do not early stop the training process, 100 epochs as the maximum number. (d) We rescale the normalized gradient to ensure that its norm does not exceed 5. (e) The dropout probability is set to 0.5.

TABLE TABREF31 shows that the proposed model trained with the English training set and Chinese training set achieves a small improvement of F1-score on English test set when compared with the model trained with only English training set, and a great improvement of F1-score on Chinese test set when compared with the model trained with only Chinese training set. We compare scores of each label when the proposed model is trained with different training sets in TABLE TABREF32 . When using the English test set, the F1-scores of labels “attack method”, “attack target” and “malware” by the model trained with the English training set and Chinese training set are lower than those scores by the model trained with only the English training set. It is mainly because that tokens of these labels can be written in different languages, which harms the model trained with the bilingual training data set. In contrast, benefiting from the extension of training set, for types of labels that are often written in English, e.g., “domain ”, “file imformation”, “IPv4” and “vlunerability”, the proposed model trained with the English training set and the Chinese training set achieves higher scores than the model trained with only the English training set.

Lample et al.  described a model where the character-level representation was concatenated with word embedding and Rei et al.  improved the model by introducing an attention mechanism to the character-level representations. We train these models by employing the same training set and training parameters as the proposed model. As shown in TABLE TABREF24 , the proposed model obtains the highest precision, recall and F1-score than other models in the task of IOCs extraction. Compared with the second-best model of Lample et al.  , the performance gain of the proposed model on the English dataset is approximately 10.1% of precision and 10.0% of recall. The performance gain of the proposed model on the Chinese dataset is approximately 4.2% of precision and 9.0% of recall. We also quantitatively compare our study with the work of Zhou et al.  , which proposed a bidirectional LSTM model with a CRF layer, including hand-crafted spelling features for the task of IOC identification. As shown in TABLE TABREF24 , the proposed model obtains a slightly higher F1-score on the English dataset and significantly higher F1-score on the Chinese dataset.

Sequence Representation Layer	The Sequence Representation Layer takes the sequence of embeddings INLINEFORM0 ( INLINEFORM1 ) as input, and outputs a sequence INLINEFORM2 , where the INLINEFORM3 element of INLINEFORM4 represents the probability that the INLINEFORM5 token has the label INLINEFORM6 . Different from the previous work of sequence labelling in news articles or patient notes  ,  , sentences from a cybersecurity report often contain a large number of tokens as well as lists of IOCs with little context, making it much more difficult for LSTM to encode the input sentence correctly. Therefore, instead of the token LSTM layer in  , we propose sequence representation layer that consists of 3 modules, i.e., attention-based Bi-LSTM module, multi-head self-attention module and token feature module. Considering that tokens cannot contribute equally to the representation of the input sequence, we introduce attention mechanism to Bi-LSTM to extract such tokens that are crucial to the meaning of the sentence. Then, we aggregate the representation of those informative words to form the vector of the input sequence. The attention mechanism is similar to the one proposed by Yang et al.

Training Details	For pre-trained token embedding, we apply word2vec  to all crawled 687 English APT reports and 5,427 Chinese cybersecurity articles described in Section SECREF21 respectively. The word2vec models are trained with a window size of 8, a minimum vocabulary count of 1, and 15 iterations. The negative sampling number of word2vec is set to 8 and the model type is skip-gram. The dimension of the output token embedding is set to 100. The ANN model is trained with the stochastic gradient descent to update all parameters, i.e., token embedding, character embedding, parameters of Bi-LSTM, weights of sentence attention, weights of multi-head self-attention, token features, and transition probabilities of CRF layers at each gradient step. For regularization, the dropout is applied to the output of each sub layer of the ANN model. Further training details are given below: (a) For attention-based Bi-LSTM module, dimensions of character embedding, hidden states of character-based token embedding LSTM, hidden states of Bi-LSTM, and sentence attention are set to 25, 25, 100 and 100, respectively.

Token Embedding Layer	The token embedding layer takes a token as input and outputs its vector representation. As shown in Fig. FIGREF2 , given an input sequence of tokens INLINEFORM0 , the output vector INLINEFORM1 ( INLINEFORM2 ) of each token INLINEFORM3 results from the concatenation of two different types of embeddings: token embedding INLINEFORM4 and the character-based token embeddings INLINEFORM5 , INLINEFORM6 that come from the output of a character-level bi-LSTM encoder.

Motivated by the successful application of self-attention in many NLP tasks  ,  , we add a multi-head self-attention module to enhance the embedding of each word with the information of other words in a text adaptively. By means of this, the local text regions where convolution performs carry the global information of text. Following the encoder part of Vaswani et al.  , multi-head self-attention module is composed of a stack of several identical layers, each of which consists of a multi-head self-attention mechanism and two convolutions with kernel size 1. Given the sequence of embeddings INLINEFORM0 as input, and the output is defined as follows: DISPLAYFORM0  where, INLINEFORM0 , INLINEFORM1 , INLINEFORM2 are parameter matrices for the projections of queries INLINEFORM3 , keys INLINEFORM4 and values INLINEFORM5 in the INLINEFORM6 head, respectively. Here, INLINEFORM7 , INLINEFORM8 and INLINEFORM9 are set as the input sequence INLINEFORM10 ( INLINEFORM11 ). The INLINEFORM12 is then given to the two convolutions and the output of multi-head self-attention INLINEFORM13 ( INLINEFORM14 ) is obtained.

CRF Layer	We also introduce a CRF layer to output the most likely sequence of predicted labels. The score of a label sequence INLINEFORM0 is defined as the sum of the probabilities of unigram labels and the bigram label transition probabilities: DISPLAYFORM0  where INLINEFORM0 is a matrix that contains the transition probabilities of two subsequent labels. Vector INLINEFORM1 is the output of the token LSTM layer, and INLINEFORM2 is the probability of label INLINEFORM3 in INLINEFORM4 . INLINEFORM5 is the probability that a token with label INLINEFORM6 is followed by a token with the label INLINEFORM7 . Subsequently, these scores are turned into probabilities of the label sequence by taking a softmax function over all possible label sequences.

Model	Fig. FIGREF2 shows the 3 components (layers) of the proposed neural network architecture.

Question: What contextual features are used?
Answer: The words that can indicate the characteristics of the neighbor words as contextual keywords and generate it from the automatically extracted contextual keywords.