Venue: NAACL 2019

# Analyzing the Perceived Severity of Cybersecurity Threats Reported on Social Media

Shi Zong\({}^{1}\), Alan Ritter\({}^{1}\), Graham Mueller\({}^{2}\) and Evan Wright\({}^{3}\)

\({}^{1}\)The Ohio State University, OH, USA

\({}^{2}\)Leidos Inc., VA, USA

\({}^{3}\)FireEye LLC, CA, USA

{zong.56, ritter.1492}@osu.edu muellerwg@leidos.com evan.wright@fireeye.com

###### Abstract

Breaking cybersecurity events are shared across a range of websites, including security blogs (FireEye, Kaspersky, etc.), in addition to social media platforms such as Facebook and Twitter. In this paper, we investigate methods to analyze the severity of cybersecurity threats based on the language that is used to describe them online. A corpus of 6,000 tweets describing software vulnerabilities is annotated with authors' opinions toward their severity. We show that our corpus supports the development of automatic classifiers with high precision for this task. Furthermore, we demonstrate the value of analyzing users' opinions about the severity of threats reported online as an early indicator of important software vulnerabilities. We present a simple, yet effective method for linking software vulnerabilities reported in tweets to Common Vulnerabilities and Exposures (CVES) in the National Vulnerability Database (NVD). Using our predicted severity scores, we show that it is possible to achieve a Precision@50 of 0.86 when forecasting high severity vulnerabilities, significantly outperforming a baseline that is based on tweet volume. Finally we show how reports of severe vulnerabilities online are predictive of real-world exploits.1

Footnote 1: Our code and data are available at [https://github.com/vlexeng/cybersecurity_threats_werty_analysis](https://github.com/vlexeng/cybersecurity_threats_werty_analysis).

+
Footnote †: leftmargin=*] **[https://mvd.nist.gov/](https://mvd.nist.gov/)**

## 1 Introduction

Software vulnerabilities are flaws in computer systems that leave users open to attack; vulnerabilities are generally unknown at the time a piece of software is first published, but are gradually identified over time. As new vulnerabilities are discovered and verified they are assigned CVE numbers (unique identifiers), and entered into the National Vulnerability Database (NVD).2 To help prioritize response efforts, vulnerabilities in the NVD are assigned severity scores using the Common Vulnerability and Scoring System (CVSS). As the rate of discovered vulnerabilities has increased in recent years,3 the need for efficient identification and prioritization has become more crucial. However, it is well known that a large time delay exists between the time a vulnerability is first publicly disclosed to when it is published in the NVD; a recent study found that the median delay between the time a vulnerability is first reported online and the time it is published in the NVD is seven days; also, 75% of threats are first disclosed online giving attackers time to exploit the vulnerability.4

Footnote 2: [https://mvd.nist.gov/](https://mvd.nist.gov/)

In this paper we present the first study of whether natural language processing techniques can be used to analyze users' opinions about the severity of software vulnerabilities reported online. We present a corpus of 6,000 tweets annotated with opinions toward threat severity, and empirically demonstrate that this dataset supports automatic classification. Furthermore, we propose a simple, yet effective method for linking software vulnerabilities reported on Twitter to entries in the NVD, using CVEs found in linked URLs. We then use our threat severity analyzer to conduct a large-scale study to validate the accuracy of users' opinions online against experts' severity

Figure 1: Example tweet discussing the dirty copy-on-write (COW) security vulnerability in the Linux kernel.



ratings (CVSS scores) found in the NVD. Finally, we show that our approach can provide an early indication of vulnerabilities that result in real exploits in the wild as measured by the existence of Symantec virus signatures associated with CVEs; we also show how our approach can be used to retrospectively identify Twitter accounts that provide reliable warnings about severe vulnerabilities.

Recently there has been increasing interest in developing NLP tools to identify cybersecurity events reported online, including denial of service attacks, data breaches and more [14, 23]. Our proposed approach in this paper builds on this line of work by evaluating users _opinions_ toward the severity of cybersecurity threats.

Prior work has also explored forecasting software vulnerabilities that will be exploited in the wild [5]. Features included structured data sources (e.g., NVD), in addition to the volume of tweets mentioning a list of 31 keywords. Rather than relying on a fixed set of keywords, we analyze message content to determine whether the author believes a vulnerability is severe. As discussed by Sabottke et al. [20], methods that rely on tracking keywords and message volume are vulnerable to adversarial attacks from Twitter bots or sockpuppet accounts [6]. In contrast, our method is somewhat less prone to such attacks; by extracting users' opinions expressed in individual tweets, we can track the provenance of information associated with our forecasts for display to an analyst, who can then determine whether or not they trust the source of information.

## 2 Analyzing Users' Opinions Toward the Severity of Cybersecurity Threats

Given a tweet \(t\) and named entity \(e\), our goal is to predict whether or not there is a serious cybersecurity threat towards the entity based on context. For example, given the context in Figure 2, we aim at predicting the severity level towards _adobe flash player_. We define an author's perceived severity toward a threat using three criteria: (1) does the author believe that their followers should be worried about the threat? (2) is the vulnerability easily exploitable? and (3) could the threat affect a large number of users? If one or more of these criteria are met, then we consider the threat to be severe.

### Data Collection

To collect tweets describing cybersecurity events for annotation, we tracked the keywords "ddoS" and "vulnerability" from Dec 2017 to July 2018 using the Twitter API. We then used the Twitter tagging tool described by Ritter et. al. (2011) to extract named entities,5 retaining tweets that contain at least one named entity. To cover as many linguistic variations as possible, we used Jaccard similarity with a threshold of 0.7 to identify and remove duplicated tweets with same date.6

Footnote 5: [https://github.com/arthe/whiten.rjb](https://github.com/arthe/whiten.rjb)

Footnote 6: We sampled a dataset of 6,000 tweets to annotate.

### Mechanical Turk Annotation

We paid crowd workers on Amazon Mechanical Turk to annotate our dataset. The annotation was performed in two phases; during the first phase, we asked workers to determine whether or not the tweet describes a cybersecurity threat toward a target entity, in the second phase the task is to determine whether the author of the tweet believes the threat is severe; only tweets that were judged to express a threat were annotated in the second phase. Each HIT contained 10 tweets to be annotated; workers were paid $0.20 per HIT. In pilot studies we tried combining these two annotations into a single task, but found low inter-rater agreement, especially for the threat severity judgments, motivating the need for separation of the annotation procedure into two tasks.

Figure 2 shows a portion of the annotation interface presented to workers during the second phase of annotation. Details of each phase are described below, and summarized in Table 1.

**Threat existence annotation:** Not all tweets in our dataset describe cybersecurity threats, for example many tweets discuss different senses of the

Figure 2: A portion of the annotation interface shown to MTurk workers during the threat severity annotation.

 

word "vulnerability" (e.g., "It's OK to show vulnerability"). During the first phase of our annotation process, workers judged whether or not there appears to be a cybersecurity threat towards the target entity based on the content of the corresponding tweet. We provide workers with 3 options: the tweet indicates (a) a cybersecurity threat towards given entity, (b) a threat, but not towards the target entity, or (c) no cybersecurity threat. Each tweet is annotated by 5 workers.

**Threat severity annotation:** In the second phase, we collect all tweets judged to contain threats by more than 3 workers in the first phase and annotated them for severity. 1,966 tweets were selected out of 6,000.7 For each tweet we provided workers with 3 options: the tweet contains (a) a severe, (b) a moderate or (c) no threat toward the target entity. During our pilot study, we found this to be a more challenging annotation task, therefore we increased the number of annotators per tweet to 10 workers, which we found to improve agreement with our expert judgments.

Footnote 7: We further deduplicate pairs of tweets where the longest common subsequence covers the majority of the text contents. During deduplication all hashtags and URLs were removed and digits were replaced with 0.

**Inter-annotator agreement:** During both phases, we monitored the quality of workers' annotations using their agreement with each other. We calculated the annotation agreement of each worker against the majority vote of other workers. We manually removed data from workers who have an agreement less than 0.5, filling in missing annotations with new workers. We also manually removed data from workers who answered either uniformly or randomly for all HITs.

**Agreement with expert judgments:** To validate the quality of our annotated corpus we compared the workers' aggregated annotations against our own expert annotations. We independently annotated 150 randomly sampled tweets, 61 tweets of which are marked as containing severe or moderate threats. For threat existence annotation, we observe a 0.66 value of Cohen's \(\kappa\)[2] between the expert judgements and majority vote of 5 crowd workers. Although our threat severity annotation task may require some cybersecurity knowledge for accurate judgment, we still achieve 0.52 Cohen \(\kappa\) agreement by comparing majority vote from 10 workers with expert annotations.

### Analyzing Perceived Threat Severity

Using the annotated corpus described in Section 2.2, we now develop classifiers that detect threats reported online and analyze users' opinions toward their severity. Specifically, given a named entity and tweet, \(\langle e,t\rangle\), our goal is to estimate the probability the tweet describes a cybersecurity threat towards the entity, \(p_{\text{threat}}(y|\langle e,t\rangle)\) and also the probability that the threat is severe, \(p_{\text{severe}}(y|\langle e,t\rangle)\). In this section, we describe the details of these classifiers and evaluate their performance.

We experimented with two baselines to detect reports of cyberthreats and analyze opinions about their severity: logistic regression using bag-of-ngram features, and 1D convolutional neural networks. In the sections below we describe the input representations and details of these two models.

**Logistic regression:** We use logistic regression as our first baseline model for both classifiers. Input representations are bag-of-ngram features extracted from the entire tweet content. Example features are presented in Table 4. We use context windows of size 2, 3 and 4 to extract features. We map extracted \(n\)-grams that occur only once to a \(\langle\text{UNK}\rangle\) token. In all our experiments, we replace named entities with a special token \(\langle\text{TARGET}\rangle;\) this helps prevent our models from biasing towards specific entities that appear in our training corpus. All digits are replaced with 0.

\begin{table}
\begin{tabular}{c|l|c|c|c|c|c} \hline \multirow{2}{*}{Annoo. Tweets Total} & \multicolumn{3}{c|}{1st Annotation (5 Workers per tweet)} & \multicolumn{3}{c}{2nd Annotation (10 workers per tweet)} \\ \cline{2-8}  & Label & \# Tweets & \% & Label & \# Tweets & \% \\ \hline \multirow{3}{*}{6,000} & \multirow{3}{*}{With threat} & 2,543 & \multirow{3}{*}{42.4} & Severe threat & \multirow{3}{*}{506} & 25.7 \\  & & (1,966 for 2nd anno.) & & Moderate threat & 1,460 & 74.3 \\ \cline{1-1} \cline{2-8}  & Without threat & 3,457 & 57.6 & & / \\ \hline \end{tabular}
\end{table}
Table 1: Number of annotated tweets with break-down percentages to each category. In 1st annotation, a tweet contains a threat if more than 3 workers vote for it. In 2nd annotation, a threat is severe if more than 6 workers agree on it. Number of workers cut-offs are determined by comparing to our golden annotations in pilot studies.

 



**Convolutional neural networks:** We also experimented with 1D convolutional neural networks (Collobert et al., 2011; Kim, 2014). Given a tweet, the model first applies convolutional operations on input sequences with various filters of different sizes. The intermediate representations for each filter are aggregated using max-pooling over time, followed by a fully connected layer. We choose convolution kernel sizes to be 3, 4 and 5-grams with 100 filters for each. We minimize cross-entropy loss using Adam (Kingma and Ba, 2015); the learning rate is set to 0.001 with a batch size of 1 and 5 epochs.

**Word embeddings:** We train our own cybersecurity domain word embeddings based on GloVe (Pennington et al., 2014), as 39.7% of our tokens are treated as OOV words in GloVe pre-trained Twitter embeddings. We used a corpus of 609,470 cybersecurity-related tweets (described in Section 2.1) as our training corpus. The dimension of word embeddings is 50. Table 2 shows nearest neighbors for some sampled cybersecurity terms based on the learned embeddings.

During network training, we initialize word embedding layer with our own embeddings. We initialize tokens not in our trained embeddings by randomized vectors with uniform distribution from -0.01 to 0.01. We fine-tune the word embedding layer during training.

#### 2.3.1 Experimental Setup

For threat existence classification, we randomly split our dataset of 6,000 tweets into a training set of 4,000 tweets, a development set of 1,000 tweets, and test set of 1,000 tweets. For the threat severity classifier, we only used data from 2nd phase of annotation. This dataset consists of 1,966 tweets that were judged by the mechanical turk workers to describe a cybersecurity threat towards the target entity. We randomly split this dataset into a training set of 1,200 tweets, a development set of 300 tweets, and a test set of 466 tweets. We collapsed the three annotated labels into two categories based on whether or not the author expresses an opinion that the threat towards the target entity is severe.

#### 2.3.2 Results

**Threat existence classifier:** The logistic regression baseline has good performance at identifying threats, which we found to be a relatively easy task; area under the precision-recall curve (AUC) on the development and test set presented in Table 5. This enables accurate detection of trending threats online by tracking cybersecurity keywords using the Twitter streaming API, following an approach that is similar to prior work on entity-based Twitter event detection (Ritter et al., 2012; Zhou et al., 2014; Wei et al., 2015). Table 3 presents an example of threats detected using this procedure on Nov. 22, 2018.8

Footnote 8: A live demo is available at: [http://kitl.cse.ohio-state.edu/s123events/threat](http://kitl.cse.ohio-state.edu/s123events/threat)

**Threat severity classifier:** Figure 3 shows precision recall curves for the threat severity classifiers. Logistic regression with bag-of-ngram features provides a strong baseline for this task. Table 4 presents examples of high-weight features from the logistic regression model. These features often intuitively indicate severe threats, e.g. "critical vulnerability", "a massive", "million", etc. Without much hyperparameter tuning on the development set, the convolutional neural network consistently achieves higher precision at the same level of recall as compared to logistic regression. We summarize the performance of our threat existence and severity classifiers in Table 5.

## 3 Forecasting Severe Cybersecurity Threats

In Section 2 we presented methods that can accurately detect threats reported online and analyze users' opinions about their severity. We now explore the effectiveness of this model for forecasting. Specifically, we aim to answer the following questions: **(1)** To what extent do users' opinions about threat severity expressed online align with expert judgments? **(2)** Can these opinions provide an early indicator to help prioritize threats based

\begin{table}
\begin{tabular}{p{85.4pt}|p{113.8pt}} \hline Token & Nearest Neighbors \\ \hline \#ddos & attacks, ddos, datacenter-insider, attack, \#cyberattack \\ \hline \multirow{2}{*}{\#hackers} & hackers, sec.cyber, \#blackberry200, \#malware, \#thickae \\ \hline threats & defenses, cyberrisk, \#cybersecurity, threat, \#iot-based \\ \hline vulnerability & risk, ..., \#vulnerability, strength, critical \\ \hline \end{tabular}
\end{table}
Table 2: Nearest neighbors to some cybersecurity related tokens in our trained word embeddings. Embeddings are trained by using GloVe. Similar tokens are sorted by cosine similarity scores.



on their severity?

**A large corpus of users' opinions:** We follow the same procedure described in Section 2.1 to prepare another dataset for a large-scale evaluation. For this purpose, we collected data from Jan 2016 to Nov 2017; this ensures no tweets overlap with those that were annotated in Section 2.2. We collect all English tweets that explicitly contain the keyword "vulnerability" within this time period, which results in a total number of 976,180 tweets. 377,468 tweets remain after removing tweets without named entities.

**National Vulnerability Database (NVD):** NVD is the U.S. government database of software vulnerabilities. Started in 2000, NVD covers over 100,000 vulnerabilities, assigning a unique CVE number for each threat. These CVES numbers serve as common identifiers. NVD uses the Common Vulnerability Scoring System (CVSS) to measure the severity of threats. CVSS currently has two versions: CVSS v2.0 and CVSS v3.0 standards. CVSS v3.0 is the latest version released in July 2015. We summarize the two standards in Table 6.9

Footnote 9: [https://hvd.nist.gov/win-metrics/cvs](https://hvd.nist.gov/win-metrics/cvs)

**Matching tweets with NVD records:** Evaluating our forecasts of high seve

\begin{table}
\begin{tabular}{l c|c c} \hline \hline Severity & Base Score & Severity & Base Score \\ \hline  & & None & 0.0 \\ Low & 0.0-3.9 & Low & 0.1-3.9 \\ Medium & 4.0-6.9 & Medium & 4.0-6.9 \\ High & 7.0-10.0 & High & 7.0-8.9 \\  & & Critical & 9.0-10.0 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Qualitative severity rankings of vulnerabilities in NVD. (Left) CVSS v2.0 standards and (Right) CVSS v3.0 standards.

Figure 3: Precision/Recall curves showing performances of convolutional model (CNN) and logistic regression model (LR) for threat severity classification task in test set.

\begin{table}
\begin{tabular}{l l|r r} \hline \hline Features & Weight & Features & Weight \\ \hline ddos attack & 1.40 & \(\langle\)TARGET\(\rangle\) , & 0.89 \\ hackers to & 1.11 & take over & 0.87 \\ a massive & 1.07 & 00 countries & 0.85 \\ critical vulnerability & 1.03 & attackers to & 0.84 \\
0 billion & 0.96 & discovered in & 0.82 \\ lets attackers & 0.95 & 000 million & 0.82 \\ \(\langle\)TARGET\(\rangle\) users & 0.91 & : dddos & 0.81 \\ a critical & 0.91 & abuse and & 0.81 \\ of a & 0.89 & , ddos & 0.81 \\ many \(\langle\)TARGET\(\rangle\) & 0.89 & a severe & 0.79 \\ \hline \hline \end{tabular}
\end{table}
Table 4: High-weight \(n\)-gram features from logistic regression model for threat severity classification task.

\begin{table}
\begin{tabular}{l|l|r r} \hline \hline Nameed Entity & Example Tweet & Existence & Severity \\ \hline apple & RT AsurSec: A kernel vulnerability in Apple devices gives access to remote code & 0.96 & 0.59 \\  & execution - Packt Hub \#infosse \#CyberSecurity https://H.... & \\ google & RT binitamshahi: Unfixed spoofing vulnerability in Google Inbox mobile apps & 0.78 & 0.17 \\  & [https://t.co/TWx7jSi1gc](https://t.co/TWx7jSi1gc) & \\ adobe & RT Anomalid: Adobe released patches for three “important-ranked” severity vulnerabilities, including one vulnerability in Adobe Acrobat and... & 0.76 & 0.32 \\ flash & Vulnerability in Flash player allowing code execution. Patch before Black Friday: & 0.71 & 0.43 \\  & [https://t.co/4idb57od1E](https://t.co/4idb57od1E) \#CyberSecurity \#vulnerability & 0.69 & 0.88 \\ mac & adobe’s flash player for windows, mac and liuws has a critical vulnerability that should be patched as a top priori.. [https://t.co/LLIPATy9vR](https://t.co/LLIPATy9vR) & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Top five threats extracted with highest confidence on Nov. 22, 2018. For each entity we aggregate tweets, and average threat existence scores. The tweet with the maximum threat severity score is shown in each instance.

 

relies on accurately matching tweets describing vulnerabilities to their associated NVD records. To achieve this we present a simple, yet effective method that makes use of content in linked webpages. We find that 82.4% of tweets contain external urls in our dataset.

Our approach to link tweets to CVEs is to search for CVE numbers either in url addresses or in corresponding web pages linked in tweets reporting vulnerabilities.10 We ignore web pages that contain more than one unique CVE to avoid potential ambiguities. Using this approach, within our dataset, 79,383 tweets were linked to 10,565 unique CVEs. In order to stimulate a forecasting scenario, we only consider CVEs where more than two associated tweets were posted at least 5 days ahead of official NVD publication date. In our dataset, 13,942 tweets are finally selected for forecast evaluation, covering 1,409 unique CVE numbers. To evaluate the accuracy of this linking procedure, we randomly sampled 100 matched pairs and manually checked them. We find the precision of our matching procedure to be very high: only 2 mismatches out of 100 are found.

Footnote 10: Readers may be wondering why a CVE number has been generated before it is officially published in the database. This is due to the mechanism of assigning CVEs. Some identified companies have the right to assign CVEs or have already reversed some CVEs. When a threat appears, a CVE number is assigned immediately before any further evaluation. NVD only officially publishes a threat after all evaluations are completed. Therefore, there is a time delay between CVE entry established date and the official publication date.

### Forecasting Models

Now that we have a linking between tweets and CVE numbers, our goal is to produce a sorted list of CVEs with those that are indicated to be severe threats the top. We consider two ranking procedures, detailed below; the first is based on users' opinions toward the severity of a threat, and the second is a baseline that simply uses the volume of tweets describing a specific vulnerability to measure its severity. To simplify the exposition below, we denote each CVE number as CVE\({}_{i}\), and the collection of tweets linked to this CVE number as \(T_{\text{CVE}_{i}}=\{k|\text{tweet $t_{k}$ is mapped to CVE${}_{i}$}\}\).

**Our model:** Our severe threat classifier assigns a severity score \(p_{\text{severity}}(y|\langle e,t\rangle)\) for each tuple of name entity \(e\) and corresponding tweet \(t\). For a specific CVE, we define our severity forecast score to be the maximum severity scores among all tuples from matched tweets \(\langle\cdot,t_{k}\rangle\) (a single tweet may contain more than one name entity):

\[(\text{CVE}_{i})_{\text{forecast~{}score}}=\max_{k\in T_{\text{CVE}_{i}}}p_{ \text{severity}}(y|\langle\cdot,t_{k}\rangle).\]

**Tweet volume baseline:** Intuitively, the number of tweets and retweets can indicate people's concern about a specific event. Specifically, the severity for threat CVE\({}_{i}\) according to the volume model is defined by the cardinality of \(T_{\text{CVE}_{i}}\):

\[(\text{CVE}_{i})_{\text{volume~{}score}}=|T_{\text{CVE}_{i}}|.\]

### Forecasting CVSS Ratings

In our first set of experiments, we compare our forecasted threat severity scores against CVSS ratings from the NVD. We define a threat as being severe if its CVSS score is \(\geq\) 7.0. This cut-off corresponds to qualitative severity ratings provided by CVSS (marked as HIGH or CRITICAL in Table 6).11 We use the newest v3.0 scoring system, which was developed to improve v2.0.12 Large software vendors have announced of the adaptation of the CVSS v3.0 standards, including Cisco, Oracle, SUSE Linux, and RedHat.

Footnote 12: The Forum of Incident Response and Security Teams (FIRST) also provides an example guideline that recommends patching all vulnerabilities with CVSS scores \(\geq\) 7.0. See [https://www.first.org/cvss-based-patch-policy.pdf](https://www.first.org/cvss-based-patch-policy.pdf).

We evaluate our models' performance at identifying severe threats five days ahead of the NVD publication date, within their top \(k\) predictions. Table 7 shows our results. We observe that tweet volume performs better than a random baseline; having a large number of tweets beforehand is a good indicator for high severity, however our approach which analyzes the content of messages discussing software vulnerabilities achieves significantly better performance; 86% of its top 50 forecasts were indeed rated as HIGH or CRITICAL severity in the NVD.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline  & P@10 & P@50 & P@100 & AUC \\ \hline Random & 59.0 & 61.2 & 58.8 & 0.595 \\ Volume model & 70.0 & 68.0 & 70.0 & 0.583 \\ Our model & 100.0 & 86.0 & 78.0 & 0.658 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Model performance of identifying severe threats (CVSS scores \(\geq\) 7.0) with Precision@\(k\) and area under the precision-recall curve (AUC) metrics. For majority random baseline, we average over 10 trails.

 

Table 8 presents top 4 forecast results from our model. We observe that our model can predict accurate severity level even 19 days ahead of the official published date in NVD (Table 8(a), (c)).

### Predicting Real-World Exploits

In addition to comparing our forecasted severity scores against CVSS, as described above, we also explored several alternatives suggested by the security community to evaluate our methods: (1) Symantec's anti-virus (AV) signatures13 and intrusion-protection (IPS) signatures,14 in addition to (2) Exploit Database (EDB).15

Footnote 13: [https://www.symartence.com/security-conteria-z](https://www.symartence.com/security-conteria-z)

Footnote 14: [https://www.symartence.com/security_response/attacksignatures/](https://www.symartence.com/security_response/attacksignatures/)

Footnote 15: [https://www.euplot-db.com/](https://www.euplot-db.com/)

Sabottke et al. (2015) suggested Symantec's AV and IPS signatures are the best available indicator for real exploitable threats in the wild. We follow their method of explicitly querying for CVE numbers from the descriptions of signatures to generate exploited threats ground truth. Exploit Database (EDB) is an archive of public exploits and software vulnerabilities. We query EDB for all threats that have been linked into NVD.16 In total we gathered 134 CVEs verified by Symantec and EDB to be real exploits within the 1,409 CVEs used in our forecasting evaluation.

Footnote 16: [http://oe.nntre.org/data/refts/netmay/isotoro-EXPLOT-DB.html](http://oe.nntre.org/data/refts/netmay/isotoro-EXPLOT-DB.html)

We evaluate the number of exploited threats identified within our top ranked CVEs. Table 9 presents our results. We observe that 7 of top 10 threats from our model were exploited in the wild. We also observe that for the actual CVSS v3.0 scores, only 1 out of the top 10 vulnerabilities was

\begin{table}
\begin{tabular}{l|p{142.3pt} p{142.3pt} p{142.3pt}} \hline  & CVE Num / Name Entity & CVSE Description / Matched Waves & CVSS Scores / Our Severity (@ Days Ahead) \\ \hline \multirow{4}{*}{(a)} & CVE-2016-0728 & The join,session.keying function in security/keys/process.keys.c in the Linux kernel before 4.4.1 mishandles object references in a certain error case, which allows local users to gain privileges or cause a denial of service (integer overflow and use-after-free) via crafted keyed commands. & 7.2 HIGH (v2.0) & 2016-02-08 \\  & Android & Vulnerability in the Linux kernel calls allow attackers to gain access to millions of Android devices: [http://homextechw.com/insider/2016/01/20newly-discovered-security-flav-could-let-hackers-control-66-of-all-android-devices/](http://homextechw.com/insider/2016/01/20newly-discovered-security-flav-could-let-hackers-control-66-of-all-android-devices/)... & 0.98 & 2016-01-20 (+19) \\  & Android & A Service Vulnerability in the Linux Kernel Hits Millions of PCs, Servers and Android Devices info://bit.t/IOWIA4H3 & 0.89 & 2016-01-20 (+19) \\  & Android & Millions of PCs android devices are at risk from a recently discovered critical zero-day vulnerability. [http://goo.glr95ZYZ](http://goo.glr95ZYZ) finotoscope & 0.89 & 2016-01-20 (+19) \\ \hline \multirow{4}{*}{(b)} & CVE-2017-6753 & A Vulnerability in Cisco WebEx browser extensions for Google Chrome and Mozilla Ferte could allow an unauthenticated, remote attacker to execute arbitary code with the privileges of the affected browser on an affected system. & 9.3 HIGH (v2.0) & 2017-07-25 \\  & Irisories & The Hacker News ; Critical RFC Vulnerability Found in Cisco WebEx Extensions, Again - Patch Now’s Infe  & 7.2 HIGH (v2.0) & 2016-11-10 \\  & Linux OS & Serious Dirty COV bug leaves millions of Linux users vulnerable to attack: A vulnerability discovered in the .. httry.tinyurl.com/jaglo28 & 0.97 & 2016-10-22 (+19) \\  & Linux OS & A critical vulnerability has been discovered in all versions of the Linux OS and is being exploited in the wild [http://bit.t/t.nzes31Xe](http://bit.t/t.nzes31Xe) & 0.95 & 2016-10-25 (+16) \\  & Linux LOW & Serious vulnerability found in the Linux COMP way have persisted for a decade. [http://www.bbcc.ohc/news/technology/2016/01/378610?cd=socialflow.twitter](http://www.bbcc.ohc/news/technology/2016/01/378610?cd=socialflow.twitter) & 0.82 & 2016-10-21 (+20) \\  & [http://arstechnica.com/security/2016/10most-serious-linux-privilege-estcalation-bug-even-usider-active-exploit/](http://arstechnica.com/security/2016/10most-serious-linux-privilege-estcalation-bug-even-usider-active-exploit/). & & 0.82 & 2016-10-21 (+20) \\ \hline \multirow{4}{*}{(d)} & CVE-2016-7855 & Use-After-free vulnerability in Apache Flash Player before 23.0.025 On Windows and OS X and before 11.202.643 on Linux allows remote attackers to execute arbitrary code via unspecified vectors, as exploited in the wild in October 2016. & 10.0 HIGH (v2.0) & 2016-11-01 \\  & Flash & TCVM Critical vulnerability found in Flash, being actively exploited. Patch Flash NOW [https://www.gshamathley.com/patelok-hash/Adobe](https://www.gshamathley.com/patelok-hash/Adobe) & 0.97 & 2016-10-27 (+5) \\  & Adobe flash released a Flash Player update to patch a critical vulnerability that malicious actors have been e... [http://bit.t/bit.ly2eaMilitary-Payer](http://bit.t/bit.ly2eaMilitary-Payer) & 0.95 & 2016-10-26 (+6) \\  & Adobe Flash & A critical vulnerability for Adobe Flash Player that allows an attacker to take control of the affected system. [https://helps.adobe.com/security/products/Hash-player/apbs16-36.html](https://helps.adobe.com/security/products/Hash-player/apbs16-36.html) ... & 0.80 & 2016-10-27 (+5) \\ \cline{1-1}  & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} & \multicolumn{1}{p{142.3pt}}{} \\ \end{tabular}
\end{table}
Table 8: Top 4 threats identified by our forecast model. Severity scores are generated by using threat severity classifier in Section 2.3.

 

exploited.

### Identifying Accounts that Post Reliable Warnings

Finally we perform an analysis of the reliability of individual Twitter accounts. We evaluate all accounts with more than 5 tweets exceeding 0.5 confidence score from our severity classifier. Table 10 presents our results. Accounts in our data whose warnings were found to have highest precision when compared against CVSS include "@securityaffairs" and "@EduardKovacs", which are known to post security related information, and both have more than 10k followers.

## 4 Related Work

There is a long history of prior work on analyzing users' opinions online Wiebe et al. (2004), a large body of prior work has focused on sentiment analysis Pang et al. (2002); Rosenthal et al. (2015), e.g., determining whether a message is positive or negative. In this paper we developed annotated corpora and classifiers to analyze users' opinions toward the severity of cybersecurity threats reported online, as far as we are aware this is the first work to explore this direction.

Forecasting real-world exploits is a topic of interest in the security community. For example, Bozorgi et al. (2010) train SVM classifiers to rank the exploitability of threats. Several studies have also predicted CVSS scores from various sources including text descriptions in NVD Han et al. (2017); Bullough et al. (2017).

Prior work has also explored a variety of forecasting methods that incorporate textual evidence Smith (2010), including the use of Twitter message content to forecast influenza rates Paul et al. (2014), predicting the propagation of social media posts based on their content Tan et al. (2014) and forecasting election outcomes O'Connor et al. (2010); Swamy et al. (2017).

## 5 Conclusion

In this paper, we presented the first study of the connections between the severity of cybersecurity threats and language that is used to describe them online. We annotate a corpus of 6,000 tweets describing software vulnerabilities with authors' opinions toward their severity, and demonstrated that our corpus supports the development of automatic classifiers with high precision for this task. Furthermore, we demonstrate the value of analyzing users' opinions about the severity of threats reported online as an early indicator of important software vulnerabilities. We presented a simple, yet effective method for linking software vulnerabilities reported in tweets to Common Vulnerabilities and Exposures (CVEs) in the National Vulnerability Database (NVD). Using our predicted severity scores, we show that it is possible to achieve a Precision@50 of 0.86 when forecasting high severity vulnerabilities, significantly outperforming a baseline that is based on tweet volume. Finally we showed how reports of severe vulnerabilities online are predictive of real-world exploits.

## Acknowledgments

We thank our anonymous reviewers for their valuable feedback. We also thank Tudor Dumitras for helpful discussion on identifying real exploited threats. Funding was provided by the the Office of the Director of National Intelligence (ODNI) and Intelligence Advanced Research Projects Activity (IARPA) via the Air Force Research Laboratory (AFRL) contract number FA8750-16-C0114, in addition to the Defense Advanced Research Projects Agency (DARPA) via the U.S. Army Research Office (ARO) and under Contract Number W911NF-17-C-0095, in addition to an Amazon Research Award and an NVIDIA GPU grant. The

\begin{table}
\begin{tabular}{l|c c} \hline \hline Account Name & \# Corr / \# Fest & Acc. (\%) \\ \hline jburnsconsult & 15 / 15 & 100 \\ securityaffairs & 10 / 10 & 100 \\ EduardKovacs & 6 / 6 & 100 \\ cripperz & 5 / 5 & 100 \\ cipherstorm & 4 / 5 & 80 \\ \hline \hline \end{tabular}
\end{table}
Table 10: List of users with top accuracies on forecasting severe cybersecurity threats.

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline  & \multicolumn{3}{c|}{Top 10} & \multicolumn{3}{c|}{Top 50} & \multicolumn{3}{c}{Top 100} \\  & P & R & P & R & P & R \\ \hline True CVSS & 10.0 & 0.7 & 16.0 & 6.0 & 16.0 & 11.9 \\ Volume model & 60.0 & 4.5 & 22.0 & 8.2 & 19.0 & 14.2 \\ Our model & 70.0 & 5.2 & 28.0 & 10.4 & 21.0 & 15.7 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Model performance against real-world exploited threats identified by Symantec and Exploit-DB. “True CVSS” refers to ranking CVEs based on actual CVSS scores in NVD. This model is only for reference and can not be used in real practice, as we do not know true CVSS scores when forecasting.

 

content of the information in this document does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright notation here on.

## References

* Allodi and Massacci (2012) Luca Allodi and Fabio Massacci. 2012. A preliminary analysis of vulnerability scores for attacks in wild: The ekits and sym datasets. In _Proceedings of the 2012 ACM Workshop on Building Analysis Datasets and Gathering Experience Returns for Security_, BADGERS '12, pages 17-24, New York, NY, USA. ACM.
* Artstein and Poesio (2008) Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. _Computational Linguistics_, 34(4):555-596.
* Bozorgi et al. (2010) Mehran Bozorgi, Lawrence K. Saul, Stefan Savage, and Geoffrey M. Voelker. 2010. Beyond heuristics: Learning to classify vulnerabilities and predict exploits. In _Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, KDD '10, pages 105-114, New York, NY, USA. ACM.
* Bullough et al. (2017) Benjamin L. Bullough, Anna K. Yanchenko, Christopher L. Smith, and Joseph R. Zipkin. 2017. Predicting exploitation of disclosed software vulnerabilities using open-source data. In _Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics_, IWSPA '17, pages 45-53, New York, NY, USA. ACM.
* Chambers et al. (2018) Nathanael Chambers, Ben Fry, and James McMasters. 2018. Detecting denial-of-service attacks from social media text: Applying nlp to computer security. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1626-1635.
* Chang et al. (2016) Ching-Yun Chang, Zhiyang Teng, and Yue Zhang. 2016. Expectation-regulated neural model for event mention extraction. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 400-410.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Han et al. (2017) Zhuobing Han, Xiaohong Li, Zhenchang Xing, Hongtao Liu, and Zhiyong Feng. 2017. Learning to predict severity of software vulnerability using only vulnerability description. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_, pages 125-136.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _International Conference on Learning Representations_.
* O'Connor et al. (2010) Brendan O'Connor, Rammath Balasubramanyan, Bryan R Routledge, and Noah A Smith. 2010. From tweets to polls: Linking text sentiment to public opinion time series. In _Proceedings of the Fourth International AAAI Conference on Weblogs and Social Media_.
* Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In _Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10_. Association for Computational Linguistics.
* Paul et al. (2014) Michael J Paul, Mark Dredze, and David Broniautowski. 2014. Twitter improves influenza forecasting. _PLOS Currents Outbreaks_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Ritter et al. (2011) Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
* Ritter et al. (2012) Alan Ritter, Oren Etzioni, Sam Clark, et al. 2012. Open domain event extraction from twitter. In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM.
* Ritter et al. (2015) Alan Ritter, Evan Wright, William Casey, and Tom Mitchell. 2015. Weakly supervised extraction of computer security events from twitter. In _Proceedings of the 24th International Conference on World Wide Web_, pages 896-905. International World Wide Web Conferences Steering Committee.
* Rosenthal et al. (2015) Sara Rosenthal, Preslav Nakov, Svetlana Kiritchenko, Saif Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. Semeval-2015 task 10: Sentiment analysis in twitter. In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_ 

Carl Sabottke, Octavian Suciu, and Tudor Dumitras. 2015. Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits. In _24th USENIX Security Symposium (USENIX Security 15)_, pages 1041-1056, Washington, D.C. USUENIX Association.
* Smith [Smith2010] Noah A. Smith. 2010. _Text-driven forecasting_.
* Soloriq et al. [2013] Thamar Soloriq, Ragib Hasan, and Mainul Mizan. 2013. A case study of sockpuppet detection in wikipedia. In _Proceedings of the Workshop on Language Analysis in Social Media_.
* Swamy et al. [2017] Sandesh Swamy, Alan Ritter, and Marie-Catherine de Marneffe. 2017. " i have a feeling trump will win.................." Forecasting winners and losers from user predictions on twitter. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
* Tan et al. [2014] Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The effect of wording on message propagation: Topic-and author-controlled natural experiments on twitter. In _ACL_.
* Wei et al. [2015] Wei Wei, Kenneth Joseph, Wei Lo, and Kathleen M Carley. 2015. A bayesian graphical model to discover latent events from twitter. In _ICWSM_.
* Wiebe et al. [2004] Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. _Computational linguistics_.
* Wilson et al. [2005] Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_, pages 347-354, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
* Zhou et al. [2014] Deyu Zhou, Liangyu Chen, and Yulan He. 2014. A simple bayesian modelling approach to event extraction from twitter. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_.

## Appendix A Linking Algorithm

We describe our approach to match tweets with NVD records in full detail in Algorithm 1.

```
1:// Linking
2:for every tweet t do
3:if CVE number in tweet context or in url links then
4: match CVEs to this tweet
5:else
6: query webpage contents to search for CVEs
7:
8: // Check linking results
9: Keep tweets that matched to only one unique CVE to avoid ambiguities
10:
11: // Apply time constraints
12: Select out tweets that are posted at least 5 days ahead of official NVD publication date (at most 365 days17)
```

**Algorithm 1** Linking tweets to NVD records.

## Appendix B Limitations of CVSS and Real-World Exploits Ground Truth

In Section 3.2 - Section 3.3, we compare our forecast results with (1) CVSS ratings, and (2) real exploited threats identified by Symantec signatures and Exploit Database. Each of these sources of ground truth have limitations, which we discuss below.

CVSS ratings are widely used as standard indicators for risk measurement in practice. However, one problem of CVSS ratings is that high severity threats do not necessarily lead to real-world exploits. Allodi and Massacci (2012) show that only a small portion (around 2%) of reported vulnerabilities were found to be exploited in the wild. Furthermore, more than half of the threats in NVD are marked as HIGH or CRITICAL, causing a large burden on vendors to fix.18 We also notice these CVSS scores are closely tied with specific categories of threats. For example, 85.6% of buffer errors are marked as HIGH or CRITICAL, while 72.5% of information leaks were marked as MEDIUM or LOW. All these issues post challenges on how to prioritize real exploitable threats, with the goal of reducing false positives and false negatives simultaneously. Our work provides one such additional source of information for helping to prioritize threats.

Footnote 18: [https://www.sublassedsecurity.com/201705/cvsv3-when-every-](https://www.sublassedsecurity.com/201705/cvsv3-when-every-) vulnerability-appears-to-be-high-priority/

The ground truth we use for real exploited threats is still an incomplete list. For example, Linux kernel vulnerabilities are less likely to appear in Symantec signatures, as Symantec does not have a security product for Linux. Identifying real exploited threats is a difficult task; to the best of our knowledge, there does not exist an easy-to-access list covering all exploited threats currently.

## Appendix C Additional Analysis of Results

In this section, we present further analyses of people's online behaviors when discussing cybersecurity threats on social media.

We find that the real severity of threats is pre 

dictable based on users' opinions online. We observe several repeated patterns in how people describe severe threats. We summarize some of these patterns below:

* describing severity levels (see Appendix C.1), such as "critical", "serious", "highly";
* describing the number of users or devices affected, such as "millions of TARGET) devices", "huge number of";
* potential consequences, such as "allows hackers to", "could allow for remote code execution", "malware";
* alerts or warnings, such as "please be aware", "warning";
* suggesting immediate actions, such as "patch now".

### Usage of Subjective Adjectives

We notice people rely on adjectives for describing the level of severity for threats, rather than numerical scores. These subjective adjectives form our initial impressions on these threats.

We examine subjective adjectives people use for measuring threats. We run POS tagging to extract all tokens marked as JJ, JJP, and JUS. We then rank subjective adjectives in Subjectivity Lexicon (SUB) Wilson et al. (2005) by log-odds ratio of their occurrences in NVD descriptions for HIGH or CRITICAL threats versus MEDIUM or LOW threats. Table 12 presents top ranked subjective adjectives. We observe variants people are using for severe threats, e.g. "serious", "severe", "malicious", etc.

### Temporal Analysis

We collect all CVEs having matched tweets posted at least 1 day ahead of the official NVD publication date, resulting in a set of 3,678 CVEs. Within our dataset, 84.7% of CVEs are reported within 60 days after the first disclosure on social media. We observe a median of 5 days delay in our dataset, whereas some of threats have significant longer delays. For example, CVE-2016-212319 (Overflow Remote Code Execution Vulnerability) first appears at Twitter on Dec. 19, 201620, but is published in NVD on Nov. 1, 2018. It again shows the difficulty of threat evaluation and management.

Footnote 19: [https://nvdr.nist.gov/wiki/detailCVE-2016-2123](https://nvdr.nist.gov/wiki/detailCVE-2016-2123)

Footnote 20: [https://nvidr.com/ytsed/status/#10981102768758784](https://nvidr.com/ytsed/status/#10981102768758784)

### Error Analysis

We evaluate two types of errors with respect to forecasting high severity vulnerabilities: false positive and false negative examples. We observe that some severe threats are difficult to predict based on contents in general, such as Table 11(a). There is no clear clue for estimating the severity level merely on tweet contents.

We present another incorrect example extracted by our forecast system in Table 11(b). We notice tokens like "expose users to attack", "opens up to a raft of problems", etc. This threat does seem to be exploitable and harmful to a lot of users. However, experts mark it as of medium severity. It might be the case that the actual severity level of some threats are overestimated by some accounts.

\begin{table}
\begin{tabular}{l|c l|l c|l} \hline  & CVE Num & Name Entity & Tweet & Our Score & Real Severity \\ \hline (a) & CVE-2017-4984 & EMC VNKNVNNX2 & Intermediate: Yulp: EMC VNKN/VNNX2 OR File CWE-2017-4984 Remote & 0.01 & 10 0 HIGHT (v.20) \\  & OF & Code Execution Vulnerability [http://bit.uiz.w?KQAs](http://bit.uiz.w?KQAs) & 9 8 CFHTICAL (v.3.0) \\ \hline (b) & CVE-2016-1730 & iPhone & A newly discovered vulnerability my capsule iPhone users to attack when & 0.76 & 5.8 MEDIUM (v.20) \\  & iPhone & using a W-Fi hebottoi via @InfluosecurityMag. import kitli/Cov.VO & 5.4 MEDIUM (v.3.0) \\  & & Apple iOS Flaw Enables Attacks via Hospital. The vulnerability opens up & 0.45 & \\  & iPhone uses to a raft of problems, inc.. [http://bit.jnl/ylsdoIDO](http://bit.jnl/ylsdoIDO) & & \\ \hline \end{tabular}
\end{table}
Table 11: Some examples of forecast errors made by our model. (a) False negative examples: there is no clear language clue for demonstrating the severity of threats, experts are needed for threats of this kind. (b) False positive examples: there exist some signals captured by our model for being severe threats, but actual severity might be overestimated.

\begin{table}
\begin{tabular}{l r|l|l l|l l} \hline Adj. & Ratio & Adj. & Ratio & Adj. & Ratio \\ \hline serious & 2.01 & aware & 1.61 & fast & 1.39 \\ pivotal & 1.95 & most & 1.61 & original & 1.39 \\ sure & 1.95 & vivid & 1.61 & able & 1.39 \\ free & 1.95 & accessible & 1.39 & blind & 1.39 \\ active & 1.79 & popular & 1.39 & arbitrary & 1.35 \\ intelligent & 1.79 & deep & 1.39 & high & 1.30 \\ static & 1.79 & black & 1.39 & incomplete & 1.25 \\ critical & 1.67 & top & 1.39 & malicious & 1.20 \\ severe & 1.61 & dangerous & 1.39 & wily & 1.10 \\ great & 1.61 & wild & 1.39 & evil & 1.10 \\ \hline \end{tabular}
\end{table}
Table 12: Top ranked log-odds ratio of subjective adjectives describing severe threats (CVSS scores \(\geq\) 7.0) versus non-severe threats (CVSS scores \(<\) 7.0). Subjective adjectives are identified by using Subjectivity Lexicon (SUB) Wilson et al. (2005).