http://www.diva-portal.org

Postprint
This is the accepted version of a paper presented at 13th International Conference on
Security of Information and Networks, SIN 2020, November 4-6, 2020, virtual, Istanbul,
Turkey.

Citation for the original published paper:
Jiang, Y., Atif, Y. (2020)
An Approach to Discover and Assess Vulnerability Severity Automatically in CyberPhysical Systems
In: Berna Örs, Atilla Elçi (ed.), Proceedings of the 13th International Conference
on Security of Information and Networks: November 4-6, 2020, virtual, Istanbul,
Turkey, 9 New York, NY, USA: Association for Computing Machinery (ACM)
ACM International Conference Proceedings Series (ICPS)
https://doi.org/10.1145/3433174.3433612

N.B. When citing this work, cite the original published paper.

Permanent link to this version:
http://urn.kb.se/resolve?urn=urn:nbn:se:his:diva-19500

© 2020 by the authors. This is the author's version of the work. It is posted here for your personal use. Not for redistribution.
The definitive version was published in 13th International Conference on Security of Information and Networks,
https://doi.org/10.1145/3433174.3433612.

An Approach to Discover and Assess Vulnerability Severity
Automatically in Cyber-Physical Systems
Yuning Jiang

University of Skövde
Skövde, Sweden
yuning.jiang@his.se

ABSTRACT
Current vulnerability scoring mechanisms in complex cyber-physicalsystems (CPSs) face challenges induced by the proliferation of bothcomponent versions and recurring scoring-mechanism versions.
Di�erent data-repository sources like National Vulnerability Database (NVD), vendor websites as well as third party security tool
analysers (e.g. ICS CERT and VulDB) may provide con�icting severity scores. We propose a machine-learning pipeline mechanism tocompute vulnerability severity scores automatically. This method
also discovers score correlations from established sources to infer
and enhance the severity consistency of reported vulnerabilities. To
evaluate our approach, we show through a CPS-based case study
how our proposed scoring system automatically synthesises accurate scores for some vulnerability instances, to support remediation
decision-making processes. In this case study, we also analyse thecharacteristics of CPS vulnerability instances.
ACM Reference Format:
Yuning Jiang and Yacine Atif. 2020. An Approach to Discover and Assess
Vulnerability Severity Automatically in Cyber-Physical Systems. In 13th
International Conference on Security of Information and Networks (SIN 2020),
November 4–7, 2020, Merkez, Turkey. ACM, New York, NY, USA, 8 pages.
https://doi.org/10.1145/3433174.3433612

1

INTRODUCTION

The rapid advancement of information and communication technologies enables seamless integration of new software and hardware, to shift from human-controlled mechanisms to autonomousmechanisms empowered by cyber-physical systems (CPS) that are
currently driving Industry 4.0 evolution. However, vulnerabilities
emerge intermittently in di�erent software, hardware, and �rmware.
These undesirable instances of vulnerability need to be rooted out
from CPS infrastructure and their severity needs to be assessed andranked, to prioritise patches that prevent threat-induced anomalies
or intrusion attempts [7].
Modern security practices promote quantitative analysis-methods
to provide predictive analytics and support patching prioritisation
exercises. A typical approach uses the standard Common Vulnerability Scoring System (CVSS) for vulnerability rating, which requires
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor pro�t or commercial advantage and that copies bear this notice and the full citationon the �rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior speci�c permission and/or afee. Request permissions from permissions@acm.org.
SIN 2020, November 4–7, 2020, Merkez, Turkey© 2020 Association for Computing Machinery.
ACM ISBN 978-1-4503-8751-4/20/09. . . $15.00
https://doi.org/10.1145/3433174.3433612

Yacine Atif

University of Skövde
Skövde, Sweden
yacine.atif@his.se
manual inputs from an expert analyst to evaluate a vulnerability
instance following some given metrics [13][9]. One primary data
source is the Common Vulnerabilities and Exposures (CVE) database, the vulnerability reports from which are further assigned withCVSS scores and other statistical views by National Vulnerability
Database (NVD). Yet, making vulnerability-remediation decisions
based solely on CVE or NVD records can be biased and misleading[4]. A knowledge-base that consolidates multiple sources throughsound classi�cation rules appears to be missing, yet it promisesa suitable level of decision-support. While CVSS mechanism iscommonly used [13][9], inconsistent scores exist for the same vulnerability instance, as vulnerability properties change over timeand across di�erent deployment environments. Other relevant data
sources, such as vendor reports, reviews from third parties, andonline security forums or blogs, can also be leveraged to factor-in
weights over existing CVSS scores [8].
In this paper, a vulnerability-severity scoring system is proposed
to re�ect the severity of a vulnerability instance. Our proposedscoring system enhances compatibility across di�erent CVSS versions, while streamlining vulnerability analysis. In doing so, we
adopt standard CVSS metrics and related scoring-mechanism toquantify the di�culty to exploit vulnerabilities and the magnitude
of resulting impacts from using those exploits maliciously. We also
correlate existing CVSS scores of vulnerability instances provided
by di�erent online cybersecurity data sources, such as NVD, vendor websites or technical reports from third party reviews (e.g.
ICS CERT and VulDB), to consolidate scores in a way that better
describe the actual severity of vulnerability instances. Our proposed
method automates vulnerability assessment and related parameters’
evaluation for CPS based infrastructures [12].
The main contributions of this work can be summarised as follows:• A novel machine-learning based pipeline that streamlines
the process of vulnerability assessment and computes CVSS
severity scores for vulnerability instances. This proposed
methods considers and reconciles inconsistent scores usingmajority voting, to improve the quality of training ground forthe machine-learning models. Our model also allows easy
customisation on selecting a preferable CVSS version, toallow vulnerability assessment while prioritising a consistentand common semantic.
• A case study centred around CPS vulnerability analysis tovalidate the proposed machine-learning based vulnerabilityassessment approach. In this case study, an approach to design a proper query logic for vulnerability-instance retrievalsis also introduced.

SIN 2020, November 4–7, 2020, Merkez, Turkey

The rest of this paper is organised as follows: In Section II, we provide some background and formally state the problem addressedin this paper, followed by Section III which discusses standard
vulnerability-severity metrics and related vulnerability-assessment
processes used in the CVSS mechanism. In Section IV, we reveal
our vulnerability assessment prototype, which correlates existing
CVSS scores against other security-alert indicators, as well as reconciles di�erent CVSS versions using some text-mining techniques
on a corpus of vulnerability reports. In Section V, we evaluate
our vulnerability-discovery and assessment methodology in CPS
contexts through some analysis. In Section VI, we provide someconcluding remarks and discuss some future research directions.

2

BACKGROUND AND PROBLEMS

The continuous evolution of risks in enterprises and the limited
availability of cyber-attack data bring notable quantitative methods
and potential metrics into security-related research agendas. Thise�ort aims to enhance previous risk management frameworks thatare mostly based on expert opinions. Among some initial attempts
made towards combining emerging metrics, the standard mechanism CVSS is widely used to support vulnerability-severity assessment in both academic research and particularly in security-criticalindustrial domains [5][9][8][14]. However, some characteristics ofCVSS mechanism are challenging in practice. More speci�cally,
current CVSS-scoring process is mostly subjective, and incurs timedelays to rate the severity of a vulnerability since its disclosure in
CVE, which widens the risk window for potential zero-day attacks
to occur [12]. Therefore, an automated vulnerability-score inference
system can decrease this risk window. However, several limitations
related to focus dilemmas require further investigation, like how toderive related measurements to determine these metric scales for agiven vulnerability. On the other hand, compatibility issues among
existing CVSS versions which result in di�erent measurements ofstandard metrics was overlooked by previous research. Reported
vulnerability instances are scored using di�erent CVSS versions
and adopted by various organisations [13], resulting in compatibility gaps. For instance, NVD provides CVSS version3 scores only tovulnerability instances reported in and after the year of 2015. In thispaper we contribute a solution to overcome these gaps by providing
a set of scores under di�erent CVSS versions, while allowing users
to select the preferable version.
Correlation studies between multiple cybersecurity data-sources
have been undertaken to combine di�erent perspectives, in order
to connect multifaceted analysis into broader statistical associations. CVE, NVD and CERT are widely used vulnerability-analytics
databases for uniquely identi�ed vulnerability recordings. These
databases are further correlated to data sources like ExploitDB. For
instance, one study from Allodi and Massacci [1] correlates the
NVD to ExploitDB and (Symantec) AttackSignature and ThreatExplorer. Another study from Geer and Roytman [6] also correlates the
NVD database to ExploitDB and (Metasploit) to support penetration
testers. A variety of approaches apply text-mining techniques notin cybersecurity reports but industrial blogs like Twitter [3] and security papers. For example, Zhu and Dumitras apply NLP to extract
malware detection features from research papers automatically [18].
These works highlight that statistical interpretations of CVE/NVD

Yuning Jiang and Yacine Atifdatasets need to be combined with other live security-related datasources, such as product vendors across deployed infrastructures, toraise indicators’ reliability and precision. However, these works areusually in the �eld of software vulnerability analysis. Correlation
studies considering di�erent terminology used in cybersecurity
and CPS domains are limited. In our method, we extracted theentities of vulnerable components and relevant vendor names in
CVE vulnerability reports, and then map retrieved entities with theCommon Platform Enumeration (CPE) as well as vendor websites,
to generate dictionary for CPS components and vendors.
The retrieved information from cybersecurity data-sources support further pattern recognition and trend analysis. Using Arti�cialIntelligence techniques, large amounts of such open-source vulnerability data [14] can be analyzed. More speci�cally, machine-learningtechniques like text-mining are applied to classify disclosed vulnerabilities automatically and to guide predictive analytics of security
gaps, as suggested by Gawron et al., [5] and by Yamamoto et al.,[16] in related previous studies. Nevertheless, using correlated cybersecurity data sources also raise potential inconsistencies, such as
the disparity between scores of the same vulnerability instances [8].
One drawback of previous AI-based CVSS computing approaches
is that they directly adopt the vulnerability reports from CVE andCVSS scores from NVD as training grounds, which may induce bias
to their model. Instead, we correlate vulnerability instances in NVD
to corresponding vendor reports and third-party cybersecurity analysers such as CERT reports, and integrate these information into auni�ed package as our training grounds. To be more speci�c, we
use the vulnerability descriptions in NVD as training input. We
then apply majority voting on inconsistence scores before using
them as training grounds. In doing so, our approach streamlines
the computation of vulnerability severity to address such inconsistencies, in order to optimise security resources allocation and to
shorten the potential risk window.

3

VULNERABILITY SEVERITY METRICSCVSS is developed and maintained by the CVSS Special Interest
Group (CVSS-SIG) and reported in the Forum for Incident Response
and Security Teams (FIRST). The development1 of CVSS has gone
through three amendment stages, and is currently in the stage ofCVSS version 3 (V3). Using CVSS, a vulnerability instance could be
identi�ed by di�erent types of properties measured through inherent metrics and following a range of corresponding measurement
values [10]. The principal properties of an asset vulnerability are
categorised into three groups, namely Base, Temporal, and Environmental properties. In this paper, we focus on the base property only.
The base-property refers to inherent characteristics of a vulnerability that do not change over time or over deployment environments,
which is the focus of this paper. Under CVSS V3, the base-property
is further categorised into a)exploitability, b)scope and c)impact
properties [9].
• Exploitability: The exploitability property is based on thepossibility, di�culty and complexity of exploiting a vulnerable component. The exploitability of a vulnerability could
be represented using four metrics, namely AttackVector (AV),
AttackComplexity (AC), PrivilegesRequired (PR) as well as
1 https://www.�rst.org/cvss/user-guideCyber-Physical Systems Vulnerability Severity

SIN 2020, November 4–7, 2020, Merkez, Turkey

UserInteraction (UI). Attack-path refers to the path throughwhich the vulnerable component could be targeted and thenexploited. Di�erent attack-path values express the di�culty
associated with exploitability. Attack-complexity describes
the di�culty to make use of the vulnerability. The required
privileges is a metric of the exploitability property that assesses the level of authority that a threat-agent needs to obtain to exploit the vulnerable asset. Finally User-interaction
metric of exploitability property di�ers by the needed levelof user’s participation so that the vulnerable component
could be compromised.
• Scope: The scope property of a vulnerability indicates whether
exploiting that vulnerability could impact the vulnerabilityof other components. The metric associated with this property is called ScopeChange (S) and states whether exploitingthat vulnerability on a component’s scope gives access toother components’ scope or not.
• Impact: The impact property of a vulnerability measuresthe consequences resulting from exploiting the vulnerability,
which could cause losses in Con�dentiality (C), Integrity(I) or Availability (A), i.e. (CIA) triad, of the corresponding
component. The values used by CVSS to measure the severityof the impact metric on the vulnerable component are none(N), low- (L) or high- (H) impact.
Ultimately, a vulnerability quanti�cation system needs to delivera CVSS vulnerability score based on some mathematical formulation. Equation 1 maps measurements of Exploitability, Scope andImpact property metrics to the Base category score of a vulnerability. Measurements for each property metric are collected withrespect to component c i in CPS components set C, as illustrated by
Equation 2. Going bottom-up from measurements to score, eachcomponent c i is subject to vulnerability measured by its Base
property-metric measurements: YExploit , YScope and YImpact . The
function f Base of Equation 1 maps these obtained measurements
into a Base category score.
f Base : P Exploit ⇤ PScope ⇤ P Impact ! P Base(1)where vulnerability measurements vector i are collected for component c i by:
Exploitf Exploit : C ! P Exploit , Such as: i

Impactf Impact : C ! P Impact , Such as: i

4

7! f Exploit (c i ) (2)7! f Impact (c i )(3)

DISCOVERING VULNERABILITY SEVERITYIn this section, we present a machine-learning (ML) based method
to discover CVSS scores of reported vulnerability instances forwhich no score is provided. As illustrated in Fig. 1, we �rst adopt
majority voting techniques [15] to deal with inconsistent scores
retrieved from di�erent CVSS scored reports across di�erent repository sources. The reconciled scores are used as training groundfor our ML models, together with vulnerability reports. Then we
streamline score-computation by using a ML pipeline that classi�esthese instances considering various CVSS-metric labels. CVSS metrics from di�erent CVSS versions are stored in a knowledge basewhereby the corresponding metric-set is retrieved through user’s
query. The same goes for measurements and severity scales. In doing so, one can select any CVSS version to compute correspondingscore and vector for vulnerability instances. The word "pipeline"
here suggests a series of steps chained together by integrating ML
cycles, where each cycle involves obtaining the data (both the training instances and the classi�cation measurements into categorised
metric classes), pre-processing it, training/testing it on a machinelearning algorithm and �nally obtaining some output (in the formof a severity-score prediction).

4.1

Majority Voting for Inconsistent Scores

A small percentage of score records in NVD is assumed to haveerrors [13]. Therefore, relying upon NVD scores alone can bringbias in vulnerability assessment. Besides statistical vulnerability
patterns mined from CVE reports, other data sources like vendors
and third-party security analysers (e.g. ICS CERT and VulDB) may
provide di�erent perspectives for vulnerability scoring. In our proposed severity-computation pipeline, we use the score that the
majority data-sources in the pipeline agree on as an estimate of thetrue score [15]. If only two data-sources provide scores, and thesescores are inconsistent, then we take the average of the scores.
Here, we give an example using the vulnerability instance CVE2014-0754 which is assigned a CVSS V2 base-score 10.0 by NVD,
VulDB, as well as ICS CERT with a vector AV:N/AC:L/Au:N/C:C/I:C/A:C.
Yet, a di�erent score 9.3 is assigned by the vendor Schneider Electricwith a vector AV:N/AC:M/Au:N/C:C/I:C/A:C. The inconsistence occurs due to di�erent measurements for Access Complexity of thisinstance, whereby Schneider Electric assigns medium complexity,
while the other three parties assign low complexity. Using our majority voting approach, we choose a �nal score of 10.0 as true score.
Another example is the vulnerability instance CVE-2018-7791 forwhich a CVSS V3 base-score of 9.8 is assigned by NVD and vendor
Schneider Electric with the vector AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H. Yet, ICS CERT assigns this vulnerability with score 7.7 withthe vector AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:L. Similarly, the inconsistence comes as a result of di�erent measurement for attack
complexity. Using our majority voting approach, we choose a �nal
score of 9.8 as the true score.

4.2

Vulnerability Severity Computing

Our proposed machine-learning pipeline supports the classi�cationof cybersecurity data, and �lls-up missing scores in vulnerability
reports using text-mining techniques [14]. We retrieve new vulnerability instances from online cybersecurity databases, together witha list of vulnerability descriptions for these instances. To classify
new vulnerability instances into di�erent CVSS-metric categories,
we build a machine-learning classi�er and train the classi�er basedon some historical data.
Formally, a dataset D with n vulnerability instance where each instance (x i , Yi ) (0 < i  N ) has a vulnerability report x i and a groundtruth vector Yi . Given a set of CVSS metrics m = [m 1 , . . . , m j ,(m 1 ). . . , m M ] (0 < j  M), then Yi = [Yi(m j ), . . . , Yi(m M ), . . . , Yi](m )where Yi j 2 {l 1 , . . . , lk , . . . , l K } (0 < k  K) for a metricm j with K classes. Here we take CVSS Version 3 as an example,

then CVSS metric-set m = [AV , AC, PR, U I, S, C, I, A], and Yi =

SIN 2020, November 4–7, 2020, Merkez, Turkey

Yuning Jiang and Yacine Atif

Figure 1: Vulnerability Severity Computing Pipeline(P R) (U I ) (S ) (C) (I ) (A)
(AV )
, Yi
, Yi , Yi , Yi , Yi ] where Yi
2
(AC)
(P R)
(U I ){N , A, L, P }, Yi
2 {L, H }, Yi
2 {N , L, H }, Yi
2 {N , R},
(S )
(C)
(I )
(A)
Yi 2 {U , C}, Yi 2 {H, L, N }, Yi 2 {H, L, N }, Yi 2 {H, L, N }.
(AV )

[Yi

(AC)

, Yi

, Yi

A severity-score is computed using Algorithm 1. For a classi�cation metric m j with K(K > 2) classes, we reduce this classi�cation metric to several binary classi�cation problems, while eachproblem discriminates between one class from the rest. So with a"one-against-all" method, we have f (x i ) = arg max fk (x i ).
k

Algorithm 1 Automatic Vulnerability Severity Computing
1: procedure S�������C��������(ML, D, m, K). ML is a machine learning model f (), m is a set of CVSS
metrics m j (0 < j  M) where each metric m j has a set of K m jclasses, and D is a dataset with n vulnerability instance where
each instance (x i , Yi ) (0 < i  N ) has a vulnerability report x iand a ground truth vector Yi .
2:
N = |D|, M = |m|3:
for j = 1, . . . , M do
4:
Train(ML j )5:f (x)j = arg max
f K m j (x)jmKj

end for7:
for i = 1, . . . , N do
8:for j = 1, . . . , M do(m )9:
Z i j = f (x I )j10:
end for
(m )
(m )
(m )
11:
Z i = [Z i 1 , . . . , Z i j , . . . , Z i M ]12:
end for13:
Severity Score Si = CV SSCALCU LAT ION (Z i )14: end procedure
6:

Using CV SSCALCU LAT ION (Z i ), CVSS measurement classi�cation labels are further mapped to scaled attributes in order tocompute severity scores. Using CVSS V3.0 as an example, AttackComplexity (AC) with high label refers to a score attribute of 0.44,
while a low label refers to a score attribute of 0.77.

4.3

Evaluation Metrics

For training/testing classi�cations, we compare predicted severity
labels to their original severity label counterparts, and apply theevaluation metrics of Balanced Accuracy (adopted from [2]) and F1score (adopted from [11]) to evaluate the machine-learning model
performance, which takes into account the e�ect of imbalanced
classes. For example, in AccessVector(AV) classes, sample-size in the
Network category are much larger than sample-size in the Physicalcategory. This class imbalance phenomenon is also highlighted inTable 2 later. For binary classi�cations such as UserInteraction (UI)classi�cation, we apply the confusion matrix to compute the valueof classi�cation balanced-accuracy and F1-score. For multi-classclassi�cations such as AccessVector(AV) classi�cation, we use microaverage to calculate the average of per class evaluation, which takes
into consideration the contribution of each class.

4.4

Performance Validation

We retrieved 138 331 vulnerability records with index year values ranging from 2002 to 2019 using NVD (updated till October
27th, 2020), and removed the reports that are marked as REJECT.
We further removed the vulnerability instances that are not scoredunder CVSS V2 mechanisms, and set up a corpus for CVSS V2 severity computing with the remaining 127 907 vulnerability reports.
We correlate these vulnerability instances with authoritative datasources such as ICS-CERT (which provides cybersecurity experts’
insights) as well as vendors’ websites, to reconcile inconsistentscores. These scores are used later as ground truths for ML modeltraining. Similarly, we removed the ones that are not scored under
CVSS V3 mechanisms, and set up a corpus with 58 813 instances
for CVSS V3 severity computation. Note that only a small amountof the vulnerability instances reported in and before the index year
2015 are scored under CVSS V3, which sums up to 4 958 items.
We built a machine-learning pipeline using an existing Python
package pipeline from Scikit-learn library, to automate the machinelearning work�ow like data processing and features extraction.
Such a pipeline structure also allows the transformation of severity scores using di�erent CVSS-version mechanisms (e.g. CVSSV2 and CVSS V3) e�ciently. To do so, we �rst conducted somedata pre-processing like tokenisation and feature extractions on

Cyber-Physical Systems Vulnerability Severity

SIN 2020, November 4–7, 2020, Merkez, Turkey

vulnerability reports from NVD. More speci�cally, CountVectorizer
and TdidfTransforer utilities are used for vectorisation and TF-IDF(referring to Term Frequency-Inverse Document Frequency) value
computation, based on which we created a TF-IDF sparse matrix
using a sequence of word features. Then we adopted train_test_split
utility to divide our data records into a 75% training dataset and a
25% validation dataset, randomly.
Table 1: Evaluation of CVSS Categorisation
CVSS-Metric
V2 AccessVector(AV)V2 AccessComplexity(AC)V2 Authentication(Au)
V2 Con�dentialityImpact(C)V2 IntegrityImpact(I)V2 AvailabilityImpact(A)V3 AttackVector(AV)V3 AttackComplexity(AC)V3 PrivilegesRequired(PR)V3 UserInteraction(UI)
V3 Scope(S)V3 Con�dentialityImpact(C)V3 IntegrityImpact(I)V3 AvailabilityImpact(A)Balanced
Accuracy
80.87%
63.68%
56.34%
81.03%
82.40%
80.12%
75.92%
78.78%
78.79%
93.45%
93.65%
88.36%
90.58%
75.75%Micro
F1-Score
95.76%
83.63%
95.00%
82.98%
84.60%
81.08%
93.68%
95.58%
90.71%
94.13%
97.48%
91.46%
92.02%
93.01%

Machine-learning algorithms are then selected and trained withextracted features. Such classi�ers are employed to categorise newvulnerability instances in order to generate severity patterns. In
our case study, we present the results using LogisticRegression (LR)classi�er. We also applied a 5-fold strati�ed cross-validation on the
training dataset for CVSS categorisation to minimise a possible
over�tting. The prediction performances of our CVSS classi�er
are listed in Table. 1, which look promising compared to similar
CVSS classi�cation works by Gawron et al., [5] and by Yamamoto
et al., [16]. Model performance in Gawron et al. is evaluated using
accuracy though, which can not re�ect the true performance of
unbalanced data classi�cation. Interestingly, classi�cations using
CVSS V3 metrics achieve an overall better performance than thoseusing CVSS V2 metrics, even though the error-rate of CVSS V3 issuperimposed by a larger number of metrics.

5

CYBER-PHYSICAL SYSTEMSVULNERABILITY ASSESSMENT

Vulnerabilities in CPS infrastructure are assessed to enumerate
and rank their severity, in order to prevent threat-induced anomalies or intrusion attempts [7]. Here we present a case study ofseveral prominent CPS components vulnerability analysis. We �rst
query for CPS relevant vulnerabilities from online cybersecurity
data-sources. Then, we compute the CVSS V3 base-scores and corresponding vectors for retrieved vulnerability instances. Finally, we
perform an analysis to explore the statistical patterns of existing
CPS vulnerabilities.

5.1

CPS Vulnerability Filter

We set up a query �lter using PyCharm Edu python environment
to retrieve CPS-relevant vulnerability instances from NVD, as wellas vendor websites, and ICS CERT. The setup of the query logic is
important to identify all the relevant vulnerability instances, andalso to �lter out possible false positives based on other keywords
that distinguish them from the CPS concept. For example, in somecases using MTU as the only keyword might retrieve vulnerabilities either relevant to Maximum Transfer Unit (e.g. vulnerability
instance CVE-2005-0065) or Master Terminal Unit (e.g. vulnerability
instance CVE-2015-0990). Therefore, to retrieve the Master Terminal Unit relevant vulnerabilities, we de�ne further keywords like
SCADA-server. In our method, we extracted the entities of vulnerable components in CVE vulnerability reports using an open-source
NER (referring to Named Entity Recognition) model [17], and thenmap retrieved entities with the Common Platform Enumeration(CPE) as well as vendor websites, to generate a list of terms for thesecomponents, as illustrated in Fig. 2 . These terms are combined withcertain versions in interest as tags to query vulnerability instancesfrom open-source vulnerability data sources.
We also retrieved vendor information for each extracted CPS vulnerability instance using NER, and correlate them against CPE database. Vendor names in CPE metadata may appear with variations.
For example, the vendor Schneider Electric SE has variant forms
like ’schneider-electic’, ’chneider-electric’, and ’schneider-electric’. We
conduct some manual checking on vendor metadata, and further
optimise our search engine to detect hidden metadata for eachvendor, to decrease possible false negatives.

Figure 2: Generate Query Tags

5.2

CPS Vulnerabilities

Considering the nature of CPS, we de�ne an asset component to beeither software, hardware (e.g. a hard drive), or operating system(OS) (e.g. a Windows system). In CPS, hardware, software, and OS
are assembled and used in di�erent ways within CPS fabrics, whichmight create various binaries with potential backdoors. Outdated
software might contain �aws in source code. Such �aws might be
exploited by a bypass threat that is further materialised by a codeinjection attack, triggered by malicious actors. CPS hardware isvulnerable by having no physical-access protection, which might beused by an attacker to gain unauthorised physical access throughUSB for instance. An OS that contains resource management error might be exploited by a Denial of Service (DoS) threat into a

SIN 2020, November 4–7, 2020, Merkez, Turkeybu�er over�ow attack, which might result in loss of control of thisOS. Therefore, a vulnerability could be regarded as an emergent
property of an asset within CPS. Combining the Criticality of vulnerable assets, the Severity of emerging Vulnerability instances, aswell as the Likelihood of exploiting attacks could contribute to thecomputation of the asset Vulnerability Index. In this paper, we focus
on the Severity score only.
We demonstrate our streamlined vulnerability-severity scoring
method considering prominent CPS components such as PLCs,
RTUs, MTUs and HMIs. A Programmable Logic-Controller (PLC)
is a crucial CPS asset that controls industrial devices, to keep
production-processes in order. A Remote Terminal Unit (RTU) transmits telemetry data from sensing devices that are associated withpower physical-components to a Master Terminal Unit (MTU) system. A Human Machine Interface (HMI) provides a communicationinterface to visualise and monitor MTU activities and RTU information �ow [7].
The results obtained from querying our database, show 89, 32,
8, and 105 reported vulnerability instances related to PLC, RTU,
MTU and HMI respectively. These data are retrieved till October
27th, 2020. Although the amounts of encountered reports for CPS
assets increase in the recent ten years, still the total reports arelimited compared to the vulnerability instances for the commonly
seen information technology systems. Some of these vulnerabilities
are not scored or classi�ed till the retrieval date, so we �rst use
our system to �ll-up any missing score or inconsistent score before
further analysis.

5.3

Yuning Jiang and Yacine AtifDi�erent cybersecurity analysis parties seem to have better agreement on Authentication metric from CVSS V2, as well as Privileges
Required and User Interaction metrics from CVSS V3.

5.4

CPS Vendors

Subsequently, we generate a word-net to illustrate the main characteristics of retrieved vulnerability instances from di�erent vendors.
We select some distinct vendors for CPS vulnerabilities, namely
Schneider Electric SE and Siemens AG, and extract vulnerabilities
relevant to these two vendors. In total, we found 24 vulnerability
instances in a�ected products from Schneider Electric SE, and 39 vulnerability instances in a�ected products from Siemens AG. Then, we
conduct some text preprocessing on compiled vulnerability reports.
We use CountVectorizer and TdidfTransforer from Scikit-learn library
to extract keywords with di�erent levels of importance. We also
customised a stop-word list to remove vendor or product-version
related information. Then, we use WordNetLemmatizer from the
Natural Language Toolkit (NLTK) to discover top keywords in vulnerability reports in the form of word-nets. The top-50 keywords ofvulnerability reports relevant to Schneider Electric SE and Siemens
AG are illustrated in Fig. 3 (a) and (b), respectively. A keyword
with bigger size in a word-net represents higher importance to thatvendor. These two word-nets have some keywords in common, likeserver, web, remote, denial, etc. These common keywords highlight atrend of network-based exploits for retrieved vulnerabilities. Denialof service seems to be a common attack exploiting vulnerabilitiesin Schneider Electric SE and Siemens AG.

Inconsistent Scores of CPS Vulnerabilities

Here we compare the inconsistent CVSS scores of CPS vulnerabilities, with a purpose of exploring what makes these scoresinconsistent. We consider three types of scoring sources, namely
NVD, ICE-CERT, and vendors.
Taking PLC relevant vulnerabilities as an example, in total, 8CVSS V2 data records of PLC vulnerabilities are inconsistent. Thesevulnerability instances have IDs CVE-2015-3938, CVE-2015-7937,
CVE-2012-3037, CVE-2013-0664, CVE-2015-6462, CVE-2011-5007, CVE2015-6461, CVE-2014-0754. We also compare the CVSS v3 scores directly retrieved from NVD with scores mined from other authoritybound sources, and amend inconsistent results when taking vendors’ and third-party reports into consideration. In total, 6 CVSS v3
scores of PLC vulnerability records reported in NVD are modi�ed.
These vulnerability instances have IDs CVE-2016-8354, CVE-201810594, CVE-2018-7791, CVE-2017-14465, CVE-2017-14470, CVE-20176023.
Taking MTU relevant vulnerabilities as another example, 3 CVSS
V2 data records (i.e. CVE-2014-0752, CVE-2014-2375, CVE-2015-0990)
of MTU vulnerabilities are inconsistent, and 1 CVSS V3 data record(i.e. CVE-2020-6970) of MTU vulnerabilities is inconsistent.
We conduct further statistical analysis on the rationale of inconsistent scores. For CVSS V2 scoring, the most potential inconsistency comes from Access Complexity metric, followed by Con�dentiality Impact and Integrity Impact metrics. For CVSS V3 scoring,the most possible inconsistence also comes from Attack Complexity metric, followed by Scope and Con�dentiality Impact metrics.

Figure 3: Top Keywords Relevant to Schneider and Siemens

Cyber-Physical Systems Vulnerability Severity

SIN 2020, November 4–7, 2020, Merkez, Turkey

Table 2: Measurements Distribution of Base Metrics
MetricAttackPath
AttackComplexity
PrivilegesRequired
UserInteraction
ScopeChangeCon�dentialityImpact
IntegrityImpact
AvailabilityImpact

Measurement
Network
AdjacentNetwork
Local
Physical
Low
High
None
Low
High
None
Required
Unchanged
Changed
None
Low
High
None
Low
High
None
Low
High

PLC
93.26%
0%
5.62%
1.12%
97.75%
2.25%
95.51%
4.49%
0%
83.14%
16.85%
92.13%
7.87%
43.82%
7.87%
48.31%
42.70%
7.87%
49.45%
13.48%
0%86.52%

RTU
100%
0%
0%
0%
100%
0%
78.13%
12.50%
9.37%
100%
0%
100%
0%
21.87%
0%78.13%
46.88%
0%53.12%
28.13%
0%71.87%

MTU
87.50%
0%12.50%
0%
100%
0%
100%
0%
0%
87.50%
12.50%
100%
0%50.00%
0%
15.00%
50.00%
0%50.00%
37.50%
0%62.50%

HMI
84.76%
0.95%
13.33%
0.95%
99.05%
0.95%
92.38%
7.62%
0%67.62%
32.38%
85.71%
14.29%
17.14%
10.48%
72.38%
39.05%
10.48%
50.48%
29.52%
1.90%
68.57%

CPS
90.17%
0.43%
8.55%
0.85%
98.72%
1.28%
91.45%
7.26%
1.28%
78.63%
21.37%
90.60%
9.40%
26.92%
8.97%
64.10%
43.16%
7.69%
49.15%
22.65%
0.85%
74.36%

CVE
74.35%
22.57%
2.01%
1.06%
91.21%
8.79%
69.55%
25.18%
5.28%
62.80%
37.20%
83.64%
16.36%
22.15%
19.10%
58.75%
31.14%
17.20%
51.66%
38.22%
2.30%
61.19%

Figure 4: CVSS Version3 Base-Scores Distribution of CPS Vulnerabilities (2002-2020)

5.5

Characteristic Analysis for CPS
Vulnerabilities

For retrieved CPS vulnerabilities, we re-compute their scores
using our scoring system introduced in Section 4. Then, we examine
the diversity of their sub-scores to better understand the metrics
composition introduced in CVSS V3 scores, through the analysis of
sub-vectors. Here, we mainly emphasise the base dimensions. Base
attributes of CPS vulnerabilities can be partially investigated by therate of actually assigned values to Exploitability, Scope and Impact
metric measurements, as presented in Table 2. We list the attributes
for each CPS asset separately in Columns 3-6, list the attributes

for CPS on average in Column 7, and list the overall percentage of
published CVE reports in Column 8.
Compared to the average characteristics of their instances in
CVE, the exploitability sub-vectors of CPS vulnerability measurements indicate that most (90.17%) of the attacks are Network-based,
especially for RTU vulnerability instances. Adjacent network-based
attacks have limited presences in CPS. In contrast, local attacks
have more presences in CPS. Most CPS vulnerability instances, i.e.
98.72%, can be exploited by low-skilled attackers, given the observed
attack-complexity patterns, and the assigned "low" AttackComplexity measurement. Limited CPS vulnerability instances need highprivileges. 21.37% of exploits require user interactions. Only 9.40%
of exploits in CPS result in changed scope and possibly more severe impact. Compared to exploitability and scope sub-vectors, theSIN 2020, November 4–7, 2020, Merkez, Turkey

impact sub-vectors of CPS vulnerability instances present higher
diversity among possible impact values. It is also noticeable that
CPS vulnerability instances bring more impact to the con�dentiality
or availability than the integrity of the vulnerable components.
We compute the severity base-scores of retrieved CPS vulnerabilities using CVSS V3, as illustrated in Fig. 4. Both HMI and
PLC vulnerabilities base-scores have high presences at 6.5 and 8.5.
Base-scores of RTU vulnerabilities have a top presence at 8.5. MTU
vulnerabilities have more variant distributions in the score threshold [4.5-8.5]. These vulnerability instances show an average CVSS
V3 Base-Scores of 7.54, 8.00, 6.88 and 7.51 for PLC, RTU, MTU and
HMI respectively. We further map these CVSS V3 base-scores to
the CVSS qualitative severity rating scale2 . Most of the CPS vulnerability instances have either medium ([4.0-6.9]) or high ([7.0-8.9])
severity.

6

CONCLUSION AND FUTURE WORKS

Understanding, pinpointing and scaling vulnerabilities in CPS networks are challenging tasks, yet vital for cybersecurity purposes.
In this paper, we introduced a streamlined vulnerability-severity
scoring system based on the industrial standard CVSS mechanisms,
to discover the severity of a vulnerability instance. Our systemcomputes severity scores for di�erent CVSS versions, which improves the consistency of vulnerability analysis and thus related
prioritisation of remediation-tasks. We applied majority voting toreconcile inconsistent scores for the same vulnerabilities provided
by di�erent cybersecurity-repository sources, and used these reconciled vulnerability instances to train our machine-learning based
scoring model. Our model achieves good performance using balanced accuracy and micro F1-score evaluation metrics. Finally, we
provided a detailed evaluation of our proposed scoring approachthrough a case study of CPS vulnerability instances, using multiple
online cybersecurity data sources. In this case study, CPS vulnerability instances are retrieved using a query �lter with customised
keywords and logic. Using these extracted vulnerability instances,
we analyse the characteristics of CPS vulnerabilities, in comparison with the general characteristics of existing vulnerabilities in
CVE repository. Typically, reported CPS vulnerability instances aremostly network-based or local-based. Exploiting a CPS vulnerability generally expects low attack complexity, low privilege, and low
user interaction. A successful attack on CPS vulnerabilities probably brings a high impact on the availability of the related CPS asset.
The scores of reported CPS vulnerabilities present a centralised
distribution in CVSS V3 base-score scale, especially in the scorescales of medium- and high-severity.
As part of our future works, we can further optimise our scoring
system by adjusting the tie of majority voting under experts’ supervision. Such reliability evaluation can start with inviting security
experts to set an initial value, and then apply some computational
intelligence techniques to adjust it further. Another possible extension may use the weighted arithmetic mean of di�erent scoresfrom several sources, while measuring the weights by evaluating
the reliability of the score sources. We also plan to explore thedi�erences between applying majority voting to the sub-metrics of2 https://www.�rst.org/cvss/v3.1/speci�cation-document

Yuning Jiang and Yacine Atif

CVSS, with the method used in this paper whereby majority voting
is applied to the CVSS base-scores directly.

REFERENCES[1] Luca Allodi and Fabio Massacci. 2014. Comparing vulnerability severity andexploits using case-control studies. ACM Transactions on Information and System
Security (TISSEC) 17, 1 (2014), 1–20.
[2] Kay Henning Brodersen, Cheng Soon Ong, Klaas Enno Stephan, and Joachim M
Buhmann. 2010. The balanced accuracy and its posterior distribution. In 2010
20th International Conference on Pattern Recognition. IEEE, 3121–3124.
[3] Haipeng Chen, Rui Liu, Noseong Park, and VS Subrahmanian. 2019. Using
twitter to predict when vulnerabilities will be exploited. In Proceedings of the 25th
ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
3143–3152.
[4] Steve Christey and Brian Martin. 2013. Buying into the bias: Why vulnerability
statistics suck. BlackHat, Las Vegas, USA, Tech. Rep 1 (2013).
[5] Marian Gawron, Feng Cheng, and Christoph Meinel. 2017. Automatic vulnerability classi�cation using machine learning. In International Conference on Risks
and Security of Internet and Systems. Springer, 3–17.
[6] Dan Geer and Michael Roytman. 2013. Measuring vs. modeling. ; login:: themagazine of USENIX & SAGE 38, 6 (2013), 64–67.
[7] Abdulmalik Humayed, Jingqiang Lin, Fengjun Li, and Bo Luo. 2017. Cyberphysical systems security—A survey. IEEE Internet of Things Journal 4, 6 (2017),
1802–1831.
[8] Pontus Johnson, Robert Lagerström, Mathias Ekstedt, and Ulrik Franke. 2016.
Can the common vulnerability scoring system be trusted? a bayesian analysis.
IEEE Transactions on Dependable and Secure Computing 15, 6 (2016), 1002–1015.
[9] Atefeh Khazaei, Mohammad Ghasemzadeh, and Vali Derhami. 2016. An automatic
method for CVSS score prediction using vulnerabilities description. Journal ofIntelligent & Fuzzy Systems 30, 1 (2016), 89–96.
[10] Marcus Pendleton, Richard Garcia-Lebron, Jin-Hee Cho, and Shouhuai Xu. 2016.
A survey on systems security metrics. ACM Computing Surveys (CSUR) 49, 4(2016), 1–35.
[11] David Martin Powers. 2011. Evaluation: from precision, recall and F-measure to
ROC, informedness, markedness and correlation. (2011).
[12] Jukka Ruohonen. 2019. A look at the time delays in CVSS vulnerability scoring.
Applied Computing and Informatics 15, 2 (2019), 129–135.
[13] Karen Scarfone and Peter Mell. 2009. An analysis of CVSS version 2 vulnerability
scoring. In 2009 3rd International Symposium on Empirical Software Engineeringand Measurement. IEEE, 516–525.
[14] Georgios Spanos, Lefteris Angelis, and Dimitrios Toloudis. 2017. Assessment ofvulnerability severity using text mining. In Proceedings of the 21st Pan-Hellenic
Conference on Informatics. 1–6.
[15] Dapeng Tao, Jun Cheng, Zhengtao Yu, Kun Yue, and Lizhen Wang. 2018. Domainweighted majority voting for crowdsourcing. IEEE transactions on neural networks
and learning systems 30, 1 (2018), 163–174.
[16] Yasuhiro Yamamoto, Daisuke Miyamoto, and Masaya Nakayama. 2015. Textmining approach for estimating vulnerability score. In 2015 4th International
Workshop on Building Analysis Datasets and Gathering Experience Returns for
Security (BADGERS). IEEE, 67–73.
[17] Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint
arXiv:1703.06345 (2017).
[18] Ziyun Zhu and Tudor Dumitraş. 2016. Featuresmith: Automatically engineeringfeatures for malware detection by mining the security literature. In Proceedingsof the 2016 ACM SIGSAC Conference on Computer and Communications Security.
767–778.