# Quality Dimensions for Automatic Assessment of Structured Cyber Threat Intelligence Data
## 2 Review of Related Work
The first class consists of five indexes, the second class consists of nineteen indexes, and the third class consists of more than fifty indexes. The typical properties of data quality, which include timeliness, relevance, accuracy, completeness, predictability, performability and customizability, can be observed among the indexes. However, the measurement of individual properties is not discussed. The evaluation is carried out manually by the experts via a questionnaire. The weights for each criterion are determined according to expert experience. Then the obtained values are adjusted using an algorithm of multi-objective optimization. Such an approach is more oriented to the evaluation of CTI data providers. The authors recognize that evaluation of CTI at the micro level is still an important research direction. The method presented is complex, based mainly on manual evaluation. [PERSON] et al. [21] introduced a CTI feed ranking approach. A prerequisite of the approach is the requirement of at least two snapshots of each considered feed. The key properties for evaluating feeds are as follows: completeness, accuracy, and speed. Completeness incorporates how many entries are covered. Accuracy includes how many entries are confirmed by other feeds. Speed includes the order in which the entries were covered. The naming of the first two properties is similar to the dimensions introduced by the other considered authors. However, the definitions of these properties are quite different. Therefore, the authors recognize that such criteria cannot be used to evaluate single entries from CTI feeds. [PERSON] et al. [22] developed threat intelligence metrics to compare threat intelligence sources and described methodology for measuring them. The metrics included the following six indicators: volume, differential contribution, exclusive contribution, latency, coverage, and accuracy. The indicators of differential contribution, exclusive contribution, and latency are measured only by comparison of two feeds. The indicators of volume, coverage, and accuracy have meanings described in their names. The authors recognize that the accuracy or coverage of a feed cannot be measured precisely in a systematic way. In addition, the feeds to be compared were divided into six categories. The comparison was made according to these categories. [PERSON] et al. [23] proposed methodology for evaluating cyber threat information sources. The methodology is based on the standardized Structured Threat Information eXpression (STIX) format. Ten quantitative parameters were introduced. They are as follows: extensiveness, maintenance, false positives, verifiability, intelligence, interoperability, compliance, similarity, timeliness, and completeness. The presented parameters were used to assess the content and the providers of the content, with more attention paid to the providers. However, the parameters were not separated into different groups to differentiate between CTI data and CTI providers. The next limitation of the approach is that all sources were compared against other sources within the monitored community of CTI sources. Many formulae for the parameters include the definitions of values that are declared as \"in the world view\". Such an approach is not suitable to evaluate single new entry in CTI feeds. [PERSON] et al. [24] proposed CTI quality metrics that rely on the standardized STIX format. The authors did not acknowledge the work by [PERSON] et al. [23] and declared that they were the first ones to provide metrics on the standardized STIX format. The proposed CTI quality metrics follow an empirical approach [15]. However, the arguments for choosing specific dimensions from the approach by [PERSON] and [PERSON] [15] were not provided. In addition, input from a number of CTI practitioners and researchers was sought as well. The proposed metric was structured into three different levels: report, object, and attribute. The attribute level is based on evaluation of the available STIX objects.