# Quality Dimensions for Automatic Assessment of Structured Cyber Threat Intelligence Data
## 2 Review of Related Work
At the attribute level, six dimensions are considered. The object level dimensions (representational consistency and reputation) are not bound to the predefined attributes. Therefore, they are subjective. At the report level, a single indicator of the amount of data is considered. Consideration is presented for calculation of the amount of data. However, there is no final decision on how to compute this value. The obtained values at all levels are then aggregated into a combined quality indicator. The proposed quality dimensions were integrated into the STIX schema with the introduction of the custom object quality indicator. The following drawbacks of the approach can be observed. Firstly, the metrics proposed cannot be fully automatically calculated; the input of either data experts or data consumers is needed. Secondly, all the focus was concentrated on the quality of CTI products; the quality of the CTI providers was not considered separately. Thirdly, the metrics were tested in a small environment. [PERSON] et al. [25] defined an automatic method for the quality and trust metrics calculation. Quality included the following factors: completeness, freshness, timeliness, extensiveness, and relevance. However, the choice of factors was not discussed and was not justified. For the final quality assessment, only three factors with weight coefficients were aggregated. They were as follows: completeness, freshness, and extensiveness. Moreover, [PERSON] et al. [25] noticed that the calculation for completeness, freshness, and extensiveness required improvement. The trust metrics included unique factors in comparison with other studies considered. The factors are as follows: partner sharing activity, sector of activity, certified cybersecurity, previous ticket ratings, privacy, and partner sector. The choice of factors was not discussed and was not justified. The formula for the aggregation of the factors was not provided. [PERSON] et al. [26] performed a systematic literature review to extract a set of quality dimensions for threat data, information, and intelligence. The search included studies published between 2014 and 2019. A total of 22 articles were selected in the field of cyber security. The search was limited, since a lot of new articles appeared in the field up to the publication of the paper, which are referenced in the paper but not included in the counting of dimensions. The most common dimensions were as follows (frequencies are shown in parenthesis): timeliness (14), completeness (13), accuracy (12), consistency (8), relevance (8), and reliability (7). [PERSON] et al. [26] decided to include only the dimensions that were mentioned in all three contexts: data, information, and intelligence. They are as follows: accuracy, completeness, interoperability (portability), relevance, reliability, timeliness, and uniqueness. The systematic literature review then was followed by a Delphi study that involved 30 threat intelligence experts around Europe. Two rounds of Delphi study were performed. During the first round, the experts reviewed the selected dimensions. The majority of experts agreed that the completeness and uniqueness dimensions should be dropped from the list. The reasons are as follows: pursuing intelligence completeness could undermine its timeliness and actionability; and uniqueness often calls for further investigation to be sure that it is not the product of faulty analysis. During the second round, the experts were asked to add dimensions that were overlooked in the literature. The dimensions of actionability and provenance were added to the list. The main drawback of the study is that a measurement of dimensions was not considered. [PERSON] et al. [27] presented an automatic CTI quality evaluation method that linked the assessment of the trustworthiness of CTI feeds to assessment of CTI content. The authors argue that CTI quality depends on the trustworthiness of CTI feeds. The original amount of information provided by the feed is considered to be a feature of CTI feed trustworthiness.