Patent Description:
A key indicator of a quality of a security product is the level of threat detection, which conventionally is evaluated using specific test collections of files and objects, such as hyperlinks, depending on the type of a detection engine.

Testing is a part of a release cycle of a security application and is conducted at specific stages in the release of the application. After the release of applications, the product is tested either for benchmarking by test laboratories, or to check for errors when releasing new versions of detecting databases, such that there are no false positives or errors that affect the performance of the product.

This approach controls the quality of the product only at certain points in time and does not allow monitoring the quality of detection in real time.

From publication Anonymous article "<NUM> things you need to know about testing AV software with VirusTotal's free online multiscanner", describes the service for external scanning of objects from a malicious collection. In particular, the advantages and disadvantages of this service are described when used by business users in comparison with the installed anti-virus protection solution at the end. The article doesn't disclose how to process missed malware by antimalware vendors and how to automate the response. Moreover, it doesn't disclose the case of false positive (in accordance with VirusTotal verdict), when the antivirus is detecting malware and VirusTotal does not.

Patent document <CIT> describes a technology for collecting and processing network information in connection with a file, including data on the detection of a file by a dynamic and static scanner. The invention is applicable to expand the set of file metadata and identify infected objects on the network, track file distribution paths. The invention <CIT> doesn't teach how to improve the quality of detection by comparing the anti-virus product's own verdicts and the results of external tests, related to any types of the system objects, not limited by only files.

<NPL>" describes the issue related to the discrepancy between the verdicts of various vendors of anti-virus software and the verdict of the virustotal service. The publication also describes a methodology for assessing the change and discrepancy of verdicts, such as the delay in changing verdicts to the correct one, the accuracy of detection of the virustotal service. This publication does an excellent job of explaining the issue that a vendor should automate the processing of verdicts of files, links and other objects, taking into account discrepancies with external scanners such as virustotal. However, the publication does not disclose the solution of a stated problem.

In addition to the product testing procedure, the quality of detection is also influenced by the sources of files, URLs, and other objects that are used to populate the database of threats and clean files. The quality of these collections (i.e., confidence in the classification of objects in the collection, the size of the collection, and the number of new previously unknown files in the collection) directly determines the level of false positives, the level of detection, and the system performance by optimizing the work of filtering techniques of detectors, which can reduce time-consuming operations for dynamic analysis of objects on end machines of users.

In conventional methods and systems, testing of the quality of detection of threats by engines are carried out without assessing dynamics of quality indicators in comparison with third-party solutions and services, which would not allow performing an in time relative assessment of the quality of detection and identifying areas for improvement.

In such systems and methods, it is required to manually obtain various collections of objects and carry out tests on the objects, which would not necessarily provide an objective assessment and a result to allow moving on to improve the products without additional analysis.

Thus, it is necessary to solve the problem of dynamic analysis of detection metrics for various versions of products, decoding engines, and file collection sources, to manage the process of filling the detection databases and to update products with fixes in the detection mechanisms.

It is an exemplary feature of the present invention to provide iterative dynamic assessment of a quality of detection by periodically scanning objects from collections of various sources with each supported product and/or scan- ning engine and comparing verdicts with the verdicts of third-party scanning services.

Based on the results of the assessment, the level of trust in the sources of collections is determined, and the classes of objects that are detected by products and engines with a quality level lower than the supporter of services are allocated, and a task is created for adjusting the rules for detection engines and products. The process is repeated cyclically to achieve a specified quality of threat detection.

The exemplary invention provides a computer-implemented method of continuous development of an internal threat scan engine based on an iterative quality assessment.

The method includes iteratively performing a dynamic assessment of a quality of a threat detection with a frequency defined for each of objects in an object collection, wherein a result of the dynamic assessment includes internal and external scan results of the objects and a consistency verdict of the internal and external scan results of the objects, changing a frequency of scanning iteration of the objects based on the consistency verdict of the external and internal scan results of the objects, classifying the objects based on the result of the dynamic assessment, creating a development task including the internal and external scan results of the objects, meta-data of the objects, and automated test results to provide details for developing a software to fix inconsistency of the internal and external scan results of the objects, controlling the dynamic assessment in accordance with a dynamic of implementation of the development task, and maintaining the quality of the threat detection on a given level based on the controlled dynamic assessment and a priority of the development task.

The exemplary invention also provides a system for continuous development of an internal threat scan engine based on an iterative quality assessment.

The system includes a processor coupled to a memory storing instructions. The processor is configured to iteratively perform a dynamic assessment of a quality of a threat detection with a frequency defined for each of objects in an object collection, wherein a result of the dynamic assessment includes internal and external scan results of the objects and a consistency verdict of the internal and external scan results of the objects, change a frequency of scanning iteration of the objects based on the consistency verdict of the external and internal scan results of the objects, classify the objects based on the result of the dynamic assessment, create a development task including the internal and external scan results of the objects, meta-data of the objects, and automated test results to provide details for developing a software to fix inconsistency of the internal and external scan results of the objects, control the dynamic assessment in accordance with a dynamic of implementation of the development task, and maintain the quality of the threat detection on a given level based on the controlled dynamic assessment and a priority of the development task.

The exemplary system is able to classify the objects that are detected with a worse quality and to speed and prioritize iteratively improving the quality. Further, it would be possible to test the detecting databases and the engine, taking into account the level of trust in collections of objects from individual sources and to individual classes of objects.

The exemplary aspects of the invention will be better understood from the following detailed description of the exemplary embodiments of the invention with reference to the drawings:.

Exemplary embodiments of the invention will now be described with reference to the drawings.

As shown exemplarily in <FIG>, the process <NUM> for the iteration of the detection quality assessment includes receiving an object from an object collection as shown in step <NUM>. The object is scanned by an internal scan engine in step <NUM> and by an external scan engine in step <NUM>.

In step <NUM>, the external scan result and object metadata that include at least the time of the first time object detection are obtained, and in step <NUM>, the internal scan result and object metadata are obtained.

Metadata may, for example, be related to VirusTotal, which includes a full list of engines in use, a list of existing privileges, etc. Metadata can be used for object classification, since the metadata is included in a scan result fully or partly.

In step <NUM>, a composite verdict including the results of internal and external scans and object meta-data from internal and external scan engines is obtained.

Step <NUM> includes updating the object history information with the composite verdict of iteration, and the composite verdict is assessed at step <NUM>.

When the composite verdict is determined to be consistent in step <NUM>, a task "A" is scheduled for the next iteration in step <NUM>, a list of object sources that provided the object is obtained in step <NUM>, and the trust level of object source, based on the completed iteration, is updated in step <NUM>.

The composite verdict is determined to be inconsistent when some pairs of verdict attributes are not equal. For example, scanning verdicts (e.g., malicious or black, suspicious, or grey and white) differ, or security rating (a number) also differ, or the class of threat is different (trojan or virus or ransomware), or meta-data contain additional data, etc..

When the composite verdict is determined to be inconsistent in step <NUM>, the object is classified with other objects within collection based on a composite verdict in step <NUM>, automated tests to verify the inconsistency of result for the class of objects is conducted in step <NUM>, and in step <NUM>, the object history information with automated test results is updated.

When the internal verdict is determined to be correct in step <NUM>, the process proceeds to step <NUM> to schedule the task "B" for the next iteration. Scheduled time, scope of external and internal engines and object collections are set for task "A" and "B" independently.

When the internal verdict is determined to be incorrect in step <NUM>, a scan engine development task with a composite verdict and automated test results are created in step <NUM>, and a task is scheduled for the next iteration in step <NUM> by obtaining a list of object sources that provided the object in step <NUM>.

<FIG> shows an exemplary system for the iteration of the detection quality assessment.

As shown in <FIG>, the composite verdict analyzer <NUM> exchanges information with an iteration scheduler <NUM>, an external scan manager <NUM>, an internal scan manager <NUM>, and object history <NUM>, by receiving object <NUM> and generating task <NUM> that is provided to a development operation system <NUM>.

The iteration scheduler <NUM> interacts with the external scan manager <NUM>, the internal scan manager <NUM>, and the automated tests <NUM>.

More specifically, the iteration scheduler <NUM> controls the flow of the scanning during the iteration and scheduling the next iteration based on the composite verdict.

The internal scan manager <NUM> communicates with security applications <NUM> and <NUM>, and internal scan engine <NUM> by sending commands or objects to internal scan engines or products to get current scan results, using API, command line, or other interfaces.

The external scan manager <NUM> communicates with external scan engines <NUM> and <NUM> by sending commands or objects to third-party tools and engines to get current scan results, using API, command line, or other interfaces.

The object <NUM> is provided from the object collections <NUM> that includes object collections <NUM> and <NUM> communicating with object sources <NUM> and <NUM>. The object <NUM> may include files (e.g., scripts, exe files, documents, webpages etc.), URLs, IP addresses, domain names, etc..

In an exemplary aspect, the system shown in <FIG> includes a processor of a computer coupled to a memory <NUM> storing instructions. The processor is configured to iteratively perform a dynamic assessment of a quality of a threat detection with a frequency defined for each of objects <NUM> in object collections <NUM>.

A result of the dynamic assessment includes internal and external scan results of the objects, by internal scan engine <NUM> and external scan engine <NUM>, respectively, and a consistency verdict of the internal and external scan results of the objects <NUM>.

The system changes a frequency of scanning iteration of the objects <NUM> based on the consistency verdict of the external and internal scan results of the objects assessed by the composite verdict analyzer <NUM>.

The system further classifies the objects <NUM> based on the result of the dynamic assessment, create a development task <NUM> including the internal and external scan results of the objects <NUM>, meta-data of the objects, and automated test results to provide details for developing a software to fix inconsistency of the internal and external scan results of the objects.

The system also controls the dynamic assessment in accordance with a dynamic of implementation of the development task <NUM> and maintains the quality of the threat detection on a given level based on the controlled dynamic assessment and a priority of the development task <NUM>.

<FIG> shows an exemplary entity relationship diagram for an exemplary malicious code detection quality control system for an antivirus engine.

In an exemplary aspect of the present invention, a system automatically receives information about malicious and clean files from various sources (e.g., VirusTotal feed, MalaShare, workstations, etc.). The system automatically classifies files into categories, for example, black, white, and gray, and allows to obtain a confusion matrix and its dependence on time for various verdict providers on different datasets.

Aggregation by file appearance and last scan dates, datasets and any combination of these parameters are available, and collection and storage of all historical information is provided (for example, as scan results). The system allows visualization of the metrics, and works in real time (i.e., behind by the time required for classification).

In the above-described process, as shown in the exemplary <FIG>, the data source includes a resource or an object that allows to receive information about malicious and clean files, which preferably include new files. The dataset is obtained from a specific data source, and the verdict provider includes an antivirus engine installed on Virus- Total, BitDefender, CleanSet, or another source that provides useful information about the file. The task includes a program that performs some work including receiving new files, scanning, receiving verdicts, storing results into a database, etc., and the detection quality includes a confusion matrix value.

The entities shown in the exemplary <FIG> include data source, file, verdict provider, and verdict.

The data source has <NUM> attributes of id and name. It can produce files (e.g., only hashes are used) and fills in all required fields. One data source produces many files. At the same time one file can be produced by different data sources (i.e., ids varies).

File has properties. All required fields are filled in by data source, optional fields could be updated by verdicts. File could be scanned by different verdict providers.

Verdict Provider has <NUM> attributes of id and name, it provides verdicts. The temporal dependence of verdicts are considered, one file can be scanned several times to get several verdicts.

Verdict has <NUM> attributes that are filled by verdict provider, including id, file_id, scan_date, and result.

Task controls all system. Firstly, new files are produced by data sources. Then, among all files, the interested files are selected (e.g., new files that were received recently). After that the selected files are rescanned by verdict providers and new verdicts are received.

The entity relationship diagram shown in <FIG> are more clearly shown database schema of <FIG>.

<FIG> shows existing tables and their fields, upon which the following aspects of the data sources, files, verdict providers, and verdicts can be presented.

In an exemplary aspect, when creating the database, <NUM> empty tables are created. The table verdicts is created with <NUM> columns (id, file_id, and scan_date). When adding a record to the verdict_providers table, a column in the verdicts table with the corresponding name is created.

With regard to the component model, MDQCS includes the following components:.

The following tables describe the responsibility of each component:.

With regard to the metrics, the time of appearance can be considered as both the time of receipt by the system (i.e., first submission) and the time of appearance in VT system (i.e., first appearance).

With regard to detection quality dependence on time, the detection quality is calculated on the day the file appears, the next day, and so on, up to the limit that is set.

Regarding averaging the previous metric over time for a specified period, the averaging detection quality for <NUM> day files, for files the next day, and so on may be considered.

Average detection quality for files is received in a specified period. The number of files from one source are known to another and how quickly do they appear depend on the following:.

Claim 1:
A computer-implemented method of continuous development of an internal threat scan engine based on an iterative quality assessment, the method comprising:
iteratively performing a dynamic assessment of a quality of a threat detection (<NUM>) with a frequency defined for each of objects in an object collection (<NUM>), wherein a result of the dynamic assessment includes internal (<NUM>,<NUM>) and external scan results (<NUM>,<NUM>) of the objects and a consistency verdict (<NUM>) of the internal and external scan results of the objects;
changing a frequency of scanning iteration of the objects based on the consistency verdict of the external and internal scan results of the objects;
classifying the objects based on the result of the dynamic assessment (<NUM>);
creating a development task including the internal and external scan results of the objects, meta-data of the objects, and automated test results to provide details for developing a software to fix inconsistency of the internal and external scan results of the objects (<NUM>,<NUM>);
controlling the dynamic assessment in accordance with a dynamic of implementation of the development task; and
maintaining the quality of the threat detection on a given level based on the controlled dynamic assessment and a priority of the development task.