Attacks against intrusion detection networks: evasion, reverse engineering and optimal countermeasures

Intrusion Detection Networks (IDNs) constitute a primary element in current cyberdefense systems. IDNs are composed of different nodes distributed among a network infrastructure, performing functions such as local detection --mostly by Intrusion Detection Systems (IDS) --, information sharing with other nodes in the IDN, and aggregation and correlation of data from different sources. Overall, they are able to detect distributed attacks taking place at large scale or in different parts of the network simultaneously. IDNs have become themselves target of advanced cyberattacks aimed at bypassing the security barrier they offer and thus gaining control of the protected system. In order to guarantee the security and privacy of the systems being protected and the IDN itself, it is required to design resilient architectures for IDNs capable of maintaining a minimum level of functionality even when certain IDN nodes are bypassed, compromised, or rendered unusable. Research in this field has traditionally focused on designing robust detection algorithms for IDS. However, almost no attention has been paid to analyzing the security of the overall IDN and designing robust architectures for them. This Thesis provides various contributions in the research of resilient IDNs grouped into two main blocks. The first two contributions analyze the security of current proposals for IDS nodes against specific attacks, while the third and fourth contributions provide mechanisms to design IDN architectures that remain resilient in the presence of adversaries. In the first contribution, we propose evasion and reverse engineering attacks to anomaly detectors that use classification algorithms at the core of the detection engine. These algorithms have been widely studied in the anomaly detection field, as they generally are claimed to be both effective and efficient. However, such anomaly detectors do not consider potential behaviors incurred by adversaries to decrease the effectiveness and efficiency of the detection process. We demonstrate that using well-known classification algorithms for intrusion detection is vulnerable to reverse engineering and evasion attacks, which makes these algorithms inappropriate for real systems. The second contribution discusses the security of randomization as a countermeasure to evasion attacks against anomaly detectors. Recent works have proposed the use of secret (random) information to hide the detection surface, thus making evasion harder for an adversary. We propose a reverse engineering attack using a query-response analysis showing that randomization does not provide such security. We demonstrate our attack on Anagram, a popular application-layer anomaly detector based on randomized n-gram analysis. We show how an adversary can _rst discover the secret information used by the detector by querying it with carefully constructed payloads and then use this information to evade the detector. The difficulties found to properly address the security of nodes in an IDN motivate our research to protect cyberdefense systems globally, assuming the possibility of attacks against some nodes and devising ways of allocating countermeasures optimally. In order to do so, it is essential to model both IDN nodes and adversarial capabilities. In the third contribution of this Thesis, we provide a conceptual model for IDNs viewed as a network of nodes whose connections and internal components determine the architecture and functionality of the global defense network. Such a model is based on the analysis and abstraction of a number of existing proposals for IDNs. Furthermore, we also develop an adversarial model for IDNs that builds on classical attack capabilities for communication networks and allow to specify complex attacks against IDN nodes. Finally, the fourth contribution of this Thesis presents DEFIDNET, a framework to assess the vulnerabilities of IDNs, the threats to which they are exposed, and optimal countermeasures to minimize risk considering possible economic and operational constraints. The framework uses the system and adversarial models developed earlier in this Thesis, together with a risk rating procedure that evaluates the propagation of attacks against particular nodes throughout the entire IDN and estimates the impacts of such actions according to different attack strategies. This assessment is then used to search for countermeasures that are both optimal in terms of involved cost and amount of mitigated risk. This is done using multi-objective optimization algorithms, thus offering the analyst sets of solutions that could be applied in different operational scenarios. -------------------------------------------------------------