## 6 How to Evaluate Defense Systems
### [PERSON] et al. [46] set up a hybrid Cloud, using Amazon EC2 as a public Cloud and simulating a private Cloud in their lab. The private Cloud consists of two Linux machines, one of which hosting DaMask and the network controller while the other emulates a virtual network to extend the private Cloud. They wanted to measure the communication costs as well as the computation overhead of DaMask. They began by computing the network bandwidth between the private and the public Cloud, with and without DaMask. For the communication overhead, caused by the traffic being examined by DaMask, they showed the overhead was a constant if the link status of network remained stable. As for the computation overhead caused by the detection algorithm, they ended up with an interference time of 80 ms, which they consider quite efficient. [PERSON] et al. [51] evaluated their Cloud Traceback (CTB) and Cloud Protector in multiple ways. They conducted experiments to see how CTB marked packets and could determine if they led to a XML-DoS, but also how accurate CTB was at identifying the source of the attack. Moreover, they wanted to make sure their method was better than traditional security mechanisms such as WS-Security, when it comes to XML-DoS attacks. The experimentation for the Cloud Protector consisted in determining if it could detect and filter HTTP-DoS and XML-DoS messages. To generate the attack traffic, they opened up three virtual servers that contained 20 Firefox browsers and 20 open tabs to each browser, in addition to a page refresher tool and targeted a particular website. First, they studied the impact of the attack without their architecture and witnessed its tremendous and lasting effect as the web server was quickly unable to handle more than a few requests. Then, they divided the evaluation of CTB in two parts: one to simulate XML-DoS attacks against a web server and the other to compare CTB to traditional mechanisms. In their simulation, 100 messages were sent. If one was an XML-DoS attack, it was supposed to crash the server with a probability of 1/2. Out of those 100 messages, nine successfully crashed the server and CTB was able to identify seven of them, with a response time that varied between 480 and 550 ms. When comparing CTB with SOAP authentication, CTB proved to be far more effective in terms of response time and the same was true for WS-Security. Then, they assessed the performance of the Cloud Protector (a neural network) by training and testing it with a dataset they developed. On the trained dataset for HTTP-DoS, the Cloud Protector was able to identify 91% of the attack (9% of false positives), but with significantly different response times (between 20 ms and 1 s). On the test dataset, the accuracy decreased to 88%, with the same variation in response time. For XML-DoS, most of the attack messages were identified, detecting and removing them taking between 10 and 140 ms, but the response times were not as scattered as in the HTTP-DoS case. In [52], [PERSON] et al. came up with a new defense system called ENDER, that has no more than 1% of false positives on the same dataset. [PERSON] et al. [30] used the Eucalyptus middleware to manage resources in their experiment. The Cloud Resource Broker along with the DDoS defense system is installed on a four CPU server. Then, they set up Eucalyptus and the web services. They wanted to assess the impact of the attack, the mitigation capabilities of their system and the cost induced by this defense mechanism (extra response time). When the defense system was not present, a flooding attack with 4900 legitimaterequests during 60 s was enough to exhaust the CPU, whereas with the defense system, only the first request was accepted and the others were discarded as they exceeded the request rate. Moreover, some addresses were blacklisted. All in all, 10% of CPU was necessary to drop malicious packets. With an oversized XML attack, ten 12 MB messages were sufficient to keep the CPU busy for 15 min, while also significantly increasing memory usage.