# PALANTIR: An NFV-Based Security-as-a-Service Approach for Automating Threat Mitigation
## 6 Validation
### Quantitative Evaluation
#### 6.3.1 Performance Evaluation of the Security Capability Selection Process  
As detailed in Section 4.3, the SM component determines the optimal deployment plan, considering financial restrictions, which satisfies the security needs to cope with any given detected malevolent activity. To that extent, this component accounts for the available SCs to consider and the configurations of the infrastructure (i.e., the entities defining the deployment location for a given SC). However, increasing the number of SCs also increases the complexity on the constraint model to be generated. This may have a dramatic impact on the performance, since the constraints engine solver (part of the SM) eventually relies on heuristic search to find an optimal solution. The objectives of the following experiments are to (i) quantify the impact of the model complexity on the component performance; and (ii) determine a point of inflexion in the model complexity, suggesting a partition in our satisfaction problem model. In practice, these experiments will help us to understand (i) in which conditions the SM should be scaled up to maintain acceptable performance for the end-user; and (ii) when the data collected by the component should be distributed into separated batches to be processed iteratively. We understand the complexity of our model as the required number of variables and constraints given a specified number of both SCs and infrastructures, the latter modelled as infrastructure configurations. Such configurations comprise the characteristics of the infrastructures, e.g., the available amount of RAM memory, CPUs, storage capacity and supported deployment models. The number of variables to be solved is directly proportionate to the number of available SCs and infrastructure configurations. Therefore, only one of these parameters needs to be modified to quantify the different complexities of the model. In this case, we chose to modify the number of available infrastructure configurations, but a similar approach could have achieved by modifying the number of available SCs. Table 2 lists the number of deployment configurations populated in the model, depending on the number of infrastructure configurations submitted to the SM component and assuming a fixed number of 40 eligible SCs. Our evaluation work was conducted on a HW comprising 8-core Intel(R) Core(TM) i7-8550U at 1.8 GHz, with 16 GB of RAM, Linux operating system version 5.15.81 and Java Runtime environment version 11.0.17. We simulated the SO and SCC components by mocking their interfaces, that is, by exposing JSON files mimicking their responses through an HTTP server. A local Kafka broker (version 2.7.1) and a PostgreSQL database management (version 14.6) were locally deployed as dependencies of the prototype for the SM component. For this evaluation, we have focused solely on the solving procedure and components, and we have limited our measurements to the context of a threat. Wehave therefore extracted the measurements by exploiting the Java ThreadMXBean and MemoryMXBean interfaces to retrieve memory and CPU time metrics, as well as the timeCount interface from the Choco solver to retrieve the duration of the solver execution. Our measurement process consisted in iterating 50 requests on the same prototype instance for each evaluated model complexity. However, the evaluation of each model complexity has been conducted on different instances to ensure independence across measurements. In our first experiment, we measured the heap RAM consumption depending on the complexity of the model, where Figure 8 exposes the results. We noticed a stable memory consumption profile for models with a complexity lower than 55 infrastructure configurations, but also that the memory consumption profile is not converging when the profile equals 65 infrastructure configurations. We also noticed that the profiles are converging (yet are not stable) for complexities exceeding 75 infrastructure configurations.