# Network Defense Strategy Selection with Reinforcement Learning and Pareto Optimization
## 5 Results and Analysis  
The results show cumulative rewards over time. If the Pareto-Q-learning leads to an optimal solution, the cumulative rewards should increase over time in all three dimensions. In order to validate that this is the result of the approach and not the game definition, the Pareto-Q-learning rewards should be compared to a base case of arbitrary (random) actions and a simple Q-learning approach (i.e., the possible action space is not limited by Pareto optimization). A visual representation of the network with configuration of 100 nodes and 0.05 sparsity is given in Figures 8-10, and the results showing improvement over time are visible. Figure 8 shows the network availability (\(f^{(1)}\)), Figure 9 shows the node security (\(f^{(2)}\)) over the same games, and Figure 10 shows the defense strategy cost effectiveness (\(f^{(3)}\)) over the same games. The configuration for these figures are derived from Table 1, reflecting 100 nodes, a sparsity of 0.05, and both the defender and attacker start with 250 game points. The weights of the nodes are randomly generation within the limits described in Table 1.  
We implemented the game using Python version 3.6 (2017) and the NumPy library (2017), Python is a free software by Python Software Foundation, America, and NumPy is the fundamental package for scientific computing with Python. The experiment was run on a MacBook Pro 2016 model machine (MacOS Sierra Version 10.12.6) with an Intel Core i5 (dual-core 3.1 GHz) CPU and 8 GB of RAM  
Figure 8: \(f^{(1)}\) Network Availability. Figure 10: \(f^{(3)}\) Defense strategy cost effectiveness. Figure 9: \(f^{(2)}\) Node security. (Random-Access Memory). We use the machine's internal SSD (Solid State Drives) hard-disk for storing data. According to the experimental configuration in Section 4.3, the algorithm performs 200 simulation calculations. The result is shown in Figures 8-11, which represent the values of the game target \(f^{(1)}\)-\(f^{(3)}\) and the cumulative reward vector. As shown in Section 4.1, the value of the cumulative reward vector is calculated based on the weights of 6:2:2 for the game targets \(f^{(1)}\), \(f^{(2)}\), and \(f^{(3)}\), where the blue lines represent the result of the PQ (Pareto-Q-learning) algorithm, the orange are Q algorithm results and the gray are random results. In order to facilitate the comparison of different algorithms, the results are processed by a linear smoothing process, as shown by straight lines. The horizontal axis in the figures represents the time of the simulation calculation, and the vertical axis represents the corresponding game target value. The higher the data point on the straight line in the figure, the better the network availability and node security that the defender can obtain at the lower cost in the network attack and defense game. The results clearly show that the Pareto-Q-learning algorithm outperforms both the basic Q-learning algorithm and a random baseline. In fact, you can see that the Pareto-Q-learning approach outperforms from the very first iteration. This is in contrast with the Q-learning approach which, although it slowly improves over time, it clearly gives lower rewards than the combined approach. In Figures 8 and 9, you can see that over time simple Q-learning does approach the results of the Pareto-Q-learning implementation. It is expected that simple Q-learning will eventually reach the same solution as the Pareto-Q-learning algorithm. The initial advantage of the Pareto-Q-learning approach means that from the beginning the results are sufficiently optimal to allow network administrators to run the algorithm for fewer iterations than they would have to when using simple Q-learning. Both network availability (\(f^{(1)}\)) and node security (\(f^{(2)}\)) see an increase in cumulative reward. This is because the goals are completely independent. A move designed to increase the cumulative reward of the network availability has no effect on node security, and vice versa.