# Coexistence Scheme for Uncoordinated LTE and WiFi Networks Using Experience Replay Based Q-Learning
## 6 Performance Evaluation
### Convergence and Complexity Analysis  
In this section, we will discuss the convergence and complexity analysis of the coexistence schemes described in Section 5. We have described that all the schemes use the saturation state of WiFi, which is determined based on our previous work [8]. The WiFi saturation status is determined based on inter-frame statistics collected from wireless signals measured in 1 s. In this previous work, we have seen that the saturation status classification of WiFi traffic in a duration of 1s practically requires an end-to-end processing time of 1.032 s. This time is the sum of time required for capturing I/Q samples, pre-processing and classifying process (which requires 951 ms [26]), and time required to classify the saturation status of the WiFi network (which requires 81 ms [8]). The first coexistence scheme described in Section 5 is the rule-based solution. The rule-based solution is used to select an optimal value of \(N_{\text{b}}\) that leads to fairness based on Algorithm 1 for 20,000 randomly generated traffic load values of LTE-U and WiFi networks. The dynamic traffic loads are generated by running the 300 s simulation multiple times and varying the traffic based on the approach described in Section 6.1. In this scenario, the average value of \(N_{\text{t}}\) was found to be 2.68, i.e., an average of 2.68 actions are executed until fairness is achieved for each fixed traffic load. Using the rule-based approach, the total time required to report an LTE frame configuration in each trial is 1 s (wireless signal measurement) + 1.032 s (processing time of technology recognition classification + saturation sensing) + (rule-based decision \(-\) negligible processing time) = 2.032 s. As the best action requires 2.87 trials on average, the best action requires a period of 2.87 \(\times\) 2.042 s = 5.832 s. Basic challenges in using the rule-based approach include:  
* It is not convenient for co-located networks that have very dynamic traffic as determining the best action requires an average time of 5.832 s.
* Each trial is a new configuration and it affects the performance of the technologies as far as it is not the best one. i.e., even though the traffic remains constant for a long time, the best action is found after 5.832 s and the performance remains poor as wrong configurations are used in the meantime until the best action is not finally taken. * Rule-based solutions are not easily scalable if the number of possible actions is increased and this will further increase the time required to find the best action. The second coexistence scheme adopted for uncoordinated LTE-U and WiFi networks is the Q-learning based solution described in Section 5.2. The Q-learning parameters \(\eta\) and \(\gamma\) were set to 0.75 and 0.85, respectively. Similarly, the initial value of \(\varepsilon\) is set to 0.9, while \(N_{\varepsilon}\) is set to 500. After 500 iterations, the value of \(\varepsilon\) is reduced by a factor of \(f_{\varepsilon}\) = 90%, until a minimum value of \(\varepsilon_{\text{min}}\) = 0.05 is reached. These values were selected by an inspection as they lead to the best convergence. The convergence of the Q-learning based solution is shown in Figure 3. We can see that a large number of iterations (about 184,000) are necessary before the Q-matrix converges. This is because wireless network traffic is mostly dynamic, and state-action pairs are not consistent over time. Figure 3 also shows that using ER-based Q-learning can lead to faster convergence. The number of experiences per batch (\(N_{E}\)) was set to 500, while the number of experience replays (\(K\)) was set to 4. These values were also chosen by inspection because they lead to the best convergence, while the remaining hyper-parameters were left unchanged from the Q-learning based scheme. Figure 3 also shows that using ER of experiences with the best reward leads to even faster convergence. This is also expected as experiences with the best outcomes are selected and replayed instead of investigating every experience.