# XSQ-Learning: Adaptive Similarity Thresholds for Accelerated and Stable Q-Learning
## 4 Experiments and Results
### Results and Analysis
#### 4.2.2 Scenario 2: Moderate Complexity  
The quadratic expansion of the state space (_cell size_ = 2) revealed more pronounced advantages for _XSQ-Learning_. It achieved policy convergence at 87 test points on average (19,225 steps), outperforming _Q-Learning_, which required 27,600 steps (43.56% more) to reach the same result. Among the similarity-based variants (_QL-SmoothActions_, _QL-SmoothObservations_, and _QS-Learning_) _QL-SmoothActions_ showed the most competitive  
\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Algorithm** & **Scenario 1** & **Scenario 2** & **Scenario 3** \\ \hline Q-Learning & 8150 \(\pm\) 281.21 & 27,600 \(\pm\) 270.48 & 71,750 \(\pm\) 858.01 \\ QL-SmoothActions & 52,525 \(\pm\) 6621.68 & 28,200 \(\pm\) 2327.86 & 59,975 \(\pm\) 534.93 \\ QL-SmoothObservations & 80,000 \(\pm\) 0.00 & 80,000 \(\pm\) 0.00 & 80,000 \(\pm\) 0.00 \\ QS-Learning & 3725 \(\pm\) 3725.00 & 2900 \(\pm\) 2821.35 & 100 \(\pm\) 45.88 \\ XSQ-Learning & 6650 \(\pm\) 269.26 & 19,225 \(\pm\) 690.61 & 45,325 \(\pm\) 826.59 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Mean number of steps to reach the first stable maximum reward \(\pm\) standard error over 20 runs.  
\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Algorithm** & **Scenario 1** & **Scenario 2** & **Scenario 3** \\ \hline Q-Learning & 93,00 \(\pm\) 0.00 & 87.00 \(\pm\) 0.00 & 52.85 \(\pm\) 19.34 \\ QL-SmoothActions & \(-\)56.50 \(\pm\) 32.92 & 86.00 \(\pm\) 0.00 & 80.00 \(\pm\) 0.00 \\ QL-SmoothObservations & \(-\)200.00 \(\pm\) 0.00 & \(-\)200.00 \(\pm\) 0.00 & \(-\)200.00 \(\pm\) 0.00 \\ QS-Learning & \(-\)200.00 \(\pm\) 0.00 & \(-\)200.00 \(\pm\) 0.00 & \(-\)200.00 \(\pm\) 0.00 \\ XSQ-Learning & 93.00 \(\pm\) 0.00 & 87.00 \(\pm\) 0.00 & 80.70 \(\pm\) 0.11 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Mean first stable maximum reward \(\pm\) standard error over 20 runs.  
\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & & **Average** & **Average** & **Average** \\
**Algorithm** & **Scenario** & **Additional\(\%\)** & **First Stable** & **First Stable** \\  & & **Steps from** & **Maximum of the** & **Maximum of the** \\  & & _XSQ-Learning_ & **Algorithm** & **XQS-Learning** \\ \hline Q-Learning & 1 & 22.56 & 93.00 & 93.00 \\ _QL-SmoothActions_ & & 689.85 & \(-\)56.50 & 93.00 \\ _QL-SmoothObservations_ & & Not Applicable & \(-\)200.00 & 93.00 \\ _QS-Learning_ & & Not Applicable & \(-\)200.00 & 93.00 \\ _QL-Learning_ & 2 & 43.56 & 87.00 & 87.00 \\ _QL-SmoothActions_ & & 46.68 & 86.00 & 87.00 \\ _QL-SmoothObservations_ & & Not Applicable & \(-\)200.00 & 87.00 \\ _QS-Learning_ & & Not Applicable & \(-\)200.00 & 87.00 \\ _QL-SmoothActions_ & 3 & 58.30 & 52.85 & 80.70 \\ _QL-SmoothActions_ & & 32.32 & 80.00 & 80.70 \\ _QL-SmoothObservations_ & & Not Applicable & \(-\)200.00 & 80.70 \\ _QS-Learning_ & & Not Applicable & \(-\)200.00 & 80.70 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results comparison all algorithms vs. _XSQ-Learning_.  
performance, needing 46.68% more steps than _XSQ-Learning_ to reach an average of 86 test points. Once again, _QL-SmoothObservations_ and _QS-Learning_ failed to achieve meaningful learning, with no successful convergence in any of the 20 trials, highlighting the challenges associated with uncontrolled similarity propagation.  
Figure 2: Results from algorithm runs in a grid-world environment. (**a**) Scenario 1 (_cellsize_\(=1\)). (**b**) Scenario 2 (_cellsize_\(=2\)). (**c**) Scenario 3 (_cellsize_\(=3\)). Each shadow corresponds to \(\pm\)one standard deviation from observed cumulative rewards per episode per seed.