# XSQ-Learning: Adaptive Similarity Thresholds for Accelerated and Stable Q-Learning
###### Abstract
In contrast, QL-Smooth relies on fixed similarity thresholds that may be either too strict or too lenient. If the threshold is too low, useful information from similar states is ignored, slowing convergence. If it is too high, updates are propagated to states that are not meaningfully similar, causing policy instability. Such scenarios illustrate the need for adaptive mechanisms that can balance generalization and stability more effectively. Table 1 summarizes these limitations and highlights how our proposed XSQ-Learning algorithm addresses them. This concise comparison provides a clear overview of the key differences in adaptation mechanisms, computational complexity, and the ability to control generalization and interference. This paper presents _XSQ-Learning_, a novel algorithm that preserves the theoretical guarantees of standard Q-learning while overcoming its limitations by introducing three distinctive components: (1) adaptive similarity thresholds that adjust dynamically during learning, (2) update propagation weighted by both similarity and temporal-difference reliability, and (3) local similarity calibration through a residual term that ensures numerical stability. These components contribute jointly to controlling generalization and mitigating interference, addressing limitations observed in prior similarity-based algorithms. In our experiments, _XSQ-Learning_ required 36.83% fewer iterations than traditional Q-learning, while achieving 24.43% faster convergence than state-of-the-art similarity-based methods in complex environments. Notably, these improvements maintain policy stability, with our method showing consistent performance (\(\pm\)0.2% reward deviation) \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Algorithm** & \begin{tabular}{c} **Adaptive** \\ **Similarity** \\ **Thresholds** \\ \end{tabular} & \begin{tabular}{c} **Confidence** \\ **Weighted** \\ **Updates** \\ \end{tabular} & \begin{tabular}{c} **Clipping/** \\ **Stabilization** \\ \end{tabular} & \begin{tabular}{c} **Main** \\ **Limitation** \\ \end{tabular} \\ \hline Q-learning & No & No & No & Slow convergence in large spaces \\ \hline QL-Smooth & No & No & No & \
\begin{tabular}{c} Fixed thresholds, \\ over/under-propagate \\ \end{tabular} & may \\ \hline QSLearning & No & No & No &
\begin{tabular}{c} Propagates updates indiscriminately to all \\ similar pairs \\ \end{tabular} & \\ \hline
**XSQ-Learning** & **Yes** & **Yes** & **Yes** &
\begin{tabular}{c} **Controls** \\ **and interference** \\ \end{tabular} & **generalization** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of Q-learning, QL-Smooth, QSLearning, and the proposed XSQ-Learning. across multiple random seeds and environment configurations. The algorithm's relative advantage increases with problem complexity, demonstrating particular effectiveness in large state spaces where conventional methods struggle most. The remainder of this paper is organized as follows. Section 2 analyzes related work in Q-learning and similarity-based approaches. Section 3 details the _XSQ-Learning_ framework and its theoretical foundations. Section 4 presents our experimental methodology and results across progressively complex environments. Finally, Section 5 discusses broader implications and future research directions.