## 3 Materials and Method
### Proposed Method
#### 3.3.3 Sparse-Feedback-Enhancement Module  
In large-scale data environments, behavioral feedback often exhibits high levels of sparsity and heterogeneity, particularly under conditions in which data recording is constrained by uneven user attention, short sampling intervals, or asynchronous event timing. These factors frequently result in missing or delayed supervisory signals, which can significantly undermine the accuracy of value estimation during training, amplify gradient propagation errors, and ultimately lead to policy degradation or convergence to suboptimal solutions. This issue critically impairs the model's generalization and stability in complex environments. To address this, a sparse-feedback-enhancement module (reward denoising module) has been designed to improve robustness, generalizability, and exploratory capabilities under sparse-feedback conditions. This module integrates data perturbation augmentation, temporal masking, state-action clustering, and weighted return smoothing as core mechanisms, as shown in Figure 3.  
The first component, data perturbation augmentation, includes adversarial noise injection and Gaussian noise perturbation to simulate uncertainty in market feedback by generating perturbed samples. Adversarial noise is computed using the Fast Gradient Sign Method (FGSM), applying perturbation \(\delta=\epsilon\cdot\text{sign}(\
abla_{x}\mathcal{L})\) to the input state features \(x\), thereby yielding perturbed samples \(x^{\prime}=x+\delta\). Gaussian noise sampled from \(\mathcal{N}(0,\sigma^{2})\) is  
Figure 3: Illustration of the sparse-feedback-enhancement module, which enhances policy robustness through data augmentation, clustering-based reward smoothing, adaptive noise scaling, and dynamic feedback integration. added to critical variables such as price and inventory to generate boundary-near training instances. To further enhance robustness to partial temporal observation, a temporal masking mechanism is employed, randomly masking a proportion of time steps \(t\) in the state sequence. This forces the model to infer action values from incomplete observations, thereby improving tolerance to missing data. The second component, clustering and reward smoothing, performs joint clustering of high-dimensional state-action pairs \((s_{t},a_{t})\) using the K-means++ algorithm. Within each cluster \(C_{i}\), a weighted average reward \(\tilde{r}_{t}\) is constructed as the smoothed feedback signal, calculated as  
\[\tilde{r}_{t}=\frac{\sum j\in\mathcal{N}(t)\omega_{j}r_{j}}{\sum_{j\in \mathcal{N}(t)}\omega_{j}}, \tag{24}\]  
where \(\mathcal{N}(t)\) denotes the KNN of \((s_{t},a_{t})\) in the embedding space and \(\omega_{j}\) is a distance-based weight defined by \(\omega_{j}=\exp(-\alpha|z_{t}-z_{j}|^{2})\), where \(z_{t}\) is the embedding of the state and \(\alpha\) is a decay coefficient. A policy confidence check mechanism is further incorporated, executing smoothing only when the output confidence \(\pi(a_{t}|s_{t})\) exceeds a predefined threshold, preventing over-reliance on noisy data during early exploration stages and preserving training validity. During training, the original reward \(r_{t}\) is fused with the smoothed reward \(\tilde{r}_{t}\) to form an enhanced reward signal \(r_{t}^{\text{aug}}\), computed as  
\[r_{t}^{\text{aug}}=\lambda\cdot r_{t}+(1-\lambda)\cdot\tilde{r}_{t}. \tag{25}\]  
The weighting coefficient \(\lambda\in[0,1]\) is adaptively adjusted based on the clustering density, confidence, and loss dynamics. A higher weight is assigned to \(\tilde{r}_{t}\) in regions of high density and low confidence to mitigate bias introduced by skewed samples. Additionally, an ensemble strategy aggregation mechanism is introduced to improve generalization, whereby policy outputs under multiple perturbation and clustering instances are aggregated through behavior-consistency weighting. This yields a more robust policy estimation and prevents training drift caused by individual perturbations.