# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 4 Proposed Method  
NeuralUCB employs neural networks to estimate the expected reward and uncertainty of each action. The formula for calculating the confidence interval is given by  
\[\text{U}=f(x_{a_{t}};\theta)+\gamma_{t-1}\sqrt{g(X_{a_{t}};\theta_{t-1})g(X_{ a_{t}};\theta_{t-1})^{T}/m} \tag{4}\]  
where \(g(X_{a_{t}};\theta_{t-1})=\
abla_{\theta}f(x_{a_{t}};\theta)\in\mathbb{R}^{p}\). Since NeuralUCB exploration covers the entire network parameter space, the dimensionality involved is \(\mathbb{R}^{p+d}\)[29]. Although this method is theoretically effective, its computational cost is extremely high in high-dimensional feature spaces or large datasets. To reduce the computational burden and improve efficiency, NPR introduces a randomization strategy in neural network updates, demonstrating that the added noise \(\gamma_{t}\) leads to optimistic estimates of rewards [5] with the model's predicted values \(f(x_{a_{t}};\theta)\) having a \(1-\delta\) probability of falling within the error term \(\epsilon(m)\) of the actual rewards \(h(x_{a_{t}})\) in each round \(t\in T\). The formula is as follows:  
\[f(x_{a_{t}};\theta)\geq h(x_{a_{t}})-\epsilon(m) \tag{5}\]  
In this context, \(\delta\) is generally a small constant. This noisy reward perturbation ensures that the model makes optimistic estimates of rewards with high probability, thereby encouraging sufficient exploration. Building on this, we introduce a dynamic noise strategy to facilitate more effective prospecting and reduce dependence on traditional confidence interval estimates. In the multi-armed bandit model, consistent exploration-exploitation interactions indicate that for arms selected frequently, their underlying reward distributions are sufficiently explored, and thus the exploration level should be reduced by injecting less noise during reselection. Conversely, if an arm is chosen infrequently, it signals insufficient exploration, requiring an increase in noise injection to encourage further exploration. To address this, we propose a formula that dynamically adjusts the noise intensity:  
\[\sigma_{a_{t}}=max\bigg{(}\alpha,min\bigg{(}\frac{\zeta}{\text{ln}(t_{k}+1)}, \beta\bigg{)}\bigg{)} \tag{6}\]  
The formula dynamically adjusts the noise intensity \(\sigma_{a_{t}}\) for each arm to balance the trade-off between exploration and exploitation. Specifically, the maximum term \(max(\alpha,\cdot)\) ensures that the noise intensity \(\sigma_{a_{t}}\) does not fall below a minimum threshold \(\alpha\), thus preventing the noise from becoming too small and avoiding insufficient exploration. The minimum term \(\min\Big{(}\frac{\zeta}{\text{ln}(t_{k}+1)},\beta\Big{)}\) ensures that the noise intensity decreases as the selection count \(t_{k}\) for a specific arm increases but does not fall below \(\beta\). Here, \(\beta\) represents the upper bound of the noise scale, preventing the noise from becoming excessively large and compromising the model's stability. The term \(\frac{\zeta}{\ln\left(h_{t}+1\right)}\) is designed to ensure that the noise scale decreases at a logarithmic rate as \(t_{k}\) increases. Compared to other decay forms, such as polynomial or exponential decay, logarithmic decay was chosen primarily for its characteristic of slow attenuation. Logarithmic decay causes the noise scale to decrease gradually with increasing time steps but at a slower rate. This slower decay helps maintain a relatively high level of exploration over a longer period, which is particularly beneficial in reducing the risk of premature convergence to suboptimal solutions. This advantage becomes especially significant in complex or high-dimensional environments, where sufficient exploration is critical for the algorithm to discover and exploit globally optimal solutions. Figure 1 illustrates the adaptive noise exploration framework we propose. After a user selects an arm using the algorithm, the selection count of that arm is reviewed to calculate the variance of the input noise.