# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 4 Proposed Method
### Exploring Adaptive Noise in Parameter Spaces  
In the neural bandit algorithm, besides introducing noise into the decision-making process to encourage exploration, another approach involves directly adding noise to the parameters \(\theta\) of the neural network. At the beginning of each iteration, noise is added to the parameters \(\theta\) to generate perturbed parameters \(\tilde{\theta}=\theta+\mathcal{N}(0,{\sigma_{a_{i}}}^{2}I)\), where \(\mathcal{N}(0,{\sigma_{a_{i}}^{2}}I)\) denotes a multivariate normal distribution with mean 0 and variance \({\sigma_{a_{i}}^{2}}\), and \(I\) is the identity matrix. This ensures noise is adaptively and independently added to each data point of \(\theta\) based on arm selection frequency. We have named this method Exploring Adaptive Noise in Parameter Spaces or EAP in short.  
Figure 3 illustrates the entire process of integrating adaptive noise within the parameter space. After selecting the arm with the highest predicted reward, the algorithm receives reward feedback \(r_{a_{i},t}\) and uses gradient descent to update neural network parameters based on the observed data \(\{X_{a_{i}},r_{a_{i},t}\}_{t=1}^{T}\). Unlike conventional approaches, this algorithm adds newly sampled noise to \(\theta\) post-update and continues training with these noise-enhanced parameters \(\tilde{\theta}\), facilitating a more stable and effective exploration.  
Figure 3: Adaptive noise exploration framework.  
Figure 2: Adaptive noise exploration framework.  
**Comparison with other algorithms:** Algorithms 1 and 2 detail the procedures for EAD and EAP, respectively. Unlike NeuralUCB, these algorithms do not use confidence intervals to select arms; instead, they rely solely on the neural network's output, determining the arm with the highest predicted reward. For exploration, the objective is achieved by adding noise. Unlike the fixed noise variance in the NPR algorithm, our algorithm dynamically adjusts the noise variance according to the frequency of arm selections. Our algorithm enhances exploration flexibility by adaptively adding noise to both the decision-making process and parameter space, thereby increasing exploration efficiency and reducing computational overhead.  
```
Input: Number of rounds T, regularization coefficient \(\lambda\), perturbation parameter \(\alpha\), \(\beta\) and \(\zeta\) network width \(m\), network depth \(L\) Initialization: \(\theta_{0}=[vec(W_{1}),...,vec(W_{L})]\in\mathbb{R}^{P}\) with Gaussian distribution: for \(1\leq l\leq L-1\), \(W_{l}=(W,0;0,W)\) with each entry of \(W\) sampled independently from \(\mathcal{N}(0,4/m)\); \(\textbf{W}_{L}=(\textbf{w}^{\top},-\textbf{w}^{\top})\) with each entry of \(w\) sampled independently from \(\mathcal{N}(0,2/m)\) for\(t=1,\ldots,T\)do  Pull arm \(a_{t}\) and receive reward \(r_{t,a_{t}}\), where \(a_{t}=argmax_{i\in[k]}f(x_{a_{t}};\theta_{t-1})\).  Compute \(\sigma_{a_{t}}=max\left(\alpha,min\left(\frac{\zeta}{\ln(L+1)},\beta\right)\right)\)  Generate \(\{\gamma_{a_{t}}\}\sim\mathcal{N}(0,{\sigma_{a_{t}}}^{2})\)  Set \(\theta_{t}\) by the output of gradient descent for solving \[\min\mathcal{L}_{\theta}=\sum_{t=1}^{T}(f(x_{a_{t}};\theta)-(r_{a_{t}}+\gamma _{a_{t}}))^{2}+m\lambda|\theta-\theta_{0}|_{2}^{2}\]  Pull arm \(a_{k}\) endfor
```  
**Algorithm 1** Exploring Adaptive Noise in Parameter Spaces (EAD)