# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 5 Experiments  
This paper presents an adaptive noise detection mechanism that adjusts the noise level dynamically based on arm selection frequency. In this framework, the noise intensity decreases logarithmically as the number of rounds increases. The logarithmic decay is characterized by slow attenuation, enabling prolonged high levels of exploration. This helps reduce the risk of premature convergence to suboptimal solutions, which is especially important in complex or high-dimensional environments. To validate the framework's effectiveness, we conducted a small experiment using a simple synthetic dataset, comparing three noise strategies: fixed, decreasing, and adaptive noise. The average rewards for each arm were as follows: [0.2, \(-\)0.85, 1.55, 0.3, 1.2, \(-\)1.5, \(-\)0.2, \(-\)1.0, 0.9, \(-\)0.6]. As shown in Figure 3(a), the logarithmic adaptive noise strategy outperforms the other two strategies in terms of the proportion of optimal actions, suggesting that dynamically adjusting the noise more effectively aids in selecting high-reward arms. Furthermore, Figure 3(b) shows that the logarithmic adaptive noise strategy also performs well in terms of average reward [30,31].  
Next, we evaluate our algorithm on two recommendation and six classification datasets, comparing it against several benchmark contextual bandit algorithms, including the six most recent neural contextual multi-armed bandit algorithms. We first introduce the experimental setup, which is followed by a regret comparison. We run 15,000 rounds for all algorithms, repeat the experiment 5 times, and compare their average results.  
Figure 4: Effect comparison of different noise adjustment strategies in stochastic multi-armed bandits. (**a**) Optimal action proportion. (**b**) Average reward.  
**Baseline:** We selected four neural network-based bandit algorithms and one linear algorithm to evaluate our algorithm comprehensively.  
1. LinUCB [9]: This approach explicitly assumes a linear relationship between the reward and the arm's context vector. Unknown parameters are estimated using ridge regression, which is integrated with the UCB strategy to select the arm.
2. Neural-Epsilon [32]: Utilizes a neural network to learn the reward function of pulling an arm. It selects the arm with the highest predicted reward with a probability of \(1-e\), i.e., \(a_{t}=argmax_{i\in[k]}f(x_{a_{t}};\theta_{t-1})\), and randomly selects an arm with a probability of \(e\).
3. NeuralUCB [13]: A neural network is employed to learn the reward function from the exploration strategy based on UCB.
4. NeuralTS [14]: It uses a neural network to learn the reward function derived from the Thompson Sampling exploration strategy.
5. NPR [5]: Utilizes a neural network to learn the reward function by exploring through the addition of perturbed rewards.
6. DQN [33]: A reinforcement learning approach that combines deep learning and Q-learning leverages neural networks to approximate the Q-value function, enabling agents to efficiently learn optimal policies in high-dimensional state spaces.
7. EE-Net [22]: A dual neural network architecture is adopted. One neural network learns the real reward function, and the other neural network learns the potential reward value, and the architecture combines the output of the two to make a comprehensive decision, so as to improve the effect of exploration and utilization.