# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 5 Experiments
### Experiments on the Recommendation Dataset
Hence, in this paper, we define exploration time as the total time the system spends from the selection of the first arm to the end of the round. Figure 7 shows the average exploration time of all slot machine models on two recommendation datasets. Since NeuralUCB and NeuralTS models require more time to build the covariance matrix based on historically selected arms, although the diagonal matrix  
Figure 5: Regret comparison on Yelp and Movielens. (**a**) Yelp. (**b**) Movielens. Figure 6: Learned parameters comparison on Yelp and Movielens. (**a**) Yelp. (**b**) Movielens. approximation can alleviate the computational burden, NeuralUCB and NeuralTS still need more time for exploration compared to noise-based exploration algorithms. In the experiment, we gradually increased the number of arms per round from 10 to 100. We observed that as the number of arms increased, the time difference between noise-based and covariance matrix-based exploration methods also gradually increased. The exploration time is shown in Figure 8.  
**Dynamic Environment Evaluation:** The experiment simulates a dynamic environment where users are randomly selected for interaction in each round. A Zipf distribution with parameter \(\alpha=1.5\) was applied to generate user visit frequencies, which were normalized to obtain the user visit probability distribution. Users were randomly selected in each round based on this weighted probability, simulating the real-world \"long-tail phenomenon\", where a few active users have high visit frequencies, while most users have low visit frequencies.This setup replicates the dynamic changes in user preferences observed in real-world scenarios. The algorithm operates in this dynamic environment, continuously making decisions and adjusting its strategy based on feedback [36]. As depicted in Figure 9, the experimental results demonstrated that noise-based exploration strategies outperformed confidence interval-based strategies. In the dynamic environment, both the EAD and EAP algorithms could adaptively adjust noise variance based on the frequency of arm selections, thereby enhancing exploration capabilities and facilitating quicker adaptation to changes in user preferences. This study underscores the significant advantage of our proposed adaptive noise mechanisms in dynamic environments, particularly in coping with frequent changes in user preferences. Figure 8: Effect of the number of candidate arm sets on exploration time. (**a,b**) Comparison of exploration times between the EAD and EAP algorithms and the NeuralUCB algorithm with arm numbers of 10, 20, 50, and 100: (**a**) Yelp. (**b**) Movielens. Figure 7: Exploration time comparison on Yelp and Movielens. (**a,b**) The average exploration time per round with 10 arms: (**a**) Yelp. (**b**) Movielens. **Analysis on Hyperparameter Sensitivity:** In the adaptive noise addition method we propose, we need to maintain the noise level within a specified range to prevent either too high or too low noise levels from negatively impacting the algorithm's performance. In the initial 1000 rounds, a grid search was conducted for parameters \(\alpha\), \(\beta\), and \(\zeta\), within the range of \(\left[10^{-4},1\right]\), with an increment of \(10^{-3}\). For each algorithm, the best hyperparameter combination was applied to a full simulation of 15,000 rounds with the trial continuing for the additional 14,000 rounds. To analyze parameter impacts on model performance, we explored a range from 0.0001 to 1 and visualized regret values in a heatmap (Figure 10). The heatmap shows the regret of EAD and EAP algorithms on the Yelp dataset under varying parameter configurations with the x-axis representing \(\beta\) (the upper bound of noise) and the y-axis showing its lower bound, which was set to 0.01 of \(\beta\). The heatmap analysis shows that selecting an appropriate parameter range is crucial for optimal performance. When the noise range is too large or too small, the algorithm fails to achieve optimal performance.