# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 5 Experiments
### Experiments on the Recommendation Dataset  
**Dataset Introduction:** The Yelp dataset, released by Yelp as part of a data challenge, includes 4.7 million ratings from 1.18 million users for 157,000 restaurants [34]. The MovieLens dataset, on the other hand, contains approximately 25 million ratings from about 160,000 users for approximately 60,000 movies [35]. In these datasets, restaurants and movies are considered arms. We selected the 2000 most active users and the top 10,000 highest-rated arms. SVD was then used to extract 10-dimensional feature vectors for each user and arm from the rating data. In this study, the algorithm aims to select arms with lower ratings. Specifically, the algorithm's rewards are based on the star ratings users give to arms. In each round, 10 arms are set up with 1 arm randomly selected for a reward of 1 and the remaining 9 arms selected to have a reward of 0. The context for each arm is the concatenation of the corresponding user's feature vector and the arm's feature vector. **Cumulative regret comparison:** In Figure 5, we compare the cumulative regrets of various algorithms, where the arms are implemented by randomly drawing from a candidate pool. The results show that the two neural bandit learning algorithms, EAD and EAP, expanded under our proposed Adaptive Dynamic Noise Exploration framework, successfully learned the mapping of potential nonlinear reward functions on two real-world datasets. In contrast, the LinUCB algorithm, relying on simple linear reward functions, failed to capture the complex reward structures present in real-world datasets effectively. Meanwhile, the Neural-Epsilon and NPR algorithms face challenges in compelling exploration due to their fixed exploration probabilities, which are difficult to adjust to complex environments. In our framework, the exploration mechanism can dynamically change the noise level based on the frequency of arm selection, thus more flexibly adapting to complex environments. **Accuracy of learned parameters:** Next, we compared the parameter estimation accuracy among six different neural bandit algorithms, including our own. Precisely, we measure the performance of each algorithm by calculating the average Euclidean distance between the learned parameter vector \(\theta_{u,t}\) and the real parameter vector \(\theta_{u,t}\), using the formula \(|\theta_{u,t}-\theta_{u,}|_{2}\). Here, \(\theta_{u,t}\) represents the estimated parameter vector of user \(u\) after each iteration, and \(\theta_{u,t}\) is the actual parameter vector obtained by averaging the results of five experiments on each dataset. A smaller average Euclidean distance indicates greater accuracy in parameter learning. In Figure 6, we display the changes in average distance every 50 iterations. The results show that across all the neural bandit algorithms examined, the average Euclidean distance gradually decreases as the number of iterations increases. This indicates that by increasing user interactions, all algorithms can gradually improve the accuracy of parameter learning. Notably, our EAP algorithm significantly outperforms the other algorithms examined regarding parameter learning accuracy. **Comparison of exploration time:** The central challenge in the context-aware multi-armed bandit problem is finding the right balance between exploration and exploitation. Excessive exploration could cause the system to spend too much time on new actions and need more on utilizing available information, thereby diminishing long-term rewards. On the other hand, excessive exploitation might lead the system to prematurely concentrate on a few known high-return actions, overlooking other potential high-return opportunities and resulting in suboptimal long-term rewards. Our algorithm dynamically balances exploration and exploitation, automatically computing this based on how frequently an arm is selected, eliminating the need to define exploration time explicitly.