# Adaptive Noise Exploration for Neural Contextual Multi-Armed Bandits
## 1 Introduction  
The contextual Multi-Armed Bandit (MAB) framework is a critical model for sequential decision making under uncertainty with broad applications in online recommendation systems [1, 2] and clinical trials [3, 4]. Agents aim to maximize cumulative rewards through iterative interactions with the environment. Each round's observed reward depends on the chosen arm, and only the selected arm's reward can be observed. Thus, agents must identify the intrinsic relationship between arms and their rewards to optimize expected outcomes [5]. The main challenge in this domain is to strike an optimal balance between exploration and exploitation: exploration involves trying arms that must be sufficiently investigated. In contrast, exploitation focuses on selecting the best arm based on existing historical data [6, 7]. The Upper Confidence Bound (UCB) algorithm is a classic method for balancing exploration and exploitation [8]. The UCB algorithm assigns a confidence interval to each arm and uses the upper bound of this interval for subsequent selections. Specifically, the algorithm integrates both the historical average reward and the current uncertainty of each arm to balance exploration and exploitation. Most MAB algorithms, such as LinUCB, ConUCB, and KNNConUCB, assume a linear relationship between each arm's reward and its corresponding context vector [9, 10]. While these algorithms have achieved significant research results, this assumption may only partially capture the natural world's complex relationships between rewards andcontext vectors. To address this limitation, neural bandit algorithms utilize neural networks to capture intricate nonlinear relationships, enabling more accurate predictions of arm rewards [11; 12]. NeuralUCB [13] and NeuralITS [14] are two prominent algorithms in the neural bandit framework. NeuralUCB extends the traditional UCB algorithm by predicting rewards through neural networks and constructing confidence intervals based on the output gradient to select arms. NeuralITS (Neural Thompson Sampling) treats each arm as a normal distribution, with the mean being the neural network's output and the standard deviation calculated from the output's gradient, using the Thompson sampling strategy to select arms. These algorithms leverage the representational power of neural networks to handle complex nonlinear relationships [15; 16]. In neural bandit algorithms, UCB and TS-based methods require inverting the induced covariance matrix of numerous parameters to construct confidence intervals [13; 14; 17]. To reduce computational complexity, approximate methods are often employed in practice, such as using only the diagonal of the covariance matrix, although this diagonal approximation lacks theoretical justification [13]. The Neural Perturbed Rewards (NPR) algorithm replaces confidence interval calculation with noisy perturbed rewards for exploration. While this method increases randomness to facilitate exploration and reduces computational resource consumption, the fixed noise variance as a hyperparameter means that the exploration rate does not automatically adjust as the model learns [5]. Specifically, we propose an adaptive noise exploration framework that dynamically adjusts the noise level based on the selection frequency of each arm. We have developed two novel neural contextual multi-armed bandit algorithms within this framework: Exploring Adaptive Noise in Decision-Making Processes (EAD) and Exploring Adaptive Noise in Parameter Spaces (EAP). The EAD algorithm adds adaptive noise to the reward of each selected arm. It adjusts the noise based on the arm's selection frequency, achieving a finer balance between exploration and exploitation. The EAP algorithm introduces adaptive noise directly to the network parameters at the neural network's hidden layers. This approach results in more stable exploration and allows for a flexible adjustment of exploration intensity throughout the learning process. The principal contributions of this paper are summarized as follows:  
1.