## 3 Proposed Work
### Deep Contextual Bandit Network Slicing Scheme  
Reinforcement learning (RL) methods facilitate agent learning through evaluative feedback. An action performed by an agent is evaluated based on its quality, for which it receives a corresponding reward. In addition, RL agents attempt to solve a problem by balancing exploitation and exploration of actions. This process, in turn, allows the agent to learn a strategy to obtain an optimal reward for each action. Additionally, the way RL agents learn is in direct contrast to other AI methods that guide the agent to take corrective actions based on existing training data. This work implements a variant of the multi-armed (or \(k\)-armed) bandit approach known as Contextual Bandits [30] for the proposed NS scheme. This approach is a type of associative search method in which the current state of the environment affects an agent's reward for a given action. A famous example of this scenario is the case of slot machines in a casino, where a player tries to get the best payoff while playing slot machines. In this scenario, the rewards for pulling the arm of a slot machine (or bandit) are randomly distributed and are unknown to the player, so the player must choose the best action strategy to get the best payoff. Therefore, we model the proposed scheme based on a Contextual Bandit scenario. Moreover, a deep reinforcement learning is utilized to compute the reward and formulate the action strategy efficiently through a trial-and-error approach. The deep neural network receives the NSR parameters as through its input layer and then outputs it through the output layer. The rewards are calculated based on the network's current state and checked for errors or losses using the mean squared error method. Such an approach leads to an optimal resource allocation strategy in the form of a network slice for each NSR. A graphical representation of the Deep RL model is shown in Figure 2.  
Figure 2: Deep Reinforcement Learning Model for the proposed work. The proposed scheme considers an input \(\eta\), and a set of actions \(A\), and each action \(a\in A\) has an associated reward \(R_{t}\). In this case, \(NSR_{i,t}\) and the mapped nodes \(\psi(n_{\alpha},n_{\Omega})_{NSR_{i,t}}\) represent the input and the agent's actions, respectively. For every action taken, the agent expects to achieve a value or utility \(q_{*}(a)\) that measures the quality of such action. This value or utility calculation is expressed as,  
\[q_{*}(a)=E[R_{t}|A_{t}=a] \tag{10}\]  
thus, the expected value \(q_{*}(a)\) for action \(a\) is equivalent to the reward for that action at time \(t\). In addition, the actions in the agent's action space are randomly distributed using a Softmax function, which is denoted as,  
\[\sigma(z)_{i}=\frac{exp^{z_{i}}}{\sum\limits_{j=1}^{K}exp^{z_{j}}} \tag{11}\]  
where the probability of choosing an action is calculated by dividing the input vector's exponential function \(exp^{z_{i}}\) over the sum of all the exponential functions for an output vector \(exp^{z_{j}}\) for a \(K\) number of classes. Moreover, this probability distribution method enables the NSA to randomly select an action out of a given set of possible actions without any bias. Since the Contextual Bandit approach considers the environment state to determine the reward for an action taken, it is also necessary to define state \(S\) as a set of states where each state \(s\in S\) comprises various reward influencing factors (RIF). The various factors are defined as follows:  
1. The total path length \(L_{hops}\) is the sum of all paths from \(n_{\alpha}\) to \(n_{\Omega}\). This value provides the agent with the candidate slice topology information. \[L_{hops}=\sum\limits_{i=0}^{n_{\Omega}}\psi(n_{i})\] (12)
2. The computing capacity utilization denoted as \(C_{\delta}\) provides information regarding the computing capacity \(\sum\limits_{i=1}^{I}C_{NSR_{i}}\) allocated for \(NSR_{i}\) at time \(t\).