# Reinforcement Learning Based Resource Management for Network Slicing
## 5 Dynamic Resource Adjustment Algorithm  
Based on the model presented in the previous section, we propose a dynamic resource adjustment algorithm that aims at maximizing slice tenant's profit by learning with the reward resulting from the previous action while satisfying the requirements of flows. Our adjustment algorithm is based on the Q-learning approach. Q-learning is one of the most widely used reinforcement learning techniques, which finds and selects an optimal action based on Q-function that defines state-action pairs. A Q-value represents the expected long-term reward of an agent under the assumption that it chooses the best actions from a specific state. First, the agent perceives the current states of the system and takes an action according to policy \(\pi\) defined as  
\[\pi=\max_{a\in A}Q(s_{t},a_{t}), \tag{13}\]  
where \(Q(s_{t},a_{t})\) is the Q-value, and the system keeps a memory for all state-action pairs. If the action \(a_{t}\) that has the maximum Q-value at the current state is selected by the policy, the reward is calculated according to Equation (11), leading to updating the Q-value. If the agent infinitely visits all state-action pairs, Q-values converge to an optimal value. However, always taking the best action with the maximum Q-value at every state (i.e., exploitation) may result in a wrong decision after the completion of learning. The agent should be exposed to as many states as possible to learn how to deal optimally with all of them. Thus, the algorithm also needs to explore all possible \((s_{t},a_{t})\) options, including those that do not maximize the Q-value (i.e., exploration). The exploration strategy involves performing random actions to experience environmental changes to preclude from focusing on immediate rewards, whereas the exploitation strategy makes use of only what the agent knows already. We adopt the \(\epsilon\)-greedy method to establish the exploration and exploitation balance. It means that the agent exploits the action that minimizes the immediate reward \((1-\epsilon)\) and explores random actions with a probability \(\epsilon\). Therefore, the probability of selecting action \(a_{t}\) in state \(s_{t}\) under policy \(\pi\), denoted as \(P_{\pi}(s_{t},a_{t})\), can be represented as  
\[\forall a_{t}\in A,\ P_{\pi}(s_{t},a_{t})= \begin{cases}1-\epsilon+\dfrac{ \epsilon}{|A|},&\text{if }a_{t}=A^{*},\\ \dfrac{\epsilon}{|A|},&\text{if }a_{t}\
eq A^{*}.\end{cases}  \tag{14}\]  
To maximize the overall reward of the agent, the algorithm constantly learns the characteristics of the system by exploring it and taking decisions many times. The action to be taken in the current state depends on the Q-value \(Q(s_{t},a_{t})\). The learning process continuously updates Q-values until the system can take the best action for all states. The agent updates the action-value function according to the following rule:  
\[Q(s_{t},a_{t})\leftarrow(1-\alpha)Q(s_{t},a_{t})+\alpha\bigg{\{}v(s_{t},a_{t}) +\gamma\max_{a\in A}Q(s_{t+1},a)\bigg{\}}, \tag{15}\]  
where \(0\leq\alpha\leq 1\) denotes the learning rate that determines how fast the learning happens, whereas the discount factor \(0\leq\gamma\leq 1\) weighs the importance of future rewards in making the current decision. When \(\alpha\) is small, it will take a long time for the algorithm to converge, whereas, with a large \(\alpha\), the algorithm might not converge. Setting \(\gamma\) to a small value results in the Q-value being optimized by immediate rewards, whereas, with large \(\gamma\) values, the algorithm heavily depends on future rewards. An appropriate selection of \(\alpha\) and \(\gamma\) can allow achieving effective control of the Q-learning process. The pseudo-code of the proposed resource adjustment algorithm is shown in Algorithm 1. The algorithm maintains the Q-values that are being iteratively updated when new events happen. First, the parameters for Q-learning are initialized. In particular, the Q-values are initialized to zeros (Lines 1-2).