## 3 Proposed System Model
### Proposed Methodology and Q-Value Updates  
In DQN, the method for updating Q-values differs slightly from traditional approaches because it utilizes a neural network to estimate values rather than storing them in a table manually. DQN combines Q-learning with a deep neural network to manage environments with large or continuous state spaces. The neural network in DQN approximates Q-values, and the updating process resembles that of traditional Q-learning but employs a neural network to calculate the Q-values. The update mechanism in DQN typically adheres to the Bellman equation, with each node acting as an agent equipped with a Q-table. Initially, the Q-values for each state-action pair are set to zero, which, in turn, depends on multiple factors outlined in the subsequent text and illustrated through the flowchart in Figure 4. The Q-values are then updated based on observed rewards and future expectations. The Q-value update rule is given by Equation (16):  
\[Q(S,A)\gets Q(S,A)+\alpha\bigg{(}R+\gamma\max_{A^{\prime}}Q(S^{\prime},A^{ \prime})-Q(S,A)\bigg{)}. \tag{16}\]  
where:  
* \(Q(S,A)\) is the Q-value for the state-action pair \((S,A)\). * \(R\) is the immediate reward received after taking action \(A\) in state \(S\). * \(\gamma\) is the discount factor that determines the weight given to future rewards. It ranges from 0 to 1. * \(\alpha\) is the learning rate, which controls the learning speed and ranges from 0 to 1. * \(\max_{A^{\prime}}Q(S^{\prime},A^{\prime})\) represents the maximum expected future reward for the next state \(S^{\prime}\), over all possible actions \(A^{\prime}\). In reinforcement learning, the balance between exploration, which involves trying new actions to discover potentially better strategies, and exploitation, which focuses on selecting actions that have already yielded the highest rewards, is maintained using an epsilon-greedy policy. The agent selects an action \(A_{t}^{*}\) based on the following rule as depicted in Equation (17):  
\[A_{t}^{*}= \begin{cases}\text{random action},&\text{with probability }\epsilon\\ \arg\max_{a}Q(s_{t},a),&\text{with probability }1-\epsilon\end{cases} . \tag{17}\]  
where:  
* \(A_{t}^{*}\) is the action selected by the agent at time \(t\),
* \(Q(s_{t},a)\) is the Q-value of action \(a\) in state \(s_{t}\),
* \(e\) is the probability of choosing a random action (exploration),
* \(1-\epsilon\) is the probability of selecting the action that maximizes the Q-value (exploitation). This strategy ensures that the agent explores new possibilities while also exploiting its learned knowledge to maximize performance. As the agent interacts with the environment, it stores experience tuples \((s_{t},a_{t},r_{t+1},s_{t+1})\) in a replay buffer. These tuples consist of the current state \(s_{t}\), the action \(a_{t}\) taken, the reward \(r_{t+1}\) received, and the next state \(s_{t+1}\). During training, the agent randomly samples mini-batches from the replay buffer to update the Q-network. DQN uses two networks: the Q-network, which is actively updated during training, and the target Q-network, a copy of the Q-network that is updated less frequently. The target Q-network helps compute the target Q-value for the Q-learning update. The target Q-value is given by Equation (18):  
\[Q_{\text{target}}=r_{t+1}+\gamma\max_{a^{\prime}}Q^{\prime}(s_{t+1},a^{\prime }). \tag{18}\]  
where:  
* \(r_{t+1}\) is the reward received after taking action \(a_{t}\) in state \(s_{t}\),
* \(\gamma\) is the discount factor, determining the importance of future rewards,
* \(\max_{a^{\prime}}Q^{\prime}(s_{t+1},a^{\prime})\) is the maximum Q-value predicted by the target network for the next state \(s_{t+1}\). At each state, agents use a decision policy based on the learned Q-values to choose an action. The agent picks an optimal action and moves to a new state based on the Q-values cached in the Q-table. The decision policy is critical for determining how the agent acts  
Figure 4: Q-value updates leveraging the Bellman equation for the proposed DQN based framework.