## 5 Proposed Scheme  
As discussed in the previous Section, opting for a smaller value of CW results in more collisions. While opting for a large value of CW can result in excess downtime and additional delays. In both the cases, the channel will not be used effectively. Hence, it is important that the CW value should be adjusted, keeping in mind the actual level of contention in the channel. With the presence of a massive number of advanced network devices, CW optimization can now be examined using DRL methods. DRL is well suitable for the problem of optimizing wireless networks as it deals with smart software agents that take actions in a specific environment in order to maximize the reward [29]. Ref. [21] showed that DRL could be applied successfully to a CW optimization problem as both algorithms, DQN and DDPG, provided close to optimal efficiency. However, DDPG performs slightly better than the DQN. The main problem with Q-Learning is the slow pace of convergence owing to its iterative nature [30]. Moreover, it does not depend on any previous information while experiencing a new state. With DRL, a Deep Neural Network (DNN) estimates the state-action value function. The NN of Deep Q-Networks (DQN) is trained to reduce the prediction error caused by the loss function [31]. To further optimize DQN for the context of WLAN, we will use DDQN in this paper. We decided to use DDQN for two main reasons. * DQN estimates a set of strongly correlated values solved by DDQN
* DQNs tend to be overly optimistic. It will overreact to you being in this situation, even if it only happened due to a statistical error, and DDQN solves this problem. The overestimation problem means that the estimated value function in DQN is greater than the actual value function. The root of this problem lies in the Q-Learning maximization process. By calculating the target Q, the maximum value of Q is obtained in the next state. For authentic approaches, the specific action that maximizes the Q value is not always selected because real strategies are stochastic strategies. Thus, choosing the maximum Q value of the action here often results in a target value more significant than the actual value. DDQN extends DQN by reducing the overestimation of the Q function. DDQN suggests using two value functions that lead to two sets of weights to avoid overly optimistic reward estimates. One weight set is used to identify the action, while the other weight set is used to assess its reward [13]. In this way, DDQN enhances the strength and performance of the learned model. In order to deal with the limitations of CSMA/CA, we have proposed an algorithm named DRL Based Contention Window Optimization (DCWO) inspired by CCOD algorithm [21]. (see Algorithm 1). The proposed algorithm operates in three steps. 1. In the initial step, Wi-Fi is being controlled by the 802.11 standard. This phase evaluates the history of observed probability of collision in the network. 2. In the second step, selected DRL model, DDQN is trained by maximizing the reward. The agent makes the decisions with regards to the value of CW after the TRAIN Procedure for the proposed algorithm. Pre-processing consists of computing the mean as well as the standard deviation probabilities of observed collisions. For exploration, every action is revised with a noise factor that degrades during the training phase. For DDQN, noise is the possibility of exceeding the action of the agent with a random action. The final phase begins after the training, which is defined by a time duration defined by the user. Now, the agent is fully trained, and updates will no longer be received. 3. In the third phase, the DRL model is deployed in the network. Now the CW is updated via OPTIMIZE Procedure of our proposed algorithm. The flow chart of DCWO is shown in Figure 3. Applying the DRL algorithm also requires the setting of some key parameters. Performance of DRL is based on the reward discount \(\gamma\), which is linked to the significance of future rewards on immediate ones.