## 3 Preliminaries
### Dqn  
RL [42] is a branch of machine learning that aims to instruct agents to make optimal decisions by interacting with their environment. Rather than being given clear instructions, in RL, an agent learns to make decisions by interacting with an environment. The agent receives feedback in the form of rewards based on the actions it takes in different states of the environment as depicted in Figure 2.  
The core components of every RL-based system include the following:  
**Agent:** The decision-maker or learner who interacts with the environment and makes decisions based on its current state. The ultimate goal of the agent is to optimize its actions to maximize cumulative rewards within the given environment. **Environment:** The system outside of the agent with which it communicates, the environment, gives feedback to the agent based on the actions taken; this feedback typically comes in the form of rewards or penalties. **State:** This is a representation of the current condition in the environment at a specific time. The agent relies on the state to inform its decision-making process and select appropriate actions. **Action:** This is a decision or choice made by the agent in the environment. The choices of actions are guided by the agent's policy, which is a set of guidelines and rules. The agent continuously improves its policy through learning to make better decisions over time. **Reward:** In the realm of RL, the concept of reward holds the utmost significance. It serves as a pivotal motivator for agents, compelling them to converge towards an optimal policy \(\pi\) by reinforcing favorable actions, which are approximated through a specific function. The primary objective of an agent within this framework is to discern an optimal behavioral policy. This optimal policy aims to maximize the expected long-term discounted reward, denoted as \(R_{t}=\sum_{i=t}^{T}\gamma^{i-t}r_{i}\), where \(T\) signifies the termination step, \(R_{t}\) represents the total discounted reward from time-step \(t\) until the termination step \(T\), \(r_{i}\) represents the immediate reward at time-step \(i\), and \(\gamma\in[0,1]\) represents the discount factor. This discount factor strategically balances the importance of immediate rewards against those in the future. One of the prominent algorithms in RL is the Deep Q Network (DQN) [43]. It combines deep neural networks with Q-learning to learn the optimal policy for a given task. Q-learning is a method that uses a Q-table of state-action values, also called Q-values [44]. DQN replaces the Q-table with a deep neural network to represent the Q-function, which allows it to learn complex relationships between states and actions. DQN typically uses two neural networks: a main network and a target network. The main network is used to estimate the Q-values for the current state and action. It is a standard neural network that takes the current state as input and outputs the Q-values for all possible actions. The Q-values are then used to select the action with the highest expected reward, as denoted in Equation (2), where \(Q_{main}\) denotes the weight matrix of the main network, and \(\phi(s)\) is the feature vector representation of state \(s\). While the target network is used to estimate the Q-values for the next state, it is a copy of the main network, and its weights are updated  
Figure 2: An agent takes action \(a_{t}\in A\) in the environment, receiving reward \(r_{t}\in R\) and transitioning to the next state \(s_{t+1}\) based on the actions taken. less frequently as denoted in Equation (3), where \(Q_{target}\) signifies the weight matrix of the target network and \(s^{\prime}\) represents the feature vector representation of state \(s^{\prime}\):  
\[Q_{main}(s,a)=W_{main}.\phi(s) \tag{2}\]  
\[Q_{target}\big{(}s^{\prime},a)=W_{target}.\phi\big{(}s^{\prime}\big{)} \tag{3}\]  
The weight parameters of the main network are duplicated in the target network.