## 3 System Model Description and Problem Statement
### DQN Model for RAN-Level Slice Admission Control
#### 3.4.4 DQN Learning Process  
The DQN learning process refines the agent's policy for resource allocation by iteratively updating a deep neural network that approximates the optimal action-value function. At each time step \(t\), the agent interacts with the environment--whose state is continuously updated by the near-RT RIC--and collects experiences in the form of tuples \((s_{t},a_{t},R_{t},s_{t+1})\). The RIC aggregates real-time data on resource availability, active slice requests, and QoS metrics, which are used to construct the state vector \(s_{t}=(\text{SR},\text{RF})\).  
Upon observing \(s_{t}\), the agent selects an action \(a_{t}\) via an \(\epsilon\)-greedy policy: with probability \(\epsilon\), a random action is chosen (exploration), and with probability \(1-\epsilon\), the action with  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{3}{c}{**SR**} & & & \\ \hline
**RAN** & \multicolumn{3}{c}{**MEC**} & & **RF** \\ \hline BW & L & CPU & ST & RAM & B & \(0\mid 1\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: State vector of model 1.  
the highest predicted Q-value is selected (exploitation). The chosen action--whether corresponding to slice admission or rejection--is executed, resulting in a transition to a new state \(s_{t+1}\) and yielding a reward \(R_{t}\) that reflects whether the slice request was correctly processed.  
Each experience tuple is stored in a replay memory \(\mathcal{D}\), facilitating the random sampling of mini-batches during training. This mechanism decouples the learning process from the sequential correlations of real-time network events, thereby enhancing stability.  
The agent employs a deep neural network parameterized by \(\theta\) to approximate the Q-function, \(Q(s,a;\theta)\). A separate target network, with parameters \(\theta^{-}\), is maintained to generate stable target Q-values. The learning objective is to minimize the temporal difference (TD) error, as quantified by the following loss function:  
\[L(\theta)=\mathbb{E}_{(s_{t}a_{t},R_{t},s_{t+1})\sim\mathcal{D}}\Bigg{[} \bigg{(}R_{t}+\gamma\max_{a^{\prime}}Q(s_{t+1},a^{\prime};\theta^{-})-Q(s_{t}, a_{t};\theta)\bigg{)}^{2}\Bigg{]}, \tag{17}\]  
where \(\gamma\) is the discount factor. Minimization of this loss via a stochastic gradient descent iteratively updates the network parameters \(\theta\) to better approximate the true action-value function.  
To further enhance training stability, the target network parameters \(\theta^{-}\) are periodically synchronized with the current network parameters \(\theta\). This decoupling from the rapidly fluctuating estimates of the primary network mitigates the risk of divergence during learning.  
By continuously sampling from the replay memory and updating the network parameters, the DQN agent incrementally improves its approximation of the optimal Q-function. Convergence is typically indicated by a stabilization of the loss function and improved resource allocation performance. Consequently, the agent's decisions increasingly satisfy QoS requirements while optimizing network resource utilization. The integration of real-time network data provided by the RIC ensures that the learning process remains responsive to dynamic conditions, enabling adaptive and efficient resource allocation.