## 3 Problem Formulation
### State, Action, and Reward Function  
In this research, the state, action, and reward function associated with the penalty mechanism for the wireless backhaul capacity constraint in the considered RL-based joint power allocation and user association are defined as follows:  
* **State**: The state at the \(t_{th}\) time slot is defined as the user data rate in that time slot \[\mathbf{s}_{t}=[\rho_{1}(t),\ldots,\rho_{K}(t)].\] (6)
* **Action**: The action in the \(t_{th}\) time slot is defined as \[\mathbf{a}_{t}=\Big{[}\mathbf{c}^{ILE}(t),\mathbf{p}^{UE}(t)\Big{]},\] (7) where \(\mathbf{c}^{ILE}=[c_{j,k}]_{j=1:J,k=1:K}\), with \(c_{j,k}\in\{0,1\},j\in\mathcal{J},k\in\mathcal{K}\), and \(\mathbf{p}^{UE}(t)\) indicating the sets of user associations and power allocations, respectively. More specifically, the power allocation set is given by \(\mathbf{p}^{UE}(t)=\big{[}p_{1}^{ILE}(n),\ldots,p_{K}^{ILE}(n)\big{]}\), where \(p_{k}^{UE}(t)=[P_{S_{k},k},f(t)]_{f:f\in\mathcal{F}_{k}}\) is the vector of allocated power for data transmission in all subchannels assigned to UE \(k\) from its associated SBS \(\mathcal{S}_{k}\). * **Reward**: We aim to maximize the overall energy efficiency as in (4a) while maintaining QoS for each US and satisfying the backhaul link capacity constraint for each SBS. Hence, the reward \(r_{t}\) at the \(t_{th}\) time slot is defined as \[r(s_{t},a_{t})=\left\{ \begin{array}{ll}r^{{}^{\prime}}(s_{t},a_{t}),&\text{ if }\rho_{j}^{ SBS}\leq R_{j}^{ SBS},\forall j\in\mathcal{J},\\ r^{{}^{\prime}}(s_{t},a_{t})-r_{th},&\text{if }\rho_{j}^{ SBS}>R_{j}^{ SBS},\text{for some }j\in\mathcal{J},\end{array} \right.\] (8) where \[r^{{}^{\prime}}(s_{t},a_{t})\triangleq\lambda_{1}Z_{\kappa_{1}(t)}-\lambda_{2 }Z_{\kappa_{2}(t)},\] with \(\kappa_{1}(t)=\frac{1}{P_{T}}\sum_{k=1}^{K}\rho_{k}(t)\) being the system energy efficiency and \[\kappa_{2}(t)=\sum_{k=1}^{K}(\rho_{k}(t)-\
u_{k})^{2}\] being the penalty term which discourages the agent from taking the actions such that the capacity of each user deviates too much from the QoS threshold, and \(Z_{\kappa_{1}(t)}\) and \(Z_{\kappa_{2}(t)}\) are the Z-scores (i.e., standardized results) of \(\kappa_{1}(t)\) and \(\kappa_{2}(t)\), respectively. \(r_{th}\) is a threshold used to reduce the likelihood of violating the backhaul capacity constraint. One of the challenges when transforming a traditional optimization problem into a DRL problem is to devise proper handling of the constraints in the original optimization problem. In this paper, the penalty term in \(\kappa_{2}(t)\) is designed to improve the QoS satisfaction in constraint \(C_{3}\) through reducing the number of UEs whose achievable rates are much higher or lower than the capacity thresholds. The weights \(\lambda_{1},\lambda_{2}\in[0,1]\) control the significance of the corresponding term. Operators can tune the weights \(\lambda_{1}\) and \(\lambda_{2}\) according to their needs, e.g., setting \(\lambda_{1}>\lambda_{2}\) at off-peak traffic periods and \(\lambda_{1}<\lambda_{2}\) at peak traffic hours to enhance each user's service experiences. Note that the reward function defined in (8) may not be feasible in practice, and judicious setting for the associate weights \(\lambda_{1}\) and \(\lambda_{2}\) are needed based on trial-and-error efforts. Furthermore, in order to guide the agent to follow the backhaul link constraint, a penalty mechanism is introduced here for the agent to adjust corresponding actions. If one of the SBSs experiences a sum rate that violates the backhaul capacity constraint, the agent receives a penalty and restarts a new episode in the learning process. On the other hand, in this paper, the cluster size constraint is dealt with by including only the legitimate discrete actions, each of which allows no more than \(N_{sub}\) users in each cluster, in the _constrained_ discrete action space for the entire learning process. This guarantees the output of the discrete action satisfies the cluster size constraint.