## 3 Problem Formulation
### Parameterized Deep Q Network  
Recent progress in deep RL (DRL) approaches has made the DRL, such as DQN, a viable technique to tackle various resource allocation problems in wireless networks. However, in order for DQN to be able to solve the joint power allocation and user association problem considered in this work, the continuous action space in the power allocation has to be quantized into discrete action space first. Quantization of the continuous action space may round off potentially optimal power allocations. Moreover, the complexity of the DQN increases exponentially with the dimension of the action space, leading to undesirable huge consumption of power and slowdown of convergence speed. To overcome this difficulty, in this paper, we propose employing the P-DQN [31; 34] for the joint power allocation and user association because of its capability of solving problems with hybrid action space. The parameterized action space is denoted by by \(\mathcal{A}^{PA}=\{(c,x_{c})|x_{c}\in\mathcal{A}_{c}\), for all \(c\in\mathcal{A}_{d}\}\), where \(\mathcal{A}_{c}\) and \(\mathcal{A}_{d}\) are the continuous and discrete action spaces, respectively. When the discrete action \(c\) takes all possible combinations into consideration without constraints, the discrete action space \(\mathcal{A}_{d}=\{[c_{k,j}]:c_{k,j}\in\{0,1\},k\in\mathcal{K},j\in\mathcal{J}\}\). Each discrete action \(c\) has a corresponding continuous parameter \(x_{c}\in\mathcal{X}_{c}\), where \(\mathcal{X}_{c}\) is the set of all users' power allocations \(\mathbf{p}^{UE}(t)\) in this work, for a discrete action \(c\). The primary network of the P-DQN (without the stabilizing target networks) is presented in Figure 2. The primary network in Figure 2 consists of an actor-parameter network \(x_{c}(s;\theta)\) with weights \(\theta\), which maps the state \(c\) and each discrete action to its corresponding continuous parameter, and an actor network \(Q(s,c,x_{c};\omega)\) with weights \(\omega\), which evaluates the action-value \(Q\)-function, i.e., the long term expected cumulative reward \(Q(s,a)\) defined in (5), and the action can be explicitly represented by the 2-tuple \(a=(c,x_{c})\) to emphasize the hybrid nature in the action. Typically, the weights \(\theta\) in the actor-parameter network can be determined by maximizing the expected action-value function \(E[Q(s,c,x_{c}(c;\theta);\omega)]\). Furthermore, the weights \(\omega\) in the actor network can be updated by minimizing the mean-squared error \(E[(y_{t}-Q(s_{t},a_{t};\omega))^{2}]\), where \(y_{n}\) is the target value in the network and \(a_{t}=(c(t),x_{c}(t))\)[28; 31]. In order to stabilize the P-DQN, an additional _target network_, combining the original primary network shown in Figure 2, is built to produce the target value \(y_{t}\) needed in the actor network [28]. Furthermore, to expedite the training process, an experience replay buffer \(\mathcal{D}\) is implemented to provide random samples for evaluating the means appearing in the loss functions of both the actor network and actor-parameter network. With the replay buffer, the loss functions for the actor-parameter \(x_{c}(s;\theta)\) and the actor \(Q(s,c,x_{c};\omega)\) can be obtained by the following sample means:  
\[L^{x}(\theta) =-\frac{1}{N}\sum_{i=1}^{N}Q(s_{i},c_{i},x_{c_{i}}(s_{i};\theta);\omega)\] \[L^{Q}(\omega) =\frac{1}{N}\sum_{i=1}^{N}\big{(}y_{i}-Q(s_{i},c_{i},x_{c_{i}}(s_{ i};\theta);\omega)\big{)}^{2},\]  
where \(y_{i}=r_{i}+\gamma\max_{c^{{}^{\prime}}}Q\Big{(}s_{i+1},c^{{}^{\prime}},x_{c^ {{}^{\prime}}}(s_{i+1};\theta^{-});\omega^{-}\Big{)}\) is evaluated by the _target network_ for stability with weights \(\theta^{-}\) and \(\omega^{-}\), \((s_{i},c_{i},x_{c_{i}},r_{i},s_{i+1})\in\mathcal{D}\) is sampled from the replay buffer, and \(N\) is the size of the mini-batch (i.e., sample size).