## 1 Introduction
### Prior Work  
In order to intelligently manage the interwoven dynamics underlying the wireless sensor or mobile networks in which a variety of network parameters are generally unknown, deep reinforcement learning (DRL)-based approaches have been applied to tackle the challenges of radio resource management in wireless networks, e.g., [11; 16; 18; 19; 20; 21; 22; 23; 24; 25; 26; 27], due to DRL's ability to extract features from raw data, learn complex correlations generated by the mobile environment, and make sequential decisions through interactions with the environment without knowledge of complete environment information. In the applications of power allocation, the objectives can often be categorized into three types: capacity maximization in [11; 19; 20], energy saving in [16], and maximization of capacity for consumed energy (which is defined as the energy efficiency) in [21; 22; 23; 24].  
[PERSON] et al. [11] propose several DL-based algorithms to handle the power allocation with the aim to maximize the sum rate in multiuser wireless cellular networks, in which the DL-based data-driven approaches are demonstrated to outperform the traditional model-based power allocation method. [PERSON] and [PERSON] [16] utilize multi-agent deep RL to adaptively control the discrete power level (i.e., the range of possible power values is quantized into a number of discrete levels) for each user where the policy can be made without requiring to know the instantaneous CSI. [PERSON] and [PERSON] [18] tackle the problem of discrete power allocation and mode selection in device-to-device (D2D) communication networks using DQN with energy efficiency as the reward in the learning process. [PERSON] et al. [19] apply the cooperative Q-learning for power control at discrete levels, but the effect of channel variations is not considered. [PERSON] and [PERSON] [20] employ deep Q-learning, which was originally proposed in [28], to update the transmission power allocated for each user at the small cell base station in HetNet, in which the power is quantized into discrete levels. [PERSON] et al. [21] present a novel DRL-based technique for resource allocation by considering power efficiency in cloud radio access networks (RANs) and ensuring QoS guarantee. [PERSON] et al. [22] propose a DRL-DPT framework, in which the agent learns directly from the expected objective instead of critic value, for energy efficiency maximization without explicit simulation results for QoS guarantee. [PERSON] et al. determine the number of users and subchannels with corresponding power allocation in HetNets using a policy-gradient based actor-critic learning algorithm [23]. Instead of quantizing the power into discrete levels, [PERSON] et al. [11] model the power as a continuous action and adaptively update the continuous power using the deep deterministic policy gradient-based (DDPG-based) reinforcement learning [29; 30]. While the above-mentioned research successfully apply the DRL-based techniques to power allocation in heterogenous networks, the problem of user association is not jointly considered, and various practical constraints in the network are not accounted for, such as the limited backhaul capacity [13] in each small cell base station. A novel energy-efficient joint power allocation and user association using deep reinforcement learning is studied by [PERSON] et al. in [24], where the power is considered as belonging to a discrete set and the learning process is not bounded by any constraints. A summary of the related work is provided in Table 1.