###### Abstract  
Mobile communications have experienced exponential growth both in connectivity and multimedia traffic in recent years. To support this tremendous growth, device-to-device (D2D) communications play a significant role in 5G and beyond 5G networks. However, enabling D2D communications in an underlay, heterogeneous cellular network poses two major challenges. First, interference management between D2D and cellular users directly affects a system's performance. Second, achieving an acceptable level of link quality for both D2D and cellular networks is necessary. An optimum resource allocation is required to mitigate the interference and improve a system's performance. In this paper, we provide a solution to interference management with an acceptable quality of services (QoS). To this end, we propose a machine learning-based resource allocation method to maximize throughput and achieve minimum QoS requirements for all active D2D pairs and cellular users. We first solve a resource optimization problem by allocating spectrum resources and controlling power transmission on demand. As resource optimization is an integer nonlinear programming problem, we address this problem by proposing a deep Q-network-based reinforcement learning algorithm (DRL) to optimize the resource allocation issue. The proposed DRL algorithm is trained with a decision-making policy to obtain the best solution in terms of spectrum efficiency, computational time, and throughput. The system performance is validated by simulation. The results show that the proposed method outperforms the existing ones. machine learning (ML); device-to-device (D2D); resource allocation (RA); reinforcement learning (RL); Markov Decision Process (MDP); deep reinforcement learning (DRL); ultra-dense network (UDN); heterogeneous networks (HetNets) +
Footnote †: journal: _Future Internet_  
_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=_title=title_title=_title=_title=title_title=_title=title_title=_title=title_title=title_title=title_title=title_title=title_title=title_title=title_title=title_title=title_titletitle=_titletitle=_titletitle=_titletitletitle=_titletitle=_titletitletitle=_title  
higher data rates and faster data-accessing applications in recent years. Cutting-edge applications such as 3D holography, artificial intelligence (AI), machine-to-machine (M2M) communication, Internet-of-Things (IoT), virtual reality (VR), e-learning, video-based applications, augmented reality (AR), and ultra-broadband demand more bandwidth than can be fulfilled with the existing 4G or even 5G networks. Only unconventional thinking in 5G and B5G cellular networks can achieve this exponential demand for higher data rates and capacity requirements [5]. Spectral efficiency and bandwidth determine wireless communication's capacity and data rates. The combined requirement of hyper-connected devices and new applications will trigger a major evolution in cellular networks, which will improve data rates, latency, energy consumption, and coverage. According to ETSI [6], the target peak data rates for downlink and uplink in 5G are 20 Gbps and 10 Gbps, respectively. The user-centric KPI for 4G, 5G, and 6G is shown in Table 1.  
To overcome the limitations of incredibly low latency and extremely high bandwidth requirements, 5G is designed with various promising technologies like D2D communications, millimeter-wave (mm), ultra-dense networks, massive MIMO, and cognitive radio (CR) networks [1]. As mentioned in Ericsson's recent mobility report and Cisco VNI [3; 4], future wireless traffic explosion is unavoidable, and to manage the wireless traffic explosion, large numbers of small cell (SCeNBs), especially femtocells (HeNBs), deployments are inevitable, which will not only increase the network capacity but also eliminate the coverage holes [1].