###### Abstract
The sum-rate maximization, latency minimization, and computation energy minimization are the different optimization problem approaches for resource allocation in wireless communications. In conventional methods, most of the formulated optimization problems for resource allocation are non-convex and non-deterministic polynomial time  
Figure 1: Interference scenario in a D2D-enabled three-tier cellular network. (NP-hard). Moreover, to obtain optimal solutions with an increasing number of users, the time complexity becomes exponentially higher. Therefore, with traditional optimization methods, it is difficult or impossible to optimize effectively the resource allocation problem, which is modeled as a combination of an optimization problem with nonlinear constraints. Fortunately, to solve decision-making problems under uncertainty, deep reinforcement learning (DRL) is highly effective [9]. In this paper, we optimize resource assignment, mode selection, and transmit power control to maximize the throughput and to ensure minimum QoS requirements for all users in D2D-enabled cellular HetNets. The main contributions of the paper are highlighted below. * A deep Q-network-based deep reinforcement learning (DRL) algorithm is proposed to minimize interference for achieving maximum throughput with guaranteed quality of services (QoS) for all active users on the network. * A mathematical model is developed to optimize resource assignment, mode selection, and power control in a D2D-enabled HetNet. The proposed DRL algorithm is analyzed and compared with the existing methods. To validate the system's performance, a simulation approach is used. * We establish the relationship between the agent's reward and system objectives in a D2D-enabled HetNet for achieving optimum benefit in a Q-network-based DRL environment. The structure of this paper is organized as follows. A literature review for interference management in D2D-enabled cellular HetNets using both conventional and machine learning approaches is shown in Section 2. System models, including the communication mode, transmission rate, and problem formation, are presented in Section 3. The proposed ML-based resource allocation model is discussed using RL and DL-based algorithms in Section 4. The analytical model, along with resource allocation algorithms, is also presented in this section. The system performance is analyzed and evaluated in Section 5, and the paper is concluded in Section 6. Table 2 illustrates the key notations and abbreviations used in this paper. \begin{table}
\begin{tabular}{c l} \hline \hline
**Notation** & **Definition** \\ \hline \(\Phi\) & PPP for eNB \\ \hline \(\Phi_{s}\) & PPP for SCeNB \\ \hline \(\Phi_{d}\) & PPP for DUEs \\ \hline \(\lambda_{s}\) & SCeNBs density \\ \hline \(\lambda_{d}\) & DUEs density \\ \hline \(P_{c}\) & Transmit power for CUEs \\ \hline \(P_{d}\) & Transmit power for DUEs \\ \hline \(P_{s}\) & Transmission power for SUEs \\ \hline \(N_{C},N_{D},N_{S}\) & Set of CUEs, DUEs and SUEs \\ \hline \(\alpha\) & Path loss exponent \\ \hline \(\gamma_{th}\) & SINR requirements for cellular and D2D links \\ \hline \(N_{0}\) & Thermal noise \\ \hline \(h_{x,y}\) & Channel co-efficient between nodes \(x\) and \(y\) \\ \hline \(\psi\) & Binary variable \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Key notations and abbreviations.