## 2 Related Work
A random non-cooperative game approach is used by the authors to formulate the resource allocation problem where each player (D2D pairs) is a learning agent, and the best strategy is learned by interacting with the environment by using a double Deep Q-network (DDQN) [22]. By using a DRL-based double-dueling deep Q-network (D3 QN) algorithm, the authors investigate efficient resource allocation and joint user association for D2D communication in ultra-dense cellular networks in [23]. By simulation, they prove that their proposed scheme can achieve better sum throughput even at higher distances between D2D pairs. In [24], the authors proposed a DRL-based double deep Q-network (DDQN) algorithm that enables D2D pairs to learn an optimal spectrum access strategy autonomously for maximizing the sum throughput without complete prior network information. To achieve faster convergence and smart power control for efficient resource allocation for D2D-enabled cellular networks, the authors proposed a Stackelberg game theory-based multi-agent DRL algorithm in [25]. In this proposal, Stackelberg game theory is used to guide the learning direction efficiently for achieving a faster optimal strategy to maximize sum throughput. In [26], a DL-based algorithm is proposed to solve the resource allocation problem where a deep neural network (DNN) model is used to select an optimal policy randomly. A multi-agent DQN-based DRL algorithm is proposed to approximate the loss function for optimal policy, and simulation results are used to prove the higher system throughput in [26]. To improve spectrum efficiency and maximize system capacity, a DQN-based resource allocation and power control algorithm for a 2-tier D2D-enabled cellular network is proposed in [27], where minimum QoS requirements are ensured. In [28], the authors formulate a non-convex and NP-hard joint power control and resource allocation optimization problem where multiple D2D users share the same cellular resource. To solve this problem, authors initially use fractional programming (FP) techniques. But this technique requires instantaneous CSI which is not feasible for a large and dynamic cellular network. Therefore, they propose a distributed DRL-based scheme where an agent (D2D user) interacts with the environment repeatedly to select an optimal strategy for maximizing system throughput and spectrum efficiency. Finally, the authors prove how their DRL-based scheme is better compared to the FP technique even without instantaneous CSI. ML/DRL-based resource allocation works are summarized in Table 3.  
**Research Gap:** Table 3 reveals that DQN-based joint resource allocation and power control have not been explored yet in a D2D-enabled HetNet considering mutual interference among macro cells, micro cells, and a large number of users simultaneously. In this paper, we formulate a DQN-based joint resource allocation and power control algorithm for D2D-enabled cellular HetNets, which consider mutual interference with all 3-tiers simultaneously to minimize interference with higher user density. As discussed, most of the state-of-the-art papers are focused on interference management for single or 2-tier cellular networks only. In most cases, researchers consider only one D2D pair, and the simultaneous impact of eNB and multiple small cells is neglected. But in ultra-dense networks (UDNs), an enormous number of small cells will be deployed, and multiple CUEs will be associated with different cellular layers. Hence, interference management will be most challenging with added D2D layers [1]. Despite having very promising prospects for D2D communications, very few research addressed mutual interference problems in mode selection and resource allocation in 3-tier cellular networks considering all tiers simultaneously. In a 3-tiers D2D-enabled cellular HetNet, the most and worst challenging interference scenario is when all 3-tiers (eNB, SCeNB, and D2D layers) will interfere with each other mutually, and none of the authors consider it [1].