# Downlink Power Control in Self-Organizing Dense Small Cells Underlaying Macrocells: A Mean Field Game

Prabodini Semasinghe, Ekram Hossain

► To cite this version:

Prabodini Semasinghe, Ekram Hossain. Downlink Power Control in Self-Organizing Dense Small Cells Underlaying Macrocells: A Mean Field Game. 2015. hal-01121946

HAL Id: hal-01121946

https://hal.science/hal-01121946

Preprint submitted on 5 Mar 2015

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Downlink Power Control in Self-Organizing Dense Small Cells Underlaying Macrocells: A Mean Field Game

Prabodini Semasinghe and Ekram Hossain

## Abstract

A novel distributed power control paradigm is proposed for dense small cell networks co-existing with a traditional macrocellular network. The power control problem is first modeled as a stochastic game and the existence of the Nash Equilibrium is proven. Then we extend the formulated stochastic game to a mean field game (MFG) considering a highly dense network. An MFG is a special type of differential game which is ideal for modeling the interactions among a large number of entities. We analyze the performance of two different cost functions for the mean field game formulation. Both of these cost functions are designed using stochastic geometry analysis in such a way that the cost functions are valid for the MFG setting. A finite difference algorithm is then developed based on the Lax-Friedrichs scheme and Lagrange relaxation to solve the corresponding MFG. Each small cell base station can independently execute the proposed algorithm offline, i.e., prior to data transmission. The output of the algorithm shows how each small cell base station should adjust its transmit power in order to minimize the cost over a predefined period of time. Moreover, sufficient conditions for the uniqueness of the mean field equilibrium for a generic cost function are also given. The effectiveness of the proposed algorithm is demonstrated via numerical results.

## Index Terms

5G cellular, self-organizing networks, small cells, differential game, mean field game (MFG), distributed power control, stochastic geometry, finite difference method, Lax-Friedrichs scheme, Lagrange relaxation

## I. INTRODUCTION

The evolving 5G cellular networks will be ultra-dense and heterogeneous with different types of small cells (e.g., femto, micro, pico cells) underlaying the traditional macrocellular networks [1]. Small cells can
increase the network capacity by providing higher quality links between the transmitters and the receivers and by exploiting more spatial spectrum reuse. Some other benefits of small cells include providing services to coverage holes, offloading traffic from the macrocellular network, and increasing the energy efficiency. Due to the increased density and independent deployments of small cells, manual network management would be highly inefficient and expensive. Therefore, self-organization is a desirable feature for dense small cell networks [2], [3], [4], [5]. Self-organization of a network consists of three phases, namely, self-configuration, self-optimization and self-healing. Self-configuration includes pre-operational functions such as registering with the network, parameter setting, and software downloading. Self-optimization and self-healing include operational level functions such as resource allocation, scheduling, and failure recovery.

For self-optimization of small cell networks, the decisions on resource allocation are expected to be taken individually at each base station (i.e., distributed resource allocation). Most of the existing resource allocation schemes are not suitable in the context of self-organizing small cells, as they rely on centralized controllers to take resource allocation decisions. Also, when it comes to dense networks, reducing the amount of information exchange among the base stations is preferred due to the limited backhaul capacity. Therefore, developing resource allocation paradigms for self-organizing dense small cell networks is challenging and has attracted a significant attention of the research community.

Game theory has been widely used to derive distributed resource allocation techniques in the context of wireless cellular networks. Different types of games can be used to model the distributed resource allocation problem. Some of the related work in this context are discussed in Section II. Since classical games have to model the interaction of each player with every other player, analysis of a system with a large number of players can be complex. Therefore, when it comes to a dense network of interconnected base stations, solving the power control problem based on classical game theory becomes very hard and sometimes impossible due to the large number of players. In this context, the theory of mean field game (MFG) [6], [7], [8], which has been used for solving a variety of problems in different research areas [9], [10], [11], [12], [13], can be used.

MFGs can be considered as a special form of differential games applicable for a system with a large number of players. While classical game theory models the interaction of a single player with all the other players of the system, an MFG models the individual's interaction with the effect of the collective behavior (mass) of the players. This collective behavior is reflected in the mean field. Individual player's interaction
with the mean field is modeled by a Hamilton-Jacobi-Bellman (HJB) equation. The motion of the mass according to the players' actions is modeled by a Fokker-Planck-Kolmogorov (FPK) equation [6]. These coupled FPK and HJB equations are also called backward and forward equations, respectively. The solution of an MFG can be obtained by solving these two equations. When modeled as an MFG, since the system can be completely defined by two equations (which are also called the mean field equations), analysis of the system becomes much easier. Moreover, solutions to the MFGs can be obtained distributively and behavior of all the players can be described by one control. In addition, MFGs can take the stochastic nature of the system into account. All of the aforementioned properties make MFG appropriate for modeling the power control problem for dense self-organizing small cell networks. However, modeling the collective effect of the players (i.e., the effect of mass/mean field) has to be done in a realistic way. Accurate modeling of the effect of the mass is a major challenge when adopting mean field games to solve problems in wireless communications.

In this paper, we formulate the downlink power control problem of a dense small cell network under-
laying a macrocellular network as an MFG. The small cell base stations (SBSs), when battery-operated,
are assumed to be constrained by a finite energy. To model the mass (or mean field), we adopt a stochastic
geometry approach. Specifically, we consider minimizing a cost function under certain constraints over
a pre-defined period of time. The cost function is derived by using a stochastic geometry approach in
such a way that it reflects the signal-to-interference-plus-noise ratio (SINR) at the receivers and the
interference caused to the macro cellular network. We propose a finite difference technique to solve the
mean field equations for the formulated MFG. The key feature of the proposed algorithm is that it can be
executed offline. By executing the algorithm, each base station can obtain a power policy which depends
on the initial energy distribution among SBSs. The SBSs can then use that power control policy for data
transmission for the pre-defined period of time. Another advantage of the algorithm is, it minimizes the
cost over a certain period of time instead of taking decisions only based on the instantaneous cost.

The contributions of the paper can be summarized as follows.

1) The downlink power control problem of a small cell network underlaying a traditional macro network (i.e., for a system model consisting of multiple transmitters and multiple receivers) is formulated as a differential game and extended to a mean field game for a dense scenario.

2) The existence of a Nash Equilibrium for the formulated differential game is proven.
3) Using stochastic geometry-based analysis, two cost functions for the mean field game are derived in such a way that the mean field game setting becomes valid. In this way, it combines the theory of MFG with that of stochastic geometry.

4) The forward and backward equations of the mean field game are solved using a finite difference technique.

5) An algorithm is proposed to obtain the mean field equilibrium.

6) The sufficient conditions are given for the uniqueness of the mean field equilibrium for a generic cost function.

The organization of the rest of the paper is as follows. In Section II, we review the related work in the literature. Section III presents the system model and assumptions. In Section IV, we formulate the differential game and the solution concept. Section V presents the formulation of the mean field game. Section VI proposes a finite difference technique to obtain the solution to the formulated mean field game. Numerical results are presented in Section VII. Section VIII concludes the paper.

## II. RELATED WORK

In [14], the authors formulate the uplink resource block allocation problem in an overlay macrocell-femtocell network as a potential game. A distributed technique is proposed using best response dynamics which guarantees the convergence to a Nash Equilibrium. In [15], the downlink resource allocation problem in a small cell network underlaid with a macro network is formulated as an evolutionary game and a distributive algorithm is proposed to achieve evolutionary equilibrium through strategy adaptation. In [16] and [17], the resource allocation problem of a two-tier network is formulated as a hierarchical game where the macro base stations (MBSs) are the leaders and SBSs are the followers. The Stackelberg equilibrium can be achieved distributively using the best response functions. In [18], the authors propose a reinforcement learning algorithm which converges to an $\epsilon$-Nash equilibrium. The equilibrium is achieved through the smoothed best response (SBR) dynamics. The convergence of the SBR algorithm to an $\epsilon$-Nash equilibrium is guaranteed for a payoff function which depends on the sum rate of the entire network. A dynamic pricing scheme based distributed joint power and admission control scheme for two tier CDMA networks is proposed in [19]. The convergence of the proposed algorithm is proven analytically. The authors also claim that the equilibrium point of the proposed algorithm is equivalent to the Nash equilibrium of the underline non-cooperative game. However, implementing most of the above algorithms
for a ultra-dense network would require an extensive amount of information exchange among the base stations.

Recently, mean field game has gained the attention of the research community as a tool to model dense heterogeneous networks (or small cell networks). In [12], [20], [21], [22], the power control problem is modeled as mean field games for scenarios where multiple transmitters transmit to a single receiver (e.g., uplink transmissions in a cellular network). The problem is first formulated as a stochastic differential game and then its convergence to a mean field game is shown for a very large number of transmitters. In [12], the authors show the power control policy obtained at the mean field equilibrium. [20] and [21] present the sufficient conditions for the uniqueness of the respective games formulated in these papers. The work presented in [10] formulates the power control problem in a cognitive radio network as a hierarchical mean field game. The mean field game formulations in all the aforementioned papers consider scaled interference (i.e., interference at the receiver is normalized by the number of transmitters) only which may not be valid for a large-scale small cell network. In addition, none of the above papers presents any technique for solving the mean field equations, which is also very challenging.

### III. SYSTEM MODEL AND ASSUMPTIONS

All the symbols that are used in the system model and the rest of the paper are listed in Table I.

#### A. Network and Propagation Model

We consider an infinite small cell network underlaying an infinite macrocell network. The spatial distributions of the SBSs and MBSs are modeled by two independent Poisson Point Processes (PPPs) denoted by $\Phi_s$ and $\Phi_m$, respectively. The intensities of $\Phi_s$ and $\Phi_m$ are given by $\lambda_m$ and $\lambda_s$, respectively. Users are connected to the base station from which they receive the highest average pilot signal power. The transmit powers of the pilot signals of SBSs and MBSs are given by $p_{s,pilot}$ and $p_{m,pilot}$, respectively. When users are associated to the base station from which they receive the highest average pilot signal power, the cell boundaries can be shown by a weighted Voronoi tessellation [23] as in Fig. 1. The pilot powers of SBSs are lesser than those of MBSs. Although several users may be associated with a base station, we assume that each base station serves only one user at a particular time instant.

We consider the problem of downlink transmit power control at the SBSs. Co-channel deployment is considered, i.e., both the MBSs and SBSs transmit on the same channel. Power control is done in order
<figure><img src="image_1.png" /><figcaption>Fig. 1. Deployment of a two-tier network</figcaption></figure>

to minimize the average cost of each SBS over a given finite time horizon $T$. We will define the cost function later. Each SBS $k$ is assumed to be with a finite amount of energy, denoted by $E_{k,max}$, to spend within the given period of time, $T$. To consider any SBS $k$ with an infinite amount of energy in the same setting, $E_{k,max}$ can be set to a large value, i.e., $E_{k,max} \gg p_{max}T$, where $p_{max}$ is the maximum allowable transmit power for an SBS. The users served by each SBS $k$ have a minimum SINR requirement denoted by $\Gamma_k$. The channels between all the transmitters and all the receivers are assumed to experience i.i.d Rayleigh fading.

The SINR at the user served by SBS $k$ at time $t$ is given by

$$ \text{SINR}_k(t) = \frac{p_k(t)g_{k,k}(t)r_{k,k}(t)^{-\alpha}}{I_{s,k}(t) + I_{m,k}(t) + N_0}, \quad (1) $$

where $I_{s,k}(t) = \sum_{l \in \mathcal{K}, l \neq k} p_l(t)g_{l,k}(t)r_{l,k}(t)^{-\alpha}$ and $I_{m,k}(t) = \sum_{\forall m \in \Phi_m} p_m g_{m,k}(t)r_{m,k}(t)^{-\alpha}$ denote the interference caused by small cell and macro cell networks, respectively. $g_{l,k}$ is fading gain between transmitter $l$ and receiver $k$, $r_{l,k}$ is the distance between the transmitter $l$ and the receiver $k$, $N_0$ is the noise power and $\alpha$ is the path-loss exponent. The following inequality should hold for any SBS to satisfy its QoS constraint:

$$ \frac{p_k(t)g_{k,k}(t)r_{k,k}(t)^{-\alpha}}{I_{s,k}(t) + I_{m,k}(t) + N_0} \geq \Gamma_k, \quad \forall k \in \mathcal{K}. \quad (2) $$

The above expression is linearized as follows:

$$ p_k(t)g_{k,k}(t)r_{k,k}(t)^{-\alpha} - \Gamma_k (I_{s,k}(t) + I_{m,k}(t) + N_0) \geq 0, \quad \forall k \in \mathcal{K}. \quad (3) $$
<table><caption>TABLE I
LIST OF SYMBOLS</caption><thead><tr><td>Symbol</td><td>Description</td></tr></thead><tbody><tr><td>α</td><td>Path-loss exponent</td></tr><tr><td>Γ<sub>k</sub></td><td>SINR requirement for the users of SBS k</td></tr><tr><td>Φ<sub>m</sub></td><td>PPP which represents the spatial distribution of MBSs</td></tr><tr><td>Φ<sub>s</sub></td><td>PPP which represents the spatial distribution of SBSs</td></tr><tr><td>λ<sub>m</sub></td><td>Density of Φ<sub>m</sub></td></tr><tr><td>λ<sub>s</sub></td><td>Density of Φ<sub>s</sub></td></tr><tr><td>c<sub>k</sub>(t)</td><td>Value of the cost at SBS k at time t</td></tr><tr><td>E<sub>k</sub></td><td>State space (i.e., possible energy levels) of SBS k</td></tr><tr><td>e<sub>k</sub></td><td>State of the system at time t</td></tr><tr><td>e<sub>k</sub>(t)</td><td>Available energy of SBS k at time t</td></tr><tr><td>E<sub>k,max</sub></td><td>Maximum available energy of SBS k</td></tr><tr><td>g<sub>k,l</sub></td><td>Fading channel gain between transmitter k and receiver l</td></tr><tr><td>I<sup>m</sup><sub>k</sub>(t)</td><td>Interference caused to the nearest macro user by SBS k at time t</td></tr><tr><td>I<sub>s,k</sub>(t), I<sub>m,k</sub>(t)</td><td>Interferences caused by small cell network and macro cell network at the user served by SBS k at time t</td></tr><tr><td>N<sub>0</sub></td><td>Variance of noise power</td></tr><tr><td>K</td><td>Set of SBSs</td></tr><tr><td>P<sub>k</sub></td><td>Set of all possible transmit powers of SBS k</td></tr><tr><td>p<sub>k</sub>(t)</td><td>Transmit power of SBS k at time t</td></tr><tr><td>p<sub>m</sub></td><td>MBS transmit power</td></tr><tr><td>p<sub>max</sub></td><td>Maximum transmit power for SBSs</td></tr><tr><td>p<sub>m,pilot</sub>, p<sub>s,pilot</sub></td><td>Pilot signal power of MBSs and SBSs</td></tr><tr><td>r<sub>k,l</sub></td><td>Distance between transmitter k and receiver l</td></tr><tr><td>T</td><td>Time period during which power control is done</td></tr><tr><td>u<sub>k</sub>(t)</td><td>Value function of SBS k at time t</td></tr><tr><td>v(t, e)</td><td>Lagrange multiplier at time t and energy e</td></tr><tr><td>w<sub>1</sub>, w<sub>2</sub></td><td>Biasing factors</td></tr><tr><td>X, Y</td><td>Number of discretization levels in time axis and energy axis, respectively</td></tr></tbody></table>
## B. Cost Function of an SBS

The cost function of SBS $k$ at time $t$ is composed of two components as follows:

* The cost associated with the satisfaction of the QoS constraint ($\Gamma_k$), denoted by $f_k^{(1)}(t)$, and

* The cost associated with the interference caused to the nearest macro user, denoted by $f_k^{(2)}(t)$.

Based on (3), we define $f_k^{(1)}(t)$ as follows:

$$f_k^{(1)}(t) = (\Gamma_k (I_{s,k}(t) + I_{m,k}(t) + N_0) - p_k(t)g_{k,k}(t)r_{k,k}(t)^{-\alpha})^2. \quad (4)$$

Minimizing $f_1$ will attempt to satisfy the QoS constraint, but it will also discourage further increase of transmit power after satisfying the QoS constraint. A similar cost function is also used in [24] and [25] for uplink power control. On the other hand, $f_k^{(2)}(t)$ is defined as the interference caused at the nearest macro user at time $t$, which is given by $I^m_k(t)$, as follows:

$$f_k^{(2)}(t) = I^m_k(t) = p_k(t)g_{k,m}r_{k,m}^{-\alpha}. \quad (5)$$

Accordingly, we define the cost function of SBS $k$ at time $t$ (i.e., $c_k(t)$) as a linear combination of above two functions:

$$c_k(t) = w_1 f_k^{(1)}(t) + w_2 f_k^{(2)}(t) \quad (6)$$

where $w_1$ and $w_2$ are biasing factors which bring the above two terms into one scale. The network operator has the freedom to set these biasing factors.

Note that for the formulation of the mean field game, in Section V, we will generalize the cost function for any generic SBS such that interchangeability (or permutation) of the actions among the SBSs does not affect the outcome of the game.

## C. State, Action Space, and Control Policy of an SBS

The state of SBS $k$ at time $t$ is defined by the amount of available energy at that time, which is given by, $e_k(t)$. Therefore, the state space $\mathcal{E}_k$ of SBS $k$ can be written as follows:

$$\mathcal{E}_k = [0, e_k(0)] = \{e_k(t) \in \mathbb{R} | 0 \le e_k(t) \le e_k(0)\}, \quad (7)$$

where $e_k(0)$ is the available energy of SBS $k$ at time 0.

We also define the state of the system at time $t$, $e(t)$ as follows:

$$e(t) = [e_k(t)]_{\forall k}^T. \quad (8)$$
The set of actions for SBS k includes all possible transmit powers as follows:

$$P_k = [0, p_{max}] \qquad (9)$$

where $p_{max}$ is the maximum allowable transmit power of any SBS. The transmit power of SBS k at time t is denoted by $p_k(t)$.

The evolution of the state (in this case, available energy) over time is decided by a control, which in this case corresponds to the transmit power given by $p_k(t) \in [0, p_{max}]$. Consequently, the state equation of the system is defined as follows.

**Definition (State equation):** The state of SBS k is given by the random variable $e_k(t) \in [0, e_k(0)]$ whose evolution is defined by the following differential equation:

$$de_k(t) = -p_k(t)dt, \quad 0 \le t \le T. \qquad (10)$$

The control policy is a mapping of the state to an action. This is defined over the given period of time, T. We denote the control policy of player k over the time period T by $p_k(0 \to T)$. An optimal power control policy, $p_k^*(0 \to T)_{\forall k}$ should minimize the average cost of each player k over the given finite time horizon, T. Therefore, we write $p_k^*(0 \to T)$ as follows:

$$p_k^*(0 \to T) = \arg \min_{p_k(0 \to T)} E \left[ \int_0^T c_k(t)dt + c_k(T) \right] \qquad (11)$$

where $c_k(T)$ is the terminal cost (i.e., cost at the end of time period T).

Our interest is to obtain the optimal power control policy distributively at each SBS in order to minimize the average cost over time interval T. This can be seen as an optimal control problem [26], but with several controllers (each SBS is a controller in this case). Such a problem can be formulated as a differential game [27]. Differential games can be seen as a generalization of the optimal control problems for the cases where there are more than one controller. A mean field game is an extention to a differential game when the system has a large number of players. In the next two sections, we show the differential game formulation and its extension to a mean field game (denoted by $G_s$ and $G_m$, respectively). The set of SBSs $K = \{1, 2, ..., K\}$ is the set of players in these game models.

## IV. DIFFERENTIAL GAME FORMULATION

In this section, we formulate the differential game to model the downlink transmit power control problem for the system model described above. To formulate the differential game denoted by $G_s$, we define the
value function $u_k(t)$ as follows:

$$u_k(t) = \min_{p_k(t \to T)} E \left[ \int_t^T c_k(\tau) d\tau + c_k(T) \right], \quad t \in [0, T] \qquad (12)$$

where $c_k(T)$ is the terminal cost.

According to Bellman's principle of optimality [28], an optimal control policy should have the property that whatever the initial state and initial decision are, the remaining decisions must form an optimal policy with regard to the state resulting from the first decision [29], [30]. Accordingly, the optimal power control policy can then be defined in-terms of the value function as follows.

**Definition (Optimal control):** The power profile $p_k^*(t \to T)$ is the optimal power control policy for SBS $k$ if for any $t \in [0, T]$ $E \left[ \int_t^T c_k(p_k^*(\tau))d\tau + c_k(T) \right] = u_k(t)$, $t \in [0, T]$.

This value function should satisfy a partial differential equation which is in the form of a Hamilton-Jacobi-Bellman (HJB) equation [31]. The HJB equation corresponding to the optimal control problem given in equation (11) satisfying the state equation (10) can be written as follows:

$$\frac{\partial u_k(t)}{\partial t} + \min_{p_k(t)} \left( c_k(p_k(t)) - p_k(t) \frac{\partial u_k(t)}{\partial e} \right) = 0 \qquad (13)$$

where $H(e_k(t), \frac{\partial u_k(t)}{\partial e}) = \min_{p_k(t)} (c_k(t) - p_k(t) \frac{\partial u_k(t)}{\partial e})$ is called the Hamiltonian. Now, the Nash equilibrium of the game $\mathcal{G}_s$ is defined as follows.

**Definition (Nash equilibrium of game $\mathcal{G}_s$):** A power profile

$$\boldsymbol{p}^* = [p_1^*(0 \to T), p_2^*(0 \to T), \dots, p_k^*(0 \to T), \dots, p_K^*(0 \to T)]$$

is a Nash equilibrium of the game $\mathcal{G}_s$ if and only if

$$p_k^*(0 \to T) = \arg \min_{p_k(0 \to T)} E \left[ \int_0^T c_k(p_k(t), \boldsymbol{p}_{-k}^*) dt + c_k(T) \right], \quad \forall k \qquad (14)$$

subject to

$$de_k(t) = -p_k(t)dt \quad (0 \le t \le T), \quad \forall k \qquad (15)$$

where $p_{-k}^*$ denotes the transmit power vector of the SBSs except SBS $k$.

When the above condition is satisfied, none of the players can have a lesser cost by deviating unilaterally from the current power control policy. Hence, it is equivalent to the Nash equilibrium of game $\mathcal{G}_s$. The Nash equilibrium of the above differential game can be obtained by solving the HJB equations associated
with each player given in equation (13) [32]. We state following theorem on the existence of the Nash
equilibrium for $\mathcal{G}_s$.

**Theorem 4.1: There exists at least one Nash equilibrium for the differential game $\mathcal{G}_s$.**

*Proof:* Existence of a solution to the HJB equation in (13) ensures the existence of the Nash
equilibrium for the game $\mathcal{G}_s$. It is known that there exists a solution to the HJB if the Hamiltonian
is smooth [20], [33]. We write the Hamiltonian for equation (13) as follows:

$$ 
\begin{aligned}
H \left( e_k(t), \frac{\partial u_k(t)}{\partial e} \right) &= \min_{p_k(t)} \left( c_k(t) - p_k(t) \frac{\partial u_k(t)}{\partial e} \right) \\
&= \min_{p_k(t)} \left[ w_1 \left( \Gamma_k (I_{s,k}(t) + I_{m,k}(t) + N_0) - p_k(t) g_{k,k}(t) r_{k,k}(t)^{-\alpha} \right)^2 + w_2 (p_k(t) g_{k,m} r_{k,m}^{-\alpha} - p_k(t) \frac{\partial u_k(t)}{\partial e}) \right].
\end{aligned} 
\quad (16) $$

The first, second, and third derivatives of the Hamiltonian w.r.t. $p_k(t)$ can be written as follows:

$$ \frac{\partial H}{\partial p_k(t)} = 2w_1 g_{k,k} r_{k,k}^{-\alpha} \left( \Gamma_k (I_{s,k}(t) + I_{m,k}(t) + N_0) - p_k(t) g_{k,k}(t) r_{k,k}(t)^{-\alpha} \right) + w_2 g_{k,m} r_{k,m}^{-\alpha} - \frac{\partial u_k}{\partial e} \quad (17) $$

$$ \frac{\partial^2 H}{\partial p_k(t)^2} = 2w_1 (g_{k,k} r_{k,k}^{-\alpha})^2 \quad (18) $$

$$ \frac{\partial^3 H}{\partial p_k(t)^3} = 0. \quad (19) $$

For any $n > 3$, $\frac{\partial^n H}{\partial p_k(t)^n} = 0$. The function has derivatives of all orders, hence it is smooth. Therefore, it
can be concluded that there exists at least one Nash equilibrium for the differential game $\mathcal{G}_s$.

<figure><img src="image_2.png" /></figure>

Obtaining the equilibrium for game $\mathcal{G}_s$ for a system with $K$ players involves solving $K$ simultaneous
partial differential equations (PDEs). However, for a dense small cell network, obtaining the Nash equi-
librium by solving $\mathcal{G}_s$ would be difficult or even impossible due to the large number of simultaneous
PDEs. Therefore, for modeling and analysis of a dense small cell network, we propose a mean field game
formulation where the system can be defined solely by two coupled equations. In the next section, we
show the extension of game $\mathcal{G}_s$ to the mean field game $\mathcal{G}_m$.
# V. FORMULATION OF MEAN FIELD GAME

## A. Assumptions

First, we define the mean field as follows:

**Definition (Mean field)**

$$m(e, t) = \lim_{K \to \infty} \frac{1}{K} \sum_{\forall k \in \mathcal{K}} \mathbb{1}_{\{e_k(t)=e\}} \qquad (20)$$

where $\mathbb{1}$ denotes an indicator function which returns 1 if the given condition is true and zero otherwise.

For a given time instant, mean field is the probability distribution of the states over the set of players.

The general setting of mean field games is based on the following four assumptions [34]:

1) Rationality of the players,

2) The existence of a continuum of the players (i.e., continuity of the mean field),

3) Interchangeability of the actions among the players (i.e., permutation of the actions among the players would not affect outcome of the game), and

4) Interaction of the players with the mean field.

The first assumption is generally applied in any type of game to ensures that the players can take logical decisions. The presence of a large number of SBSs in the system model ensures the existence of the continuum of the players. We derive the cost function (will be shown in next subsection) in order to ensure the interchangeability of the actions among the players. The idea of the fourth assumption is that each player interacts with the mean field instead of interacting with all the other players.

## B. Deriving the Cost Function

A cost function, which depends only on control (and/or state) and mean field, would ensure that the third assumption of the mean field game setting is valid. To derive such a cost function for $G_m$, we follow a stochastic geometry-based approach. In this case, for simplicity, we assume an interference-limited network setting (i.e., $N_0 = 0$). This assumption can be justified due the fact that the network is highly dense. It is also assumed that all SBSs have the same QoS constraint given by $\gamma$.

By taking the spatial averages over the point process, we generalize $f_k^{(1)}(t)$ and $f_k^{(2)}(t)$ for any generic player transmits with power $p(t)$ at time $t$ as follows. We denote the new functions by $f^{(1,mean)}(t)$ and
$f^{(2,mean)}(t)$. The function $f^{(1,mean)}(t)$ is given by

$$f^{(1,mean)}(t) = \left( -p(t)\mathbf{E}[g_{k,k}(t)]\mathbf{E}[r_{k,k}(t)]^{-\alpha} + \Gamma\mathbf{E}_{\mathcal{I}_s,g_{l,k}(t),p_l(t)} \left[ \sum_{l \in \mathcal{I}_s} p_l(t)g_{l,k}(t)r_{l,k}(t)^{-\alpha} \right] \right. \\ \left. + \Gamma\mathbf{E}_{\mathcal{I}_m,g_{m,k}(t),p_m(t)} \left[ \sum_{m \in \mathcal{I}_m} p_m(t)g_{m,k}(t)r_{m,k}(t)^{-\alpha} \right] \right)^2 \quad (21)$$

where $\mathcal{I}_s$ and $\mathcal{I}_m$ are the sets of interfering SBSs and MBSs, respectively. The function $f^{(2,mean)}(t)$ for any generic SBS is given by

$$f^{(2,mean)}(t) = p(t)\mathbf{E}_{\phi_s}[g_{k,m}]\mathbf{E}_{\phi_s}[r_{k,m}]^{-\alpha}. \quad (22)$$

Then we write the cost function for a generic SBS as follows:

$$c(t) = w_1 f^{1,mean}(t) + w_2 f^{2,mean}(t). \quad (23)$$

1) Derivation of $f^{(1,mean)}(t)$:

Derivation of $\mathbf{E}[I_s(t)]$ and $\mathbf{E}[I_m(t)]$: We derive $\mathbf{E}[I_s(t)] = \mathbf{E}_{\mathcal{I}_s} [\sum_{l \in \mathcal{I}_s} p_l(t)g_{l,k}(t)r_{l,k}(t)^{-\alpha}]$ for a generic SBS $k$ at the origin. According to Slivnyak's theorem [35], the statistics for a PPP is independent of the test location. Therefore, the analysis holds for any small cell user at a generic location. Since the channel gains and the transmit powers of the interferes are independent of the point process $\Phi_s$,

$$\mathbf{E}[I_s(t)] = \mathbf{E}[p_k(t)]\mathbf{E}[h_{k,k}(t)]\mathbf{E}_{\Phi_s}\left[\sum_{l \in \mathcal{I}_s} r_{l,k}(t)^{-\alpha}\right]. \quad (24)$$

For Rayleigh fading, assuming $h_{l,k} \sim \exp(1)$ for $\forall k, l \in \Phi_s$, by using Campbell's theorem [36], we have the following:

$$\mathbf{E}[I_s(t)] = \mathbf{E}[p_k(t, t)] \int_{\mathcal{R}^2} r_{l,k}(t)^{-\alpha} d(\mathcal{R}). \quad (25)$$

Since the received power cannot be larger than transmit power, the path-loss is assumed to be 1 when $r_{l,k}(t) < 1$. Then, we derive the average interference at a generic user at the origin as follows:

$$\begin{aligned} \mathbf{E}[I_s(t)] &= \mathbf{E}[p_k(t)]2\pi\lambda_s \left[ \int_0^1 r dr + \int_1^\infty r^{-\alpha} r dr \right], \\ &= 2\pi\lambda_s \mathbf{E}[p_k(t)] \left( \frac{1}{2} + \frac{1}{\alpha - 2} \right). \end{aligned} \quad (26)$$

By following similar steps, we can derive the average interference caused from the macro network as follows:

$$\mathbf{E}[I_m(t)] = 2\pi\lambda_m p_m \left( \frac{1}{2} + \frac{1}{\alpha - 2} \right). \quad (27)$$
**Derivation of $E[r_{k,k}(t)]$:** Each small cell user is assumed to be connected to the nearest SBS. It is also known that the distance to the nearest base station from any generic point is Rayleigh distributed [37]. Therefore, the probability density function (PDF) of $r_{k,k}(t)$ can be written as

$$f_{r_{k,k}}(r) = 2\pi\lambda_s re^{-\pi\lambda_s r^2} dr. \qquad (28)$$

Therefore, the average distance is the given by

$$E[r_{k,k}(t)] = \frac{1}{2\sqrt{\lambda_s}}. \qquad (29)$$

2) Derivation of $f^{(2,mean)}$: In order to determine $f^{(2,mean)}$, we need to determine PDF of the distance to the nearest possible macro user (i.e., $r_{k,m}$) from any generic small cell user. The nearest macro user can be just beyond the edge of coverage area of the small cell. In practice, cell edges can be created both due to MBSs and SBSs. However, for analytical tractability, we assume that the small cell edges are formed only due to MBSs. We find the PDF of $r_{k,m}$ as follows.

<figure><img src="image_3.png" /><figcaption>Fig. 2. A cell edge of an SBS.</figcaption></figure>

Considering the cell edge between an SBS and an MBS (see Fig. 2), we can write,

$$p_{s,pilot} R^{-\alpha} = p_{m,pilot} (X - R)^{-\alpha} \qquad (30)$$

where $R$ is the distance from SBS to the closest cell edge and $X$ is the distance to the nearest MBS. Since the distribution of MBSs is PPP with intensity $\lambda_m$, the cumulative distribution function (CDF) $F_{R_{k,m}}(r_{k,m})$ and PDF $f_{R_{k,m}}(r_{k,m})$ of $r_{k,m}$ are given by

$$F_{R_{k,m}}(r_{k,m}) = 1 - e^{-\lambda_m \pi b^2 r^2}, \quad f_{R_{k,m}}(r_{k,m}) = 2\pi \lambda_m r b^2 e^{-\lambda_m \pi b^2 r^2} \qquad (31)$$

where $b = [1 + \frac{p_{m,pilot}}{p_{s,pilot}}]^{\frac{1}{\alpha}}$.

The above equations imply that $r_{k,m}$ is Rayleigh distributed and the expected value is given by

$$E_{m \in \phi_s}[r_{k,m}] = \frac{1}{2\sqrt{\lambda_m} \left[ 1 + \left( \frac{p_{m,pilot}}{p_{s,pilot}} \right)^{\frac{1}{\alpha}} \right]} \qquad (32)$$
We assume that the path-loss exponent $\alpha$ equals to 4. By substituting the values from equations (26), (27), (29), and (32) for $f^{(1,mean)}$ and $f^{(2,mean)}$ in expression (23), the cost function of a generic SBS transmitting with power $p(t)$ at time $t$ can then be written as follows:

$$c(t) = w_1 (-16\lambda_s^2 p(t) + 2\pi\Gamma [p_m\lambda_m + \mathbf{E}[p_l(t)]\lambda_s])^2 + w_2 16p(t)\lambda_m^2 \left[1 + \left(\frac{p_{m,pilot}}{p_{s,pilot}}\right)^{\frac{1}{4}}\right]^4. \quad (33)$$

For a generic cost function, the control (i.e., transmit power) of time $t$ would only depend on the state of the SBS. Hence, the expectation of the transmit power over all interfering SBSs can be written in terms of the mean field. Then, the above equation can be re-written as follows:

$$c(t, e) = w_1 \left( -16\lambda_s^2 p(t, e) + 2\pi\Gamma \left[ p_m\lambda_m + \lambda_s \int_{e \in \mathcal{E}} p(t, e)m(t, e)de \right] \right)^2 \\ + w_2 16p(t, e)\lambda_m^2 \left[ 1 + \left( \frac{p_{m,pilot}}{p_{s,pilot}} \right)^{\frac{1}{4}} \right]^4. \quad (34)$$

For comparison purpose, we also introduce another cost function<fnref n="1" /> denoted by $\hat{c}(t, e)$, which is similar to that in [38], as follows:

$$\hat{c}(t, e) = -\hat{w}_1 \mathbf{E}_{\phi_s} [\text{SINR}_k(p(t, :), m(t, :))] + \hat{w}_2 p_k(t) \mathbf{E}_{\phi_s} [g_{k,m}] \mathbf{E}_{\phi_s} [r_{k,m}]^{-\alpha} \quad (35)$$

where $\hat{w}_1$ and $\hat{w}_2$ are weighting factors. This cost function does not take the QoS constraint into account. The SBSs can increase their transmit power even after satisfying the QoS constraint. A performance comparison of these two cost functions will be shown in Section VII.

The first term of $\hat{c}(t, e)$ is derived using stochastic geometry analysis, in [15] and then $\hat{c}(t, e)$ can be written as follows:

$$\hat{c}(t, e) = -\hat{w}_1 \frac{8p(t, e)}{A^2 \left( \lambda_m \sqrt{p_m} + \lambda_s \int_{\forall \bar{e} \in \mathcal{E}} \sqrt{p(t, \bar{e})} m(t, \bar{e}) \, de \right)^2} + \hat{w}_2 16p(t, e) \lambda_m^2 \left[ 1 + \left( \frac{p_{m,pilot}}{p_{s,pilot}} \right)^{\frac{1}{4}} \right]^4. \quad (36)$$

### C. Mean Field Equations

Since the cost functions now only depend on the mean field and the control, the optimal control problem given in equation (11) is similar for all the players in the system. The HJB in equation (13) can then be modified as follows [9]:

$$\frac{\partial u(t, e)}{\partial t} + \min_{p(t,e)} \left( c(p(t, e), m(t, e)) - p(t, e) \frac{\partial u(t, e)}{\partial e} \right) = 0 \quad (37)$$

<footnote n="1">¹We will see later in the paper that these two cost functions result in different power control policies.</footnote>
where $\min_{p(t,e)} (c(p(t,e), m(t,e)) - p(t,e) \frac{\partial u}{\partial e})$ is the Hamiltonian, generally denoted by $H(e, m(t,e), \frac{\partial u(t,e)}{\partial e})$, and $u(t,e)$ is the value function. The same equations are applicable for $\hat{c}(t, e)$ as well. The HJB equation models an individual player's interaction with the mass (i.e., mean field). This is also called the backward equation.

The motion of the mean field corresponds to a Fokker-Planck-Kolmogorov(FPK) equation which is called as the forward equation. The forward equation of game $G_m$ is given as

$$ \frac{\partial m(t, e)}{\partial t} + \frac{\partial}{\partial e} \left( m(t, e) \frac{\partial H}{\partial z} \right) = 0 \qquad (38) $$

where $z = \frac{\partial u}{\partial e}$. It was proven that $\frac{\partial H}{\partial z}$ can be replaced by the control [29], which is in this case $p(t, e)$. Hence, the modified FPK equation can be written as

$$ \frac{\partial m(t, e)}{\partial t} - \frac{\partial}{\partial e} (m(t, e)p(t, e)) = 0. \qquad (39) $$

The mean field equilibrium (MFE) can be obtained by solving the two coupled PDEs given in equations (37) and (39).

# VI. SOLUTION OF MEAN FIELD GAME: MEAN FIELD EQUILIBRIUM

## A. Mean Field Equilibrium (MFE)

The solution of the mean field game, namely, the mean field equilibrium (MFE) can be obtained by solving the mean field equations. There is no general technique to solve the mean field equations. In this section, we propose a finite difference technique to obtain the MFE based on the method proposed in [29]. The coupled equations (37) and (39) are iteratively solved until the equilibrium is achieved. The convergence point of the algorithm is guaranteed to be the optimal solution (i.e., MFE) if the objective function of the optimal control problem, $E[\int_{t=0}^{T} c(t, e)dt + c(T)]$ is convex.

As we propose a finite difference method, the time axis $[0, T]$ and the state space $[0, E_{max}]$ are discretized into $X \times Y$ spaces. Hence, we have $X + 1$ points in time and $Y + 1$ points in state space. We also define

$$ \delta t := \frac{T}{X} \quad \text{and} \quad \delta e := \frac{E_{max}}{Y}. $$

1) Solution to the forward equation: The forward equation is solved using the Lax-Friedrichs scheme to guarantee the positivity of the mean field. The Lax-Friedrichs scheme is first order accurate in both space and time [39]. By applying the Lax-Friedrichs scheme to equation (39), we have

$$ M(i+1, j) = \frac{1}{2} [M(i, j-1) + M(i, j+1)] + \frac{\delta t}{2(\delta e)} [P(i, j+1)M(i, j+1) - P(i, j-1)M(i, j-1)] $$
where $M(i, j)$ and $P(i, j)$ denote, respectively, the values of the mean field and power at time instant $i$ and energy level $j$ in the discretized grid.

2) Solution to the backward equation: The existing finite difference techniques to solve partial differential equations can not be applied directly to solve the HJB equation due to the Hamiltonian. Therefore, we reformulate the problem by writing the HJB equation as its corresponding optimal control problem with the forward equation as a constraint. The reformulated problem is stated below.

$$ \min_{p(t,e), m(t,e)} \mathbf{E} \left[ \int_{t=0}^{T} c(t)dt + c(T) \right], $$

subject to

$$ \frac{\partial m(t, e)}{\partial t} - \frac{\partial}{\partial e} (m(t, e)p(t, e)) = 0, \quad \forall (t, e) \in [0, T] \times [0, E_{max}] $$

and

$$ \int_{e \in \mathcal{E}} m(t, e)de = 1, \quad \forall t \in [0, T]. \qquad (40) $$

The second constraint is to guarantee that the mean field gives the PDF of the state distribution over SBSs at each time instant.

Then, we write the Lagrangian $L(m(t, e), p(t, e), v(t, e))$ for the above problem with the Lagrange multiplier $v(t, e)_{\forall t, e}$ as follows:

$$ \begin{aligned} L(m(t, e), p(t, e), u(t, e)) &= E \left[ \int_{t=0}^{T} c(t, e)dt \right] + \int_{t=0}^{T} \int_{e=0}^{E_{max}} v(t, e) \left[ \frac{\partial m(t, e)}{\partial t} - \frac{\partial (m(t, e)p(t, e))}{\partial e} \right] dedt \\ &= \int_{t=0}^{T} \int_{e=0}^{E_{max}} m(t, e)c(t, e)de \, dt + \int_{t=0}^{T} \int_{e=0}^{E_{max}} v(t, e) \left[ \frac{\partial m(t, e)}{\partial t} - \frac{\partial (m(t, e)p(t, e))}{\partial e} \right] dedt \end{aligned} \qquad (41) $$

where we have assumed the terminal cost $c(T)$ to be equal to zero.

As we use a finite difference scheme to solve the forward equation (i.e., first constraint in the reformulated optimization problem), we also discretize the Lagrangian to solve the above given optimal control problem. The discretized Lagrangian $L_D$ is given as follows:

$$ L_D = \delta e \delta t \sum_{i=1}^{X+1} \sum_{j=1}^{Y+1} \left[ M(i, j)C(i, j) + V(i, j) \left( \frac{M(i+1, j) - 0.5(M(i, j+1) + M(i, j-1))}{\delta t} \right) \right. \\ \left. - V(i, j) \left( \frac{P(i, j+1)M(i, j+1) - P(i, j-1)M(i, j-1)}{2\delta e} \right) \right] \qquad (42) $$
where $V(i, j)$ and $C(i, j)$ denote the Lagrange multiplier and the value of the cost function at point $(i, j)$ on the discretized grid.

The optimal decision variables (given by $P^*, M^*, V^*$) must satisfy the Karush-Kuhn-Tucker (KKT) conditions. For an arbitrary point $(\bar{i}, \bar{j})$ in the discretized grid, by evaluating and re-arranging the KKT condition, $\frac{\partial L_D}{\partial M(i,j)} = 0$, we deduce the following equation to update $V$:

$$ V(\bar{i}-1, \bar{j}) = 0.5[V(\bar{i}, \bar{j}-1) + V(\bar{i}, \bar{j}+1)] - \delta t C(\bar{i}, \bar{j}) - \delta t \sum_{j=1}^{Y+1} \left( M(\bar{i}, j) \frac{\partial C(\bar{i}, j)}{\partial M(\bar{i}, \bar{j})} \right) \\ + \frac{\delta t P(\bar{i}, \bar{j})}{2\delta e} [V(\bar{i}, \bar{j}-1) - V(\bar{i}, \bar{j}+1)]. \quad (43) $$

If $V(N+1, :)$ is known, the values of the Lagrange multipliers can be updated iteratively using the above equation.

Assume an optimization problem whose objective function is given by $f(x)$ and has $l$ equality constraints each denoted by $h_i(x)_{i \in \{1, 2, \dots, l\}}$. It is known that the following relationship exists at the optimal solution [40]:

$$ \nabla f(\boldsymbol{x}^*) = \sum_{i=1}^{l} v_i \nabla h_i(\boldsymbol{x}^*) \quad (44) $$

where $x^*$ is the optimal solution and $v_i$ is the Lagrange multiplier corresponding to $h_i$.

Let $(p^*(t, e), m^*(t, e))_{\forall(t,e) \in [0,T] \times [0, E_{max}]}$ denote the solution for the optimal control problem given in (40). Now, consider the optimal control problem given below for any arbitrary $e'$ at time $T$:

$$ \begin{gathered} \min_{p(T,e'), m(T,e')} f_T(p(T, e'), m(T, e')) = \boldsymbol{E} \left[ \int_{t=T}^{T} c(t)dt + c(T) \right] \\ \frac{\partial m(T, e')}{\partial t} - \frac{\partial}{\partial e} (m(T, e')p(T, e')) = 0. \end{gathered} \quad (45) $$

According to Bellman's principle of optimality [28], it can be concluded that the optimal solution to the above problem (45) is given by $p^*(T, e')$. As $c(T) = 0$, $\nabla f_T(p(T, e'), m(T, e')) = 0$. Assuming that the derivative of the first constraint is non-zero at the optimal point and from equation (44), it can be concluded that $v(T, e) = 0$ for all $e$. Hence, by setting $V(T, e) = 0, \forall e \in \mathcal{E}$, and then using the expression in equation (43) we can update the values of the Lagrange multipliers.

Next, we consider the KKT condition, $\frac{\partial L_D}{\partial P(i,j)} = 0$ for any arbitrary point $(\bar{i}, \bar{j})$ in the discretized grid. Then

$$ \sum_{j=1}^{Y+1} \left( M(\bar{i}, j) \frac{\partial C(\bar{i}, j)}{\partial P(\bar{i}, \bar{j})} \right) - \frac{M(\bar{i}, \bar{j})}{2\delta e} [V(\bar{i}, \bar{j}-1) - V(\bar{i}, \bar{j}+1)] = 0. \quad (46) $$
The equation (46) has to be solved for $P(\bar{i}, \bar{j})$ to obtain the transmit power at point $(\bar{i}, \bar{j})$.

3) **Obtaining the MFE:** The equations (40), (43), and (46) can be solved iteratively until it converges. The complete algorithm to obtain the converging point is given in **Algorithm 1**. We state the following theorem regarding the convergence point.

**Theorem 6.1:** The convergence point of the given algorithm is the mean field equilibrium of game $\mathcal{G}_m$ with cost function, $c(t, e)$.

*Proof:* The Hessian w.r.t. $P(\bar{i}, \bar{j})$ and $M(\bar{i}, \bar{j})$ of the decritized version of objective the function of the optimization problem given in equation (40) can be proven to be positive for any arbitrary $(\bar{i}, \bar{j})$. Hence, the problem given in equation (40) is a convex optimization problem. Since the KKT conditions are necessary and sufficient conditions for the optimal solution of a convex optimization problem, the convergence point of the algorithm is equivalent to the MFE of game $\mathcal{G}_m$ with cost function, $c(t, e)$. ■

## B. Uniqueness of the MFE

In the following theorem, we state the sufficient conditions for $\mathcal{G}$ to have a unique solution.

**Theorem 6.2:** The game $\mathcal{G}$ has a unique solution if the following conditions are satisfied.

$$1) \quad \frac{\partial}{\partial m} H(p, z, m) > 0$$

$$2) \quad \frac{\partial}{\partial z} (mp) > 0$$

$$3) \quad \frac{\partial}{\partial m} (mp) > 0$$

where $z = \frac{\partial u}{\partial x}$.

*Proof:*

Assume that $(m_0(t, e), u_0(t, e))$ and $(m_2(t, e), u_2(t, e))$ are two different solutions for the game $\mathcal{G}$. Here we use the notation $x(t, e)$ to denote a continuous function of $t \in [0, T]$ and $e \in [0, E_{max}]$. Consider the following integration:

$$I(1) = \frac{d}{dt} \int_{e \in \mathcal{E}} (u_1(t, e) - u_0(t, e)) (m_1(t, e) - m_0(t, e)) \, de. \quad (47)$$

We rearrange the above integration as follows:

$$\begin{aligned} I(1) = & \int_{e \in \mathcal{E}} \left( \frac{\partial u_1(t, e)}{\partial e} - \frac{\partial u_0(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\ & + \int_{e \in \mathcal{E}} (u_1(t, e) - u_0(t, e)) \left( \frac{\partial m_1(t, e)}{\partial e} - \frac{\partial m_0(t, e)}{\partial e} \right) \, de. \end{aligned}$$
**Algorithm 1** Computing the Mean Field Equilibrium

Initialization: Initialize $M(0, :), V(N+1, :), iteration = 1$

repeat

for all $i = 1: 1: X$ do

for all $j \in \{1, ..., Y\}$ do

Calculate $M(i + 1, j)$ using equation (40)

end for

end for

if $P(i, M + 1) = 0$ then

$$M(i + 1, Y + 1) = M(i, Y + 1)$$

else

$$M(i + 1, Y + 1) = 0$$

end if

∀i, Normalize $M$

for all $i = X + 1: -1: 1$ do

for all $j \in \{1, ..., Y + 1\}$ do

Update $V(i - 1, j)$ using equation (43)

end for

end for

for all $i = 1: 1: X + 1$ do

for all $j \in \{1, ..., Y + 1\}$ do

Update $P(i, j)$ using equation (46)

end for

end for

$iteration = iteration + 1$

until $iteration \ge Iter_{max}$
After substituting equations (37) and (39), we obtain

$$
\begin{align*}
I(1) &= \int_{e \in \mathcal{E}} H \left( e, m_0(t, e), \frac{\partial u_0(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad - \int_{e \in \mathcal{E}} H \left( e, m_1(t, e), \frac{\partial u_1(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad + \int_{e \in \mathcal{E}} \frac{\partial}{\partial e} (m_1(t, e)p_1(t, e)) (u_1(t, e) - u_0(t, e)) \, de \\
&\quad - \int_{e \in \mathcal{E}} \frac{\partial}{\partial e} (m_0(t, e)p_0(t, e)) (u_1(t, e) - u_0(t, e)) \, de \\
&= \int_{e \in \mathcal{E}} H \left( e, m_0(t, e), \frac{\partial u_0(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad - \int_{e \in \mathcal{E}} H \left( e, m_1(t, e), \frac{\partial u_1(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad + \int_{e \in \mathcal{E}} (m_0(t, e)p_0(t, e)) \left( \frac{\partial}{\partial e} u_1(t, e) - \frac{\partial}{\partial e} u_0(t, e) \right) \, de \\
&\quad - \int_{e \in \mathcal{E}} (m_1(t, e)p_1(t, e)) \left( \frac{\partial}{\partial e} u_1(t, e) - \frac{\partial}{\partial e} u_0(t, e) \right) \, de.
\end{align*}
$$

Let $\forall(t, e), m_{\theta}(t, e) = m_{0}(t, e) + \theta(m_{1}(t, e) - m_{0}(t, e))$ and $u_{\theta}(t, e) = u_{0}(t, e) + \theta(u_{1}(t, e) - u_{0}(t, e))$.

Consider the integral

$$
I(\theta) = \int_{e \in \mathcal{E}} \left[ H \left( e, m_0(t, e), \frac{\partial u_0(t, e)}{\partial e} \right) - H \left( e, m_\theta(t, e), \frac{\partial u_\theta(t, e)}{\partial e} \right) \right] (m_\theta(t, e) - m_0(t, e)) \, de \\
+ \int_{e \in \mathcal{E}} \left( \frac{\partial}{\partial e} u_\theta(t, e) - \frac{\partial}{\partial e} u_0(t, e) \right) (m_0(t, e)p_0(t, e) - m_\theta(t, e)p_\theta(t, e)) \, de.
$$

Next, we write

$$
\begin{align*}
\frac{I(\theta)}{\theta} &= \int_{e \in \mathcal{E}} H \left( e, m_0(t, e), \frac{\partial u_0(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad - \int_{e \in \mathcal{E}} H \left( e, m_\theta(t, e), \frac{\partial \theta(t, e)}{\partial e} \right) (m_1(t, e) - m_0(t, e)) \, de \\
&\quad + \int_{e \in \mathcal{E}} \left( \frac{\partial}{\partial e} u_1(t, e) - \frac{\partial}{\partial e} u_0(t, e) \right) m_0(t, e) p_0(t, e) \, de \\
&\quad - \int_{e \in \mathcal{E}} \left( \frac{\partial}{\partial e} u_1(t, e) - \frac{\partial}{\partial e} u_0(t, e) \right) m_\theta(t, e) p_\theta(t, e) \, de.
\end{align*}
$$

Using the chain rule, we have

$$
\frac{dI(\theta)}{d\theta} = \frac{\partial I(\theta)}{\partial m} \frac{\partial m}{\partial \theta} + \frac{\partial I(\theta)}{\partial c} \frac{\partial m}{\partial \theta}.
$$

By evaluating $\frac{dI(\theta)}{d\theta}$, we can write

$$
\frac{dI(\theta)}{d\theta} = \begin{pmatrix} a & b \end{pmatrix} \begin{pmatrix} c & d \\ e & f \end{pmatrix} \begin{pmatrix} a \\ b \end{pmatrix} \qquad (48)
$$

where $a = m_1(t, e) - m_0(t, e)$, $b = \frac{\partial}{\partial e}u_1(t, e) - \frac{\partial}{\partial e}u_0(t, e)$, $c = -\frac{\partial}{\partial m}H(e, m_0(t, e), \frac{\partial u_0(t, e)}{\partial e})$, $f = -\frac{\partial}{\partial \frac{\partial u}{\partial e}}(m_\theta(t, e)p_\theta(t, e))$, $e = -\frac{\partial}{\partial m}(m_\theta(t, e)p_\theta(t, e))$, and $d = -p_\theta(t, e)$.
<table><caption>TABLE II
SIMULATION PARAMETERS</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>λ<sub>m</sub></td><td>0.00005 base stations/m<sup>2</sup></td></tr><tr><td>λ<sub>s</sub></td><td>50λ<sub>m</sub></td></tr><tr><td>p<sub>m</sub>, p<sub>m,pilot</sub></td><td>43 dBm</td></tr><tr><td>p<sub>max</sub></td><td>10 dBm</td></tr><tr><td>p<sub>s,pilot</sub></td><td>13 dBm</td></tr><tr><td>w<sub>1</sub> , w<sub>2</sub></td><td>1000, 50000</td></tr><tr><td>T</td><td>0.5 s</td></tr></tbody></table>

We can also deduce that $\left. \frac{dI(\theta)}{d\theta} \right|_{\theta=0} = 0$. If $\left. \frac{dI(\theta)}{d\theta} \right|_{\theta=1} \le 0$ and hence $I(1) \le 0$. From equation (47), $\frac{d}{dt} \int_{e \in \mathcal{E}} (u_1(t, e) - u_0(t, e)) (m_1(t, e) - m_0(t, e)) de \le 0$. According to the definition $m_1(0, :) = m_0(0, :)$ and $u_1(T, :) = u_0(T, :)$.

$$ \frac{d}{dt} \int_{e \in \mathcal{E}} (u_1(t, e) - u_0(t, e)) (m_1(t, e) - m_0(t, e)) de = 0. $$

Therefore, if $\begin{pmatrix} c & d \\ e & f \end{pmatrix}$ is negative all the time $u_1(t, e) = u_0(t, e)$ and $m_1(t, e) = m_0(t, e), \forall (t, e) \in [0, T] \times [0, E_{max}]$. Hence, the solution is unique.

<figure><img src="image_4.png" /></figure>

## VII. NUMERICAL RESULTS AND DISCUSSION

This section presents numerical results on the performance of the proposed algorithm. We also validate the stochastic geometry-based expressions derived in Section V-B. The values of the main simulation parameters are given in Table II.

### A. Validating the Expressions Derived by Stochastic Geometry Analysis

First, we validate the expressions derived by stochastic geometry analysis. To validate the average interference given in equation (26), we have only considered the interference caused due to the SBSs (i.e., only one PPP is considered for simulation). The same result would hold for the interference caused by the macro network. A comparison of the simulation results with the expression in equation (26) is shown in Fig. 3. In Fig. 4, we validate the expression for the average distance to the closest possible macro user
<figure><img src="image_5.png" /><figcaption>Fig. 3. Average interference experienced by a generic small cell user (for $\lambda_s = 50\lambda_m$).</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Variation of the distance to the closest edge of an SBS with $\lambda_m$.</figcaption></figure>

given in equation (32). The exact match of the theoretical and simulation results validates the accuracy of the derived expressions.

## B. Behavior of Mean Field at Equilibrium

In this section we observe the behavior of the mean field at the equilibrium. First, we set $E_{max} = 0.1$ J, $p_{max} = 0.01$ W, and $T = 0.5$ (i.e., 50 LTE frames). The initial energy distribution $m(0, :)$ is assumed to be uniform. The mean field at the equilibrium for cost function $c_{t,e}$ is shown in Fig. 5.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Mean field at the equilibrium for $c(t, e)$ with uniform initial energy distribution.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6. Cross-section of the mean field at equilibrium for $c(t, e)$.</figcaption></figure>
It can be seen from the figure that the number of SBSs with higher energy levels decreases with time. The probability of base stations having zero energy increases at the beginning of the time frame and later settle to a constant. This means, although some SBSs empty their battery while transmission, all SBSs do not empty their batteries. This is because, the quadratic term (i.e., $f^{(1,mean)}$ in equation 21) of cost function $c$ discourages the SBSs to increase their transmit power after satisfying the QoS constraint. Therefore, the SBSs which start transmission with higher initial energy do not empty their batteries throughout the transmission.

For illustration, we also plot several cross-sections of the mean field in Fig. 6, which shows the variation of the probability distribution of SBSs having a certain energy with time. Since the initial distribution is uniform, the initial probabilities are similar for all energy levels. After the transmission starts, there is no SBS with full energy as everybody transmits with non-zero power. Therefore, the probability of SBSs with maximum energy (i.e., 0.1 J) drops to zero right after the start of the transmission. The probability of SBSs having zero energy increases for sometime, as SBSs who had smaller initial energy would eventually empty their batteries.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Mean field at the equilibrium for $\hat{c}(t, e)$ with uniform initial energy distribution.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 8. Cross-section of the mean field at equilibrium for $\hat{c}(t, e)$.</figcaption></figure>

In Fig. 7 and Fig. 8, we show the MFE considering $\hat{c}(t, e)$ in equation (36). $\hat{w}_1$ and $\hat{w}_2$ are set to 1. Unlike in the previous cost function $c(t, e)$, this cost function does not discourage SBSs to increase transmit power after satisfying the QoS constraint. Therefore, the SBSs tend to use more energy during $T$ and result a different mean field behavior. (Note that in this case, there are SBSs with a higher energy
than the previous case as $E_{max} = 2J$.) It can be seen in Fig. 7 that, the probability of an SBS having zero energy is equal to one at the end of the time period $T$ (i.e., $m(T; 0) = 1$). This means all the SBSs have emptied their energy allowance during the transmission and have zero available energy at the end of the considered time frame $T$. Therefore, it can be concluded that the cost function $c(t, e)$ performs better than the cost function $\hat{c}(t, e)$ interms of energy saving.

### C. Power Control Policy at the Mean Field Equilibrium

We show the transmit power policies for the game $G_m$ with both cost functions $c(t, e)$ and $\hat{c}(t, e)$. Once the power policy is calculated, an SBS can decide on the transmit power based on its available energy at each time instant. Re-computation of the power policy is needed at the beginning of each time interval $T$ (i.e., 0, T, 2T, 3T, ...) only if the probability distribution of allowable energy changes.

<figure><img src="image_11.png" /><figcaption>Fig. 9. Equilibrium power policy for $\hat{c}(t, e)$ with uniform initial energy distribution.</figcaption></figure>

Fig. 9 shows the equilibrium power policy for the cost function $c(t, e)$. A uniform initial energy distribution is considered. All SBSs start transmission at low power levels. The SBSs with lower energy may empty their batteries after sometime decreasing the average interference caused to the other users. Then, the SBSs, which have sufficient energy to transmit throughout $T$, increase their transmit power. As the cost function $c(t, e)$ discourage the SBSs to increase power after satisfying the QoS constraint,
the transmit power remains almost constant. However, the cost function $\hat{c}(t, e)$ results a different system behavior.

<figure><img src="image_12.png" /><figcaption>Fig. 10. Equilibrium power policy for $\hat{c}(t, e)$ with uniform initial energy distribution.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 11. Cross-section of the power policy for $\hat{c}(t, e)$.</figcaption></figure>

Fig. 10 shows the transmit power policy at the equilibrium for cost function $\hat{c}$. We also consider a uniform distribution of initial energy. This figure also shows that, the SBSs with higher energy start transmitting with maximum allowable transmit power while the SBSs with lower energy start with lower power. However, the SBSs with lower energy tend to increase their transmit power after some time. A cross-section of the power policy plot is shown in Fig. 11 for energy levels $2J, 0.2J, 0.05J$, and $0J$. The figure shows that the SBSs with higher energy start transmitting with maximum allowable transmit power while SBSs with lower energy start with lower power. However, the SBSs with lower energy tend to increase their transmit power after some time.

The above phenomenon is illustrated more in Fig. 12 where we show the transmit power policies with three different initial energy levels. The SBSs who start the game with an initial energy of $0.05J$ do not transmit at higher power at the beginning of the time period $T$. They increase the transmit power later in the time slot. By that time, the SBSs who started the game with higher energy have spent most of their energy and lowered their transmit power. The SBSs with less initial energy can have a better cost by increasing their transmit power later in time period $T$ due to reduced interference.

## D. Comparison With Uniform Transmit Power

To compare the performance of the proposed algorithm, we use uniform transmit power setting as a benchmark algorithm. In this case, the uniform transmit power $p_k$ of an SBS $k$ with initial energy $e_k(0)$
<figure><img src="image_14.png" /><figcaption>Fig. 12. Transmit power variation of SBSs with different initial energy for $\hat{c}(t, e)$.</figcaption></figure>

equals to $\frac{e_k(0)}{T}$. Fig. 13 plots the variation of average SINR over $T$ with $\lambda_s$ for both uniform transmit power setting and the proposed algorithm for cost function $\hat{c}(t, e)$. The results show that the transmit power policy given by the proposed algorithm performs better when the network becomes more dense. The variation of average SINR over $T$ with $\lambda_s$ for $c(t, e)$ is compared with the uniform transmit power policy in Fig. 14. Also in this case also the proposed algorithm outperforms the uniform power policy. However, SINR does not increase after satisfying the QoS constraint.

VIII. CONCLUSION

We have proposed an energy-aware distributed power control algorithm for self-organizing small cell networks. The power control problem of a co-channel deployed small cell network underlaid with a macro network is first formulated as a stochastic game. The stochastic game for power control is then extended to a mean field game for a dense networks. An iterative finite difference technique is proposed to solve mean field equations based on Lax-Friedrichs scheme and Lagrange relaxation. We also have shown the sufficient conditions for the uniqueness of the mean field equilibrium. The performance of the algorithm has been analyzed for two cost functions. The main advantage of the proposed algorithm is that it can be distributively executed offline. Also, the algorithm considers minimizing the cost over a pre-defined period
<figure><img src="image_15.png" /><figcaption>Fig. 13. Variation of SINR at the receiver of a generic user with SBS density.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 14. Variation of SINR at the receiver of a generic user with SBS density.</figcaption></figure>

of time, instead of minimizing the running cost. Numerical results have been presented to demonstrate
the performance of the proposed algorithm.

REFERENCES

[1] E. Hossain, M. Rasti, H. Tabassum, and A. Abdelnasser, "Evolution toward 5g multi-tier cellular wireless networks: an interference management perspective," *IEEE Wireless Communications*, vol. 21, pp. 118–127, June 2014.
[2] D. López-Pérez, A. Valcarce, G. De La Roche, and J. Zhang, “Ofdma femtocells: a roadmap on interference avoidance,” *IEEE Communications Magazine*, vol. 47, pp. 41–48, 2009.

[3] H. Claussen, L. T. Ho, and L. G. Samuel, “An overview of the femtocell concept,” *Bell Labs Technical Journal*, vol. 13, pp. 221–245, 2008.

[4] M. Bennis, L. Giupponi, E. M. Diaz, M. Lalam, M. Maqbool, E. C. Strinati, A. De Domenico, and M. Latva-aho, “Interference management in self-organized femtocell networks: The befemto approach,” in *2nd International Conference on Wireless Communication, Vehicular Technology, Information Theory and Aerospace & Electronic Systems Technology*. IEEE, 2011.

[5] J.-H. Yun and K. G. Shin, “Ctrl: a self-organizing femtocell management architecture for co-channel deployment,” in *Proceedings of the sixteenth annual international conference on Mobile computing and networking*. ACM, 2010, pp. 61–72.

[6] J.-M. Lasry and P.-L. Lions, “Mean field games,” *Japanese Journal of Mathematics*, vol. 2, pp. 229–260, 2007.

[7] M. Huang, P. E. Caines, and R. P. Malhamé, “Large-population cost-coupled lqg problems with nonuniform agents: individual-mass behavior and decentralized $\epsilon$-nash equilibria,” *IEEE Transactions on Automatic Control*, vol. 52, pp. 1560–1571, 2007.

[8] M. Huang, R. P. Malhame, and P. E. Caines, “Large population stochastic dynamic games: closed-loop mkeean-vlasov systems and the nash certainty equivalence principle,” *Communications in Information & Systems*, vol. 6, pp. 221–252, 2006.

[9] O. Guéant, J.-M. Lasry, and P.-L. Lions, “Mean field games and applications,” in *Paris-Princeton Lectures on Mathematical Finance*. Springer, 2011, pp. 205–266.

[10] H. Tembine, R. Tempone, and P. Vilanova, “Mean field games for cognitive radio networks,” in *American Control Conference (ACC)*. IEEE, 2012, pp. 6388–6393.

[11] R. Couillet, S. M. Perlaza, H. Tembine, and M. Debbah, “Electrical vehicles in the smart grid: a mean field game analysis,” *IEEE Journal on Selected Areas in Communications*, vol. 30, pp. 1086–1096, 2012.

[12] F. Mériaux, V. Varma, and S. Lasaulce, “Mean field energy games in wireless networks,” in *Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR)*. IEEE, 2012, pp. 671–675.

[13] D. Bauso, B. M. Dia, B. Djehiche, H. Tembine, and R. Tempone, “Mean-field games for marriage,” *PloS one*, vol. 9, p. e94933, 2014.

[14] I. W. Mustika, K. Yamamoto, H. Murata, and S. Yoshida, “Potential game approach for self-organized interference management in closed access femtocell networks,” in *IEEE 73rd Vehicular Technology Conference (VTC Spring)*. IEEE, 2011.

[15] P. Semasinghe, E. Hossain, and K. Zhu, “An evolutionary game approach for distributed resource allocation for self-organizing small cells,” 2013, to appear.

[16] X. Kang, Y.-C. Liang, and H. K. Garg, “Distributed power control for spectrum-sharing femtocell networks using stackelberg game,” in *IEEE International Conference on Communications*. IEEE, 2011, pp. 1–5.

[17] S. Guruacharya, D. Niyato, D. I. Kim, and E. Hossain, “Hierarchical competition for downlink power allocation in ofDMA femtocell networks,” *IEEE Transactions on Wireless Communications*, vol. 12, pp. 1543–1553, 2013.

[18] M. Bennis and S. M. Perlaza, “Decentralized cross-tier interference mitigation in cognitive femtocell networks,” in *Communications (ICC), 2011 IEEE International Conference on*. IEEE, 2011, pp. 1–5.

[19] D. T. Ngo, L. B. Le, T. Le-Ngoc, E. Hossain, and D. I. Kim, “Distributed interference management in two-tier cdma femtocell networks,” *IEEE Transactions on Wireless Communications*, vol. 11, pp. 979–989, 2012.

[20] F. Mériaux, S. Lasaulce, and H. Tembine, “Stochastic differential games and energy-efficient power control,” *Dynamic Games and Applications*, vol. 3, pp. 3–23, 2013.

[21] F. Mériaux and S. Lasaulce, “Mean-field games and green power control,” in *5th International Conference on Network Games, Control and Optimization (NetGCoop)*. IEEE, 2011.
[22] H. Tembine, S. Lasaulce, and M. Jungers, "Joint power control-allocation for green cognitive wireless networks using mean field theory," in *Proceedings of Fifth International Conference on Cognitive Radio Oriented Wireless Networks & Communications (CROWNCOM)*. IEEE, 2010.

[23] A. Okabe, B. Boots, K. Sugihara, and S. N. Chiu, *Spatial tessellations: concepts and applications of Voronoi diagrams*. John Wiley & Sons, 2009, vol. 501.

[24] M. Huang, P. E. Caines, and R. P. Malhamé, "Uplink power adjustment in wireless communication systems: a stochastic control analysis," *IEEE Transactions on Automatic Control*, vol. 49, pp. 1693-1708, 2004.

[25] M. Huang, P. Caines, and R. P. Malhamé, "Individual and mass behaviour in large population stochastic wireless power control problems: centralized and nash equilibrium solutions," in *Proceedings of 42nd IEEE Conference on Decision and Control*, vol. 1. IEEE, 2003, pp. 98-103.

[26] T. Basar, G. J. Olsder, G. Clsder, T. Basar, T. Baser, and G. J. Olsder, *Dynamic noncooperative game theory*. SIAM, 1995, vol. 200.

[27] Z. Han, *Game theory in wireless and communication networks: theory, models, and applications*. Cambridge University Press, 2012.

[28] R. Bellman, "Dynamic programming and lagrange multipliers," *Proceedings of the National Academy of Sciences of the United States of America*, vol. 42, no. 10, p. 767, 1956.

[29] M. Burger and J. M. Schulte, "Adjoint methods for hamilton-jacobi-bellman equations," 2010.

[30] R. Bellman, *Dynamic Programming*. Princeton University Press, 1957.

[31] M. Bardi and I. Capuzzo-Dolcetta, *Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations*. Springer, 2008.

[32] H. Soner, "Controlled markov processes, viscosity solutions and applications to mathematical finance," *Viscosity Solutions and Applications*, pp. 134-185, 1997.

[33] L. C. Evans, "Partial differential equations," 1998.

[34] O. Guéant, "A reference case for mean field games models," *Journal de mathématiques pures et appliquées*, vol. 92, pp. 276-294, 2009.

[35] F. Baccelli and B. Blaszczyszyn, *Stochastic Geometry and Wireless Networks: Volume 1: THEORY*. Now Publishers, Inc., 2009, vol. 1.

[36] M. Haenggi, J. G. Andrews, F. Baccelli, O. Dousse, and M. Franceschetti, "Stochastic geometry and random graphs for the analysis and design of wireless networks," *IEEE Journal on Selected Areas in Communications*, vol. 27, no. 7, pp. 1029-1046, 2009.

[37] J. G. Andrews, F. Baccelli, and R. K. Ganti, "A tractable approach to coverage and rate in cellular networks," *IEEE Transactions on Communications*, vol. 59, no. 11, pp. 3122-3134, 2011.

[38] P. Semasinghe and E. Hossain, "Energy-aware downlink power control in self-organizing dense small cell networks: a mean field game," 2015, submitted to IEEE ICC.

[39] R. Ahmed, "Numerical schemes applied to the burgers and buckley-leverett equations," Ph.D. dissertation, Department of Mathematics, University of Reading, 2004.

[40] M. Carter, "The fredholm alternative and the lagrange multiplier theorem," Supplementary note for foundations of mathematical economics, 2001.