# A Reinforcement Learning Based Intercell Interference Coordination in LTE Networks
## 5 The Proposed ICIC Scheme
### Dynamic Power Allocation Step  
An agent learns to find the best policy, which maximizes SINR values for each user on each cell. The environment of the agent is a multi-cell network composed of enodeBs and users' geographic locations. A state represents the location of a user and its SINR value. Only users that have been allocated at least one physical resource block are considered. The agent is located at a centralized unit that may be located on one of the enodeBs and collects geographic location of users on each enodB. The agent learns the best policy to set transmit power for each enodeB on each physical resource block allocated to users based on their locations. A reward is associated to each action taken by the agent. If the action taken by an agent is such that the obtained SINR is less than the minimum SINR acceptable, a negative (\(-1\)) reward is given to the agent. If the action taken by an agent is such that the obtained SINR is greater than the minimum SINR, a positive reward (\(+1\)) is given to the agent. The reinforcement learning agent-environment interaction [13] is presented in Figure 6 where \(S_{t}\), \(A_{t}\), \(S_{t+1}\), \(R_{t+1}\) and \(R_{t}\) have been described in the resource allocation step. A single policy represents a chromosome [41]. The transmit power of an enodeb on a RB represents a gene. A chromosome is a set of transmit power on each enodeB on each RB. Each policy is evaluated by the minimum SINR obtained with the corresponding enodeB's transmit power. A gene represents an enodeB transmit power on a RB. A chromosome sample is shown in Table 2.  
The power allocation problem can be seen as a matrix represented in Table 3 for a network composed by n cells with a bandwidth corresponding to m RBs. In Tables 2 and 3, \(P_{ij}\) represents transmit power of enodeB in cell \(i\) on resource block \(RB_{j}\). For example, \(P_{11}\) is the transmit power of enodeBin cell \(1\) on the first resource block and \(P_{21}\) is the transmit power of enodeBin cell \(2\) on the first resource block. An initial population of policies is generated at the beginning of the learning process. Each gene in a policy is assigned randomly a transmit power value between the minimum transmit power and the maximum transmit power. An agent will evaluate each policy through policy evaluation. The minimum SINR is used as fitness function of the power allocation GA. Precisely, the range of SINR values is used as fitness function. To lower the computation process time, a SINR is calculated for each \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \(P_{11}\) & \(P_{12}\) &... & \(P_{1m}\) & \(P_{21}\) &... & \(P_{2m}\) &... & \(P_{n1}\) &... & \(P_{nm}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Power allocation chromosome. Figure 6: Reinforcement learning agent–environment interaction of the power allocation problem. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \(P_{11}\) & \(P_{12}\) &... & \(P_{1m}\) & \(P_{21}\) &... & \(P_{2m}\) &... & \(P_{n1}\) &... & \(P_{nm}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Power allocation matrix. user only on RBs that have been allocated to the user. Each policy is evaluated during a process where the initial population is modified through crossover and mutation operations. A good policy is the one that has a minimum SINR value greater than the minimum acceptable SINR value. Policies evaluation is repeated many times corresponding to a number of generations in order to find the best policy. Policies are improved during policies evaluation. Crossover operation is done similar to the one in resource allocation step. An example of crossover operation is shown in Figure 7 for a power allocation in a network that has three cells and a bandwidth of 5 MHz corresponding to 25 RBs. Mutation operation is done similar to the one in resource allocation step. An example of mutation operation is shown in Figure 8 for a three-cell network with 25 RBs to allocate on each cell.