# A Reinforcement Learning Based Intercell Interference Coordination in LTE Networks
## 5 The Proposed ICIC Scheme
### Resource Allocation Step
The population of policies is modified through crossover and mutation operations over a defined number of generations with the aim of solutions converging to the best solution. This corresponds to the learning process during which policies are improved. During crossover operation, some chromosomes are selected according to the crossover probability and divided into two groups. Pairs of chromosomes are composed by choosing one chromosome on \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \(RB_{1}\) & \(RB_{2}\) &... & \(RB_{m-1}\) & \(RB_{m}\) \\ \(Cell_{1}\) & \(UE_{11}\) & \(UE_{12}\) &... & \(UE_{1(m-1)}\) & \(UE_{1m}\) \\ \(Cell_{2}\) & \(UE_{21}\) & \(UE_{22}\) &... & \(UE_{2(m-1)}\) & \(UE_{2m}\) \\... &... &... &... &... &... \\ \(Cell_{n-1}\) & \(UE_{(n-1)1}\) & \(UE_{(n-1)2}\) &... & \(UE_{(n-1)(m-1)}\) & \(UE_{(n-1)m}\) \\ \(Cell_{n}\) & \(UE_{n1}\) & \(UE_{n2}\) &... & \(UE_{n(m-1)}\) & \(UE_{nm}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Resource allocation matrix. Figure 3: Policy example for 25 Resource Blocks and 60 active users. The numbers inside the blocks indicate users. each of the two groups to be crossed in order to produce new chromosomes. An example of crossover operation is shown in Figure 4. The number of RBs of each chromosome is 25. In this example, two chromosomes, namely Child 1 and Child 2, are obtained from two chromosomes, namely Parent 1 and Parent 2. During mutation operation, some chromosomes are selected according to the mutation probability to be mutated. Six genes are randomly chosen to be mutated on each chromosome selected for mutation to add diversity to population. A gene value represents a user to whom the corresponding RB is allocated. Mutation of a gene consists of changing its value. In other words, mutating a gene means deallocating a RB from a user and allocating it to another user. The RB is allocated to a user in the waiting list, which contains users selected in the first step of the proposed ICIC scheme but have not been selected to form the initial population of chromosomes. An example of mutation operation is shown in Figure 5. An initial chromosome is replaced (mutated) to obtain a mutated chromosome by changing the color of genes, which means deallocating RBs previously allocated to users associated to these genes from the initial chromosome and allocating these RBs to users in the waiting list. The best policy is obtained after selection, crossover, mutation and evaluation have been repeated a certain number of generations, and then the learning process terminates. Elitism strategy is used for chromosomes selection and the population size is kept constant. Chromosomes are sorted according to their fitness function value after crossover and mutation operations. Since crossover and mutation operations increase the population number, selection operation ignore will retain only a number of solutions equal to the number of the initial population and discard the remaining solutions. Retained solutions are chosen among the best solutions. The whole process is summarized in Algorithm 2. Details are shown in Algorithm 3. Algorithm 4 details the update of buffers and HoL as well as packet loss calculation operation in Algorithm 3. Figure 4: Example of crossover operation. Figure 5: Example of mutation operation.