# A Reinforcement Learning Based Intercell Interference Coordination in LTE Networks
## 5 The Proposed ICIC Scheme
### Resource Allocation Step  
This step is responsible for allocating resource blocks to users selected in the previous step. In this step, policy space based reinforcement learning is used instead of value function space to allocate RBs to users selected in the previous step. Policy space search methods use explicit policies and modify them through search operators [41]. We use GA as a policy space search, where a policy is represented by a chromosome. An agent interacts with its environment and learns the best policy to allocate RBs to users. The reinforcement learning agent-environment interaction [13] for resource allocation is represented in Figure 2, where \(S_{t}\) is a representation of the state at time \(t\) and \(A_{t}\) represents action taken by an agent at time \(t\), which leads to the state \(S_{t+1}\) after receiving reward \(R_{t+1}\) at time \(t+1\). \(R_{t}\) represents reward received from previous action. There are two possible rewards: \(-1\) and \(1\). The environment is composed of the following parameters: user buffer status and user head of line delay (HoL). A state represents the number of users with their HoL, their buffer status, the overall packet loss ratio, and the QoS be  
Figure 2: Reinforcement learning agent–environment interaction for resource allocation. An agent is a packet scheduler located at each enodeB, thus we have a multi-agent environment. A policy is an action taken by an agent in a given state. An action chooses for each RB a user that the RB will be allocated to. For simplicity, a RB will carry one packet. A reward is then associated on each action taken by the agent. If action taken by an agent increases the packet loss ratio, a negative reward is given to the agent. If the action taken by an agent decreases the packet loss ratio, a positive reward is given to the agent. The two reward values are \(-1\) and \(+1\). A single policy represents a chromosome [41]. A good policy should have a fitness value below a given threshold corresponding to the QoS requirements of a given service. The fitness function for the resource allocation GA algorithm is the sum of HoL. A RB represents a gene. Its value corresponds to a user that the RB have been allocated to. A chromosome is a set of available RBs that are allocated each time to interval (TTI). Each gene will be encoded in decimal to ease the computational process. The size of the population will not vary. Elitism strategy is used to keep the best solutions and the least feasible solutions are been discarded. A sample chromosome is shown in Figure 3 that represents a policy used for a sub agent located in a cell with a bandwidth that have 25 RBs and 60 actives users. The numbers represent users. We can see that User 60 has been allocated two non-contiguous RBs, User 9 has been allocated two non-contiguous RBs, User 25 has been allocated two contiguous RBs, and User 16 has been allocated two non-contiguous RBs. The resource allocation can be modeled by a matrix where each row represents cells and each column represents a resource block to be allocated. A sample resource allocation matrix can be seen in Table 1 for a network composed of n cells and a bandwidth corresponding to m RBs where \(Cell_{i}\) identifies cells and \(RB_{i}\) identifies resource blocks. For example, \(Cell_{1}\) is the first cell and \(RB_{1}\) is the first resource block. \(UE_{ij}\) represents user \(i\) in cell \(j\). A vector is used for chromosome representation to ease GA operations crossover and mutation. An initial population of policies is generated at the beginning of the reinforcement learning process. An agent evaluates each policy through policy evaluation. A fitness function is used to do this. Each policy is evaluated and improved to obtain the best policy. The best policy will be the policy that delivers packet loss ratio value less than or equal to the value of the maximum packet loss acceptable according to the QoS requirements [39].