Dynamic scheduler selection using deep reinforcement learning
Creators: name: Hall, James
Publisher: name: University of Surrey
Year: 2023
Subjects: Scheduling
Reinforcement Learning
Multi-Armed Bandit

description: Future network deployments will rely on dense networks of cells consisting of both macro cells and small cells. The deployment of a large number of cells will require rapid deployment of new cells. Rapid deployment of cells will require a reduction in the cost and time invested into network planning.<br>
description: <br>
description: The scheduler is responsible for assigning resources to users directly. As a result, the scheduler has significant influence on many aspects of the network performance, including the user’s service and the inter-cell interference. Schedulers are not subject to standardisation by important bodies such as the 3GPP. As such, there exists a large
description: and varied group of scheduling techniques, each of which is suited for differing user deployment, network scenario or traffic type.<br>
description: <br>
description: Distributed scheduling methods focus on serving the requirements of the users in the cells they operate in. Many of these methods focus on sharing the cell’s resources fairly between the users. There are also a subset of these methods, known as margin adaptive schedulers, that look to reduce the number of resources used by each cell while still fulfilling user requirements. Such methods reduce the inter-cell interference that they cause to their neighbours and have been shown to converge to frequency reuse patterns. However, the deployment of distributed scheduling methods is difficult to coordinate as each method is typically only tested in networks consisting of the same scheduling methods. The result of this is that distributed scheduling methods are often distributed homogeneously across a network.<br>
description: <br>
description: Coordinated scheduling methods schedule multiple cells simultaneously. This solves the issue of homogeneous deployment of schedulers. However, these methods require a certain cloud radio access network configuration that requires the functional split between the central unit and the distributed cells to centralise the MAC functional layer.<br>
description: <br>
description: Studies on user traffic patterns have shown that the traffic on the network encounters temporal variations. As each scheduling method is suited to different traffic and user scenarios, this can result in a dynamic change in the type of scheduling that performs best in the network. Dynamic schedulers have been proposed to address this. These methods change the scheduling policy within a cell depending on the traffic in each cell on the network. However, these methods affect only a single cell and consider only the traffic and QoS requirements of users in that cell.<br>
description: <br>
description: As a result we start our examination by exploring the effects of heterogeneous scheduling deployment on the user’s quality of service. Specifically, we simulate a network area where 2 scheduling methods have been deployed in a heterogeneous manner; a proportional fair scheduler and a margin adaptive scheduler under various user traffic conditions. These experiments find that the heterogeneous scheduling policy achieved the best satisfaction rate for users at 98.14% against the best homogeneous satisfaction rate of 88.51%. Conversely, heterogeneous scheduling policy can reduce the quality of service for users in cases where the scheduling policy acts against the user distribution achieving a minimum satisfaction rate of 20.74% against the minimum satisfaction rate for homogeneous scheduling of 38.69%.<br>
description: <br>
description: We then define the dynamic scheduler selection problem, where a central entity called the scheduler selection entity is able to dynamically select scheduling methods for each cell in a network area that maximises the quality of service provided to users on the network. We propose 2 schemes for dynamic scheduler selection: the first based on a nature inspired genetic algorithm and the second based on deep reinforcement learning. This study finds that the reinforcement learning based scheduler selection technique can successfully identify the scheduling policies that improve the quality of service of users in the network in 70% of the scenarios simulated.<br>