CENTRAL NODE AND A METHOD FOR REINFORCEMENT LEARNING IN A RADIO ACCESS NETWORK

A method performed by a central node for controlling an exploration strategy associated to Reinforcement Learning, RL, in one or more RL modules in a distributed node in a Radio Access Network, RAN, is provided. The central node evaluates a cost of actions performed for explorations in the one or more RL modules, and a performance of the one or more RL modules. Based on the evaluation, the central node determines one or more exploration parameters associated to the exploration strategy. The central node controls the exploration strategy by configuring the one or more RL modules with the determined one or more exploration parameters to update its exploration strategy, enforcing the respective one or more RL modules to act according to the updated exploration strategy to produce data samples for the one or more RL modules in the distributed node.

TECHNICAL FIELD

Embodiments herein relate to a central node and a method therein. In some aspects they relate to controlling an exploration strategy associated to Reinforcement Learning (RL) in one or more RL modules in a distributed node in a Radio Access Network (RAN).

Embodiments herein further relates to computer programs and carriers corresponding to the above method, and central node.

BACKGROUND

In a typical wireless communication network, wireless devices, also known as wireless communication devices, mobile stations, stations (STA) and/or User Equipment (UE), communicate via a Local Area Network such as a Wi-Fi network or a Radio Access Network (RAN) to one or more core networks (CN). The RAN covers a geographical area which is divided into service areas or cell areas, which may also be referred to as a beam or a beam group, with each service area or cell area being served by a radio network node such as a radio access node e.g., a Wi-Fi access point or a radio base station (RBS), which in some networks may also be denoted, for example, a NodeB, eNodeB (eNB), or gNB as denoted in 5G. A service area or cell area is a geographical area where radio coverage is provided by the radio network node. The radio network node communicates over an air interface operating on radio frequencies with the wireless device within range of the radio network node.

Specifications for the Evolved Packet System (EPS), also called a Fourth Generation (4G) network, have been completed within the 3rd Generation Partnership Project (3GPP) and this work continues in the coming 3GPP releases, for example to specify a Fifth Generation (5G) network also referred to as 5G New Radio (NR) or Next Generation (NG). The EPS comprises the Evolved Universal Terrestrial Radio Access Network (E-UTRAN), also known as the Long Term Evolution (LTE) radio access network, and the Evolved Packet Core (EPC), also known as System Architecture Evolution (SAE) core network. E-UTRAN/LTE is a variant of a 3GPP radio access network wherein the radio network nodes are directly connected to the EPC core network rather than to RNCs used in 3G networks. In general, in E-UTRAN/LTE the functions of a 3G RNC are distributed between the radio network nodes, e.g. eNodeBs in LTE, and the core network. As such, the RAN of an EPS has an essentially “flat” architecture comprising radio network nodes connected directly to one or more core networks, i.e. they are not connected to RNCs. To compensate for that, the E-UTRAN specification defines a direct interface between the radio network nodes, this interface being denoted the X2 interface.

Deep Reinforcement Learning (RL)

A neural network is essentially a Machine Learning model, more precisely, Deep Learning, that is used in both supervised learning and unsupervised learning. A Neural Network is a web of interconnected entities known as nodes wherein each node is responsible for a simple computation.

RL is a powerful technique to efficiently learn a behavior of a system within a dynamic environment. By incorporating recent advances in deep artificial neural networks, deep RL (DRL) has been shown to enable significant autonomy in complex real-world tasks. DRL uses deep learning and reinforcement learning principles to create efficient algorithms applied on areas like robotics, video games, computer science, computer vision, education, transportation, finance, healthcare, etc. As a result, DRL approaches are quickly becoming state-of-the-art in robotics and control, online planning, and autonomous optimization.

Despite its significant success, the intuition behind DRL is relatively simple. For an observed environment state, a DRL agent attempts to learn the optimal action by exploring the space of available actions. For an observed state ‘S[t]’ at time ‘t’, the DRL agent selects an action ‘a[t]’ that is predicted to maximize the cumulative discounted rewards over the next several time intervals. The heuristically-configured discounting factor avoids actions that maximize the immediate, short-term, reward but lead to poor states in the future. After taking an action, the DRL agent feeds back the reward into a learning module, typically a neural network, which learns to make better action choices in subsequent time intervals.

At the beginning of its operation, DRL agent has incomplete, often zero, knowledge of the system. Depending on the tolerance of the system to occasional failures, the agent may either choose to collect data for offline learning through an existing policy, which is safer, or select actions online in some randomized manner, which is efficient. In either case, the collected data is used to iteratively update the model, for example the weight and bias variables within a neural network. The training parameters, such as the size of the neural network, number of iterative updates, and parameter update scheme are all configured heuristically based on empirical findings from state-of-the-art DRL implementations. As the DRL agent learns the true value of actions over time, the need for exploring random actions decreases as well. This decrease is encoded in an exploration rate variable whose value is slowly reduced to nearly zero with time.

Majority of radio network management and optimization problems are about tuning parameters to adapt to local propagation environment, traffic patterns, service types and UE device capabilities. DRL is a promising technique to automate such tuning. In the context of radio networks, DRL has recently been proposed for several challenging cellular network problems, ranging from data rate selection, beam management, to trajectory optimization for aerial base stations.

Machine Learning Architectures in Radio Networks

A radio network consists for multiple distributed base stations. The RL policy may be trained and/or inferred in a centralized, distributed or hybrid manner.FIG.1a, bandcdepict three RL architectures in a radio network such as a RAN where the RL model training and inference take place in different locations.FIG.1aillustrates distributed learning,FIG.1billustrates centralized learning local inference, andFIG.1cillustrates hybrid learning.

FIGS.1a, bandcdepict a global data pipeline200, a Data pipeline for Local distributed node1referred to as201a, a data pipeline for Local distributed node n referred to as201n.

Further a Training for global node210, a Training for local distributed node1referred to as211aand a Training for local distributed node n referred to as211n, an Inference for local distributed node1referred to as221a, and an inference for local distributed node n referred to as221n.

Yet further, a Global Training orchestrator, e.g. a learning orchestrator, referred to as230, a Distributed node1referred to as222aand a Distributed node n referred to as222n.

Solid lines illustrate data movement of training data. Dotted lines illustrate model deployments, i.e. from trained models to inference using the trained models. Dashed lines illustrate the communication of model weights and training also referred to as learning, hyper parameters.

In the distributed learning architecture inFIG.1a, both training and inference are located in the distributed nodes. One advantage of this architecture is the low inference latency especially for latency critical applications.

Since the memory and computation power of the distributed nodes are usually limited, the training can be moved to a central node as shown in the centralized learning local inference architecture inFIG.1b. Another advantage of this solution is the higher amount of training data collected from the multiple distributed nodes.

The hybrid learning architecture inFIG.1cprovides different dynamics between the central and distributed nodes. In this scheme, a central learning orchestrator controls or instructs the training and inference in the distributed nodes.

E-UTRAN and NG-RAN Architecture Options

The current 5G RAN (NG-RAN) architecture is depicted and described in 3GPP TS 38.401v15.4.0 as follows. Mapped to the RL architecture, centralized learning functions may be located in either Fifth Generation Core network (5GC) or gNB-Central Unit (CU), and gNB-Distributed Unit (DU) is an example of the distribute node.

FIG.2depicts an overall architecture of NG architecture. The NG architecture may be further described as follows. The NG-RAN comprises a set of gNBs connected to the 5GC through the NG. A gNB can support FDD mode, TDD mode or dual mode operation. gNBs can be interconnected through the Xn interface. A gNB may comprise a gNB-CU and one or more gNB-DUs. A gNB-CU and a gNB-DU are connected via F1 logical interface. One gNB-DU is connected to only one gNB-CU. For resiliency, a gNB-DU may be connected to multiple gNB-CU by appropriate implementation. NG, Xn and F1 are logical interfaces. The NG-RAN is layered into a Radio Network Layer (RNL) and a Transport Network Layer (TNL). The NG-RAN architecture, i.e., the NG-RAN logical nodes and interfaces between them, is defined as part of the RNL. For each NG-RAN interface, NG, Xn, and F1, the related TNL protocol and the functionality are specified. The TNL provides services for User Plane (UP) transport and signalling transport.

A gNB may also be connected to an LTE eNB via the X2 interface. In this architectural option an LTE eNB connected to the Evolved Packet Core network is connected over the X2 interface with a so called nr-gNB. The latter is a gNB not connected directly to a CN and connected via X2 to an eNB for the sole purpose of performing dual connectivity.

In yet another architecture option a gNB may be connected to an eNB via an Xn interface. In this option both gNB and eNB are connected to the 5GC and can communicate over the Xn interface.

It is worth noticing that RAN nodes can not only communicate via direct interfaces such as the X2 and Xn but also via CN interfaces such as the NG and S1 interfaces. Such communication requires the involvement of CN nodes and/or transport nodes (such as IP packet routers, Ethernet switches, microwave links or optical ROADMs) to route and forward messages from the source RAN node to the target RAN node.

The architecture inFIG.2can be expanded by spitting the gNB-CU into two entities. One gNB-CU-UP, which serves the user plane and hosts the Packet Data Convergence Protocol (PDCP) protocol, and one gNB-CU-Control Plane (CP), which serves the control plane and hosts the PDCP and Radio Resource Control (RRC) protocol. For completeness it should be mentioned that a gNB-DU hosts the Radio Link Control (RLC) protocol, the Medium Access Control (MAC) protocol and the Physical Layer (PHY) protocol.

RL Exploration and Exploitation in Radio Networks

One challenge about the RL technique, comparing with rule-based methods, is the risk of significant performance degradation in the radio network when taking random actions. For example, performance degradation in the form of coverage holes might be a result of an action of reducing cell transmission power. Such risks are rooted in the way a RL agent explores the environment.

The balance between exploration and exploitation is a key aspect of RL when deciding which action to take. While exploitation is about taking advantage of the learning in the past, exploration is a procedure to learn new knowledge, e.g. by taking random actions and observing the consequences. Usually, a RL agent applies a high exploration rate in the beginning phase of learning when the policy has only been trained with limited amount of data samples. As the training continues and the trained policy becomes more reliable, the exploration rate is gradually reduced to a value close to zero.

One way to reduce the risk of taking random actions during exploration is to craft the action space so that all actions are more or less safe to the system. To craft used herein means to define a set of allowed actions for an individual or a group of states. At least, no catastrophic consequences should occur by taking any action. In one prior-art method, a heuristic model is deployed in parallel to a RL policy. When the performance of the RL policy degrades below a threshold, the heuristic model is activated to replace the RL policy.

Learning an RL strategy, also referred to as a policy or a model, that performs well requires proper exploration to produce rich training data samples. During explorations, an RL agent may follow a randomization exploration strategy to explore combination of state and actions that would otherwise be unknown. While this allows to possibly learn better state-action combinations from which the agent policy can be improved upon, taking an action at random in a given state of the system may also lead to suboptimal behavior and therefore a performance degradation of the user experience and/or system availability, accessibility, reliability and retainability.

SUMMARY

As a part of developing embodiments herein a problem was identified by the inventors and will first be discussed.

As such, while it is necessary to explore actions at random to learn unseen parts of the state-action space, the resulting RAN system performance, e.g. availability, accessibility, reliability and retainability, and user experience may be negatively affected by the exploration. It is therefore necessary to control and optimize the collection of data samples via proper exploration strategies, so as to minimize the system performance degradation due to exploration.

In addition to the exploration rate, efficient operation of DRL requires careful tuning of training parameters, including but not limited to, the discount factor, the number of parameter update iterations, the parameter update scheme, etc. A discount factor when used herein means the weight of future rewards respect to the immediate reward. It is computationally very expensive to obtain the optimal training parameters. The agent typically tries out different parameter configurations and selects those that best improve the learning performance. Hence, techniques that efficiently select the optimal training parameters lead to improvements in the overall system performance.

An object of embodiments herein is to provide an improved performance of a RAN using RL with low risk of instantaneous performance degradation due to the exploration.

According to an aspect, the object is achieved by a method performed by a central node for controlling an exploration strategy associated to RL in one or more RL modules in a distributed node in a RAN. The central node evaluates a cost of actions performed for explorations in the one or more RL modules, and a performance of the one or more RL modules. Based on the evaluation, the central node determines one or more exploration parameters associated to the exploration strategy. The central node controls the exploration strategy by configuring the one or more RL modules with the determined one or more exploration parameters to update its exploration strategy. This enforces the respective one or more RL modules to act according to the updated exploration strategy to produce data samples for the one or more RL modules in the distributed node.

According to another aspect, the object is achieved by a central node configured to control an exploration strategy associated to RL in one or more RL modules in a distributed node in a RAN. The central node is further configured to:Evaluate a cost of actions performed for explorations in the one or more RL modules, and a performance of the one or more RL modules,based on the evaluation, determine one or more exploration parameters associated to the exploration strategy, andcontrol the exploration strategy by configuring the one or more RL modules with the determined one or more exploration parameters to update its exploration strategy, to enforce the respective one or more RL modules to act according to the updated exploration strategy to produce data samples for the one or more RL modules in the distributed node.

Thanks to that the evaluated a cost of actions performed for explorations in the one or more RL modules, and a performance of the one or more RL modules e.g. identifies services of high importance or strict requirements according to the evaluation, the central node may determine the one or more exploration parameters associated to the exploration strategy to achieve a reduced exploration in the presence of the identified services of high importance or strict requirements according to the evaluation. This results in a reduced impact of performance degradation of the RAN is achieved by a reduced exploration in the presence of services of high importance or strict requirements according to the evaluation. This in turn provides an improved performance of the RAN and improved level of user satisfaction using RL.

DETAILED DESCRIPTION

An example of embodiments herein relates to methods for controlling exploration and training strategies associated to RL in a wireless communications network.

Embodiments herein are e.g. related to Radio network optimization, Network Management, Reinforcement Learning, and/or Machine Learning.

In some examples of embodiments herein it is provided a signaling method between a central node and a distributed node to exchange control messages to properly configure exploration and training parameters of an RL algorithm in the distributed node.

FIG.3ais a schematic overview depicting a wireless communications network100.FIG.3billustrates a network architecture with one distributed node110and one central node130in the wireless communications network100. wherein embodiments herein may be implemented. The wireless communications network100comprises one or more RANs, such as the RAN102and one or more CNs. The wireless communications network100may use 5 Fifth Generation New Radio, (5G NR) but may further use a number of other different Radio Access Technologies (RAT)s, such as, (LTE), LTE-Advanced, Wideband Code Division Multiple Access (WCDMA), Global System for Mobile communications/enhanced Data rate for GSM Evolution (GSM/EDGE), Worldwide Interoperability for Microwave Access (WiMax), or Ultra Mobile Broadband (UM B), just to mention a few possible implementations.

Network nodes, such as a distributed node110, operate in the RAN102. The distributed node110may provide radio access in one or more cells in the RAN102. This may mean that the distributed node110provides radio coverage over a geographical area by means of its antenna beams. The distributed node110may be a transmission and reception point e.g. a radio access network node such as a base station, e.g. a radio base station such as a NodeB, an evolved Node B (eNB, eNode B), an NR Node B (gNB), a base transceiver station, a radio remote unit, an Access Point Base Station, a base station router, a transmission arrangement of a radio base station, a stand-alone access point, a Wireless Local Area Network (WLAN) access point, an Access Point Station (AP STA), an access controller, a UE acting as an access point or a peer in a Device to Device (D2D) communication, or any other network unit capable of communicating with a radio device within the cell served by network node110depending e.g. on the radio access technology and terminology used.

The distributed node110comprises one or more one or more RL modules111. The distributed node110is adapted to execute RL in the one or more RL modules111.

UEs such as the UE120operate in the wireless communications network100. The UE120may e.g. be an NR device, a mobile station, a wireless terminal, an NB-IoT device, an eMTC device, a CAT-M device, a WiFi device, an LTE device and an a non-access point (non-AP) STA, a STA, that communicates via such as e.g. the distributed node110, one or more RANs such as the RAN102to one or more CNs. It should be understood by the skilled in the art that the UE120relates to a non-limiting term which means any UE, terminal, wireless communication terminal, user equipment, (D2D) terminal, or node e.g. smart phone, laptop, mobile phone, sensor, relay, mobile tablets or even a small base station communicating within a cell.

Core network nodes, such as e.g. a central node130, operate in the CN. The central node130is adapted to control exploration strategies associated to RL in the one or more RL modules111in the distributed node110, e.g. by means of an exploration controller132in the central node130.

Methods herein may e.g. be performed by the central node110. As an alternative, a Distributed Node (DN) and functionality, e.g. comprised in a cloud140as shown inFIG.3a, may be used for performing or partly performing the methods.

FIG.3bFigure illustrates a hybrid RL architecture in the RAN102network architecture with one distributed node110and one central node130, wherein embodiments herein may be implemented.

In some example embodiments, the distributed node110is an eNB and/or gNB and the central node130may be an Operation and Maintenance (OAM) node. One or more RL modules111are located in the distributed node110. The respective one or more RL module111is a module that trains a policy and uses the policy to infer an action, e.g. changing the values of one or multiple configuration parameters in the distributed node110. An exploration controller132may be located in the central node130. The exploration controller132is a unit that may decide the value of one or multiple exploration parameters for the RL modules111.

The central node130has access to knowledge related to the cost of random actions taken by the RL modules111for exploration and the performance of the RL modules111in distributed nodes such as the distributed node130.

Based on this knowledge, the central node130maydetermine one or more parameters associated to a training strategy for one or more RL modules111of the distributed node110, andconfigure the one or more RL modules111by transmitting to the distributed node110, a control message comprising the determined one or more parameters associated to an exploration strategy for the one or more RL modules111of the distributed node110.

Based on this knowledge, the central node130may furtherdetermines one or more parameters associated to an exploration strategy for the one or more RL modules111of a distributed node110, andconfigure the one or more RL modules111by transmitting to the distributed node110, a control message comprising the determined one of more training parameters for the one or more RL modules111of the distributed node110.

Exploration

The wordings exploration and exploration strategy when used herein e.g. means the behaviour of the one or more RL modules111to probe state transition and resulted reward in an environment by randomly selecting an action.

The one or more exploration parameters to be determined herein will e.g. be used for the one or more RL modules111to decide the frequency of selecting a random action and/or the candidate actions that can be randomly selected in a given state.

Training

Compared to exploration and exploration strategy, the wordings training and training strategy when used herein e.g. means the process to update a policy based on the observed state transition and resulted reward after taking an action.

The one or more training parameters to be determined may e.g. be used for the RL module to control the training process by specifying the configuration of methods for ML model update.

The types of and the formats of the parameters associated to an exploration strategy that may be signaled with the control message explained more in detailed below.

Upon the reception of the message, the distributed node110applies the exploration and the training parameters configured by the central node130to the corresponding exploration strategy and training strategy for one or more RL modules111.

Embodiments herein may provide following advantages:

Example embodiments of the provided method controls the exploration strategy and possibly the training strategy in the distributed node110, e.g. by the exploration controller132located in the central node130where a richer knowledge is available e.g. compared to the distributed node110. The richer knowledge may comprise, in the serving area of the distributed node110, whether there are prioritized users, whether the served traffic is critical, whether there is an important event, etc.

This results in:A reduced impact of performance degradation of the RAN and user experiences by a reduced exploration in the presence of services of high importance or strict requirements. This is since unpredictable outcomes of random actions are avoided.An improved RL policy performance by an increased exploration when the performance of a RL policy in the distributed node degrades below a certain level.An improved learning performance of RL by configuring efficient training parameters for the one or more RL modules111in the distributed node130.

FIG.4shows example embodiments of a method performed by the central node130for controlling an exploration strategy associated to RL in the one or more RL modules111in the distributed node110in the RAN102.

The method comprises one or more of the following actions, which actions may be taken in any suitable order. Actions that are optional are marked with dashed boxes in the figure.

The central node130evaluates a cost of actions performed for explorations in the one or more RL modules111and a performance of the one or more RL modules111.

The cost of actions performed for explorations e.g. means degraded user experience with lower throughput and/or higher latency and degraded system performance with worse availability, accessibility, reliability and/or retainability. The cost of actions performed for explorations may e.g. be evaluated by predicting the outcome of the actions based on knowledge obtained from domain experts and/or past experiences.

The performance of the one or more RL modules111means the capability to achieve high rewards which is related to user experiences and system performance. The performance of the one or more RL modules111may e.g. be evaluated by the value of reward signals and/or Key Performance Indicators (KPIs) indicating user experience and system performance.

Based on the evaluation, the central node130determines one or more exploration parameters associated to the exploration strategy.

These one or more exploration parameters may later be used by the distributed node110for an exploration procedure according to the exploration strategy. i.e. the procedure to learn new knowledge, e.g. by taking random actions according to the determined one or more exploration parameters and observing the consequences.

In some embodiments the one or more exploration parameters is determined for a specific cell or group of cells controlled by the distributed node110.

The one or more exploration parameters may be determined further based on any one or more out of: Which may mean that the cost of actions performed for explorations in the one or more RL modules111and the performance of the one or more RL modules111may comprise any one or more out of:a performance of the RAN102,service requirements associated to services and applications provided by the distributed node110, andimportance of services provided by the distributed node110.

The one or more exploration parameters may comprise any one or more out of:an index indicating a type of the exploration strategy, anda value of the respective one or more exploration parameters.

The central node130controls the exploration strategy by configuring the one or more RL modules111with the determined one or more exploration parameters to update its exploration strategy. To update its exploration strategy e.g. means to change the frequency of selecting a random action and/or changing the candidate actions that may be randomly selected in a given state.

This enforces the respective one or more RL modules111to act according to the updated exploration strategy to produce data samples for the one or more RL modules111in the distributed node110. To act according to the updated exploration strategy to produce data samples means to select an action according to the updated exploration strategy and observe system transition and resulted reward.

It is an advantage that the central node130controls the exploration strategy since the central node130may possess more knowledge than the distributed node110to evaluate the cost of the exploration in the distributed node110.

In some embodiments the central node130configures the one or more RL modules111with the determined one or more exploration parameters by sending the one or more exploration parameters in a first control message.

In some embodiments the method is further performed for controlling a training strategy associated to the RL in the one or more RL modules111in the distributed node110. In these embodiments, the below actions404-405are performed.

In these embodiments the central node130determines one or more training parameters based on the evaluation. The one or more training parameters are associated to the training strategy.

The one or more training parameters may be determined further based on any one or more out of: Which may mean that the cost of actions performed for explorations in the one or more RL modules111and the performance of the one or more RL modules111may in these embodiments comprise any one or more out of:Importance of services provided by the distributed node110,requirements of services provided by the distributed node110,a search policy at the central node130, andobserved performance of the distributed node110for a variety of KPIs.

The one or more training parameters may comprise any one or more out of:A discount factor for calculating the value of an action,a type of gradient and the corresponding one or more training parameters, andan index indicating a type of learning scheme.

In these embodiments the central node130further configures the one or more RL modules111with the determined one or more training parameters to update its training strategy. It is an advantage that the central node130controls the training strategy since the central node130may possess more knowledge than the distributed node110about the best strategy for training.

This enforces the respective one or more RL modules111in the distributed node110to act according to the updated training strategy to use the produced data samples to update an RL policy of the RL module. To act according to the updated training strategy to use the produced data samples to update an RL policy of the RL module means to apply the method and hyperparameters specified in the updated training strategy to update the RL policy of the RL module.

In some embodiments the central node130configures the one or more RL modules111with the one or more training parameters, by sending the one or more training parameters in a second control message.

The embodiments described above will now be further explained and exemplified. The example embodiments described below may be combined with any suitable embodiment above.

Method in the Central Node130and its Embodiments.

Example embodiments herein discloses methods performed in the central node130for optimizing and controlling the configuration of the exploration strategy and possibly also the training strategy associated to RL, also referred to as machine learning, algorithms executed by the distributed node130. In one embodiment, the distributed node110is an eNB or gNB, and the central node130is an OAM node.

Exploration

As mentioned above the method may e.g. comprise the following related to the Actions described above:Determining402one or more parameters associated to the exploration strategy for one or more RL modules111of the distributed node110;Transmitting403a control message to the distributed node110comprising the one or more parameters associated to an exploration strategy for one or more RL modules111of the distributed node110.

In some embodiments, the central node130determines the one or more parameters associated to the exploration strategy for the one or more RL modules111of the distributed node110for a specific cell or group of cells controlled by the distributed node110.

In some other embodiments of the method, the central node130determines the one or more parameters associated to the exploration strategy for the one or more RL modules111of a distributed node110based on network performance and/or service requirements associated to services and applications provided by the distributed node110. Such examples may comprise:The importance and criticality of services provided by the distributed node110. The wordings importance and criticality when used herein means the level of impact to user satisfaction and/or the level of impact to the system availability, accessibility, reliability and retainability KPI. The distributed node110may provide services of different importance and criticality such as e.g. critical IoT services, services for a critical event, etc.Existence of VIP users in the coverage area of one or more radio cells controlled by the distributed node110; VIP users means users of high business values, e.g. golden subscription users.Requirements of services provided in of one or more radio cells controlled by the distributed node110, for instance in terms of required data rate, latency, reliability, energy efficiency, etc. Such requirements may be expressed in terms of minimum requirement, maximum requirement, average required, statistical deviation from a reference requirement or a combination thereof.In one example, such requirements are defined as requirements associated to one or more network slices supported in the coverage area of one or more radio cell of the distributed node110.In another example, the requirements are derived based on the type of services provided by the distributed node110. The service type, e.g. web browsing, file sharing or YouTube video, may be identified by deep package inspection.

For instance, in case the central node130detects critical or prioritized services, or VIP users, or services with stringent requirements in terms of data rate, latency, reliability, energy efficiency, etc. to be provided within the coverage area of one or more radio cells controlled by a distributed node110where exploration is configured, the central node130may determine to reduce the amount of exploration by changing the one or more parameters of the exploration strategy.

For example, with an E-greedy exploration strategy, wherein a control policy is tasked to explore with probability ∈∈[0, 1], i.e., acting according to a random probability distribution, such as taking an action with uniform probability among all available actions, and to act according to the control policy with probability 1−∈, the central node130may determine to reduce the current value E configured for the distributed node110so as to reduce the average number of actions taken according to a random probability distribution. Vice versa, when the central node130detects that there are no critical traffic or services to be supported in any of the cells controlled by the distributed node110, the central node130may determine to increase the explorative behavior of the distributed node110.

In some embodiments of the method, the central node130determines the one or more parameters associated to the exploration strategy for the one or more RL modules111of the distributed node110based on network performance experienced in the coverage area of the radio cells controlled by the distributed node110. For instance, if the network performance measured in the radio cells controlled by the distributed node110falls below a threshold or is lower compared to the performance of other radio cells controlled by other distributed nodes, for instance with similar deployment and radio conditions, the central node130may infer that the RL policy used by the distributed node110in one or more controlled radio cells is not sufficiently good, and may thereby determine to increase the explorative behavior of the distributed node110in one or more of its controlled cells in order to collect new data that could improve the current policy.

In some embodiments the central node130may determine to change exploration strategy for the distributed node110. Examples of possible exploration strategies include, but are not limited to:Random exploration according to a given probability distribution over the action space, such as uniform distribution, e.g., such as ∈-greedy exploration strategy, or a non-uniform distribution, such as a semi-uniform distributed exploration, etc.Boltzmann-Distributed Exploration, which considers the estimated utility of all actions a∈according to the probability distribution

Pa=ef⁡(a)⁢θ-1/∑i∖in⁢𝒜ef⁡(i)⁢θ-1wherein Pais the probability of taking an action a, i\inmeans action i in an action set A, a is the action whose taken probability is under calculation and i is an action in the action set A andwhere the amount of randomness is controlled by the parameter θ∈[0, ∞), with θ→0 indicating pure random behavior, andCounter-Based Exploration which uses the difference between the counter value for the current state c(s) and the expected counter value for the state that results from taking an action E[c|s, a]Counter/Error-Based Exploration

Therefore, the central node130may signal to the distributed node110which exploration strategy to use and the corresponding the one or more parameters. For instance, the central node130may signal an exploration strategy as one element of an enumerated list or using a bitmap with each bit indicating one specific exploration strategy and setting the bit to equal to 1 only for the selected exploration strategy.

In case the distributed node110is using an exploration strategy where the one or more parameters are changed dynamically and locally by the distributed node110, the central node130may further:Transmit a signal to the distributed node110requesting the current one or more parameters used for the exploration procedure.Receive a response from the distributed node110comprising the one or more parameters currently used for exploration.Then determining one or more updated parameter associated to the exploration strategy for the one or more RL modules111of the distributed node110based on the response message.

For instance, if the distribute node110is configured to explore according to an E-greedy exploration strategy with decaying and/or annihilating exploration over time, the value of the exploration parameter E initially configured by the central node130for the distributed node110may be reduced by the distributed node110over time so as to reduce the amount of exploration. If the central node130has not configured the distributed node110with a specific decaying and/or annihilating exploration parameters, the central node130may not be aware of the current value of the parameter E governing the amount of exploration at the distributed node110. The knowledge of such parameter would be necessary to the central node130to determine whether the exploration strategy used by the distributed node110, or its associated one or more parameters, need to be updated, e.g., due to critical or prioritized services or users according to other embodiments.

Training

As mentioned above the method may in some embodiments further comprise the following, related to the Actions described above:Determining404one or more parameters associated to the training strategy for the one or more RL modules111of a distributed node110;Transmitting405a control message to the distributed node110comprising one of more training parameters for one or more RL modules111of the distributed node110.

In some embodiments, the central node130determines one or more training parameters such as one or more efficient training parameters. For example, the central node130may signal different learning parameters to each of different distributed nodes such as e.g. the distributed node110. For a distributed node, e.g. the distributed node110, that handles critical or prioritized traffic, the central node130may configure training parameters that have provided a high training performance, also referred to as learning performance, in previous instances. Learning performance when used herein may mean the achieved accuracy of the model prediction after trained with a given number of samples. For other distributed nodes, which in some embodiments also may be the distributed node110, the central node130may configure training parameters for which the impact on learning performance is insufficiently known. In this manner, the central node130may efficiently obtain knowledge about the best training parameter configurations comprising the one or more training parameters, while minimizing the adverse impact on the overall system performance. Periodically, the central node130may update the training parameters for all or a subset of the distributed nodes, e.g. comprising the distributed node110, in response to the type of traffic being currently served by that distributed node, and the knowledge about the training parameters collected so far. The central node130may choose training parameters based on, for example,Random selection from a grid of feasible training parameters such as the one or more training parameters.Linear interpolation between the one or more training parameters that provide the best performance across multiple distributed node110s.Linear interpolation between the one or more training parameters where the weighting is done based on the number of training samples, the type of network traffic served by the distributed node110, or any combination of related metrics.Bayesian optimization, where the observed performance for the distributed node110is probabilistically modeled, and this model is sampled to get the next set of training parameters.Population-based training, where the observed performance across the distributed nodes, e.g. comprising the distributed node110, is used to estimate a next set of one or more training parameters to be applied.

At the distributed node110the following actions may be performed.Receiving, from the central node130, a control message comprising one or more exploration parameters associated to an exploration strategy for the one or more RL modules111of the distributed node110.Appling the exploration parameters configured by the central node130to the corresponding exploration strategy for the one or more RL modules111.Responding, to a request from the central node130, with a message comprising the current parameters used for exploration,Receiving, form the central node130, a control message comprising one or more parameters associated to a training strategy for the one or more RL modules111of the distributed node110.Applying the learning parameters configured by the central node130to the corresponding training strategy for the one or more RL modules111.Transmitting to the central node130, a message comprising the current training parameters and the KPIs related to the performance of the learning scheme, from the distributed node110.

Example of embodiments herein provide:A signaling method between a central node such as the central node130and a distributed node such as the distributed node110to communicate one or more exploration parameters associated to an exploration strategy for the one or more RL modules111sof a distributed node110.The one or more exploration parameters are determined by the central node130e.g. based on:The importance of the services provided by the distributed node110The requirements of the services provided by the distributed node110The performance of the RL policies located in the distributed node110.The one or more exploration parameters associated to the exploration strategy may include:An index indicating a type of the exploration strategyA value of a parameter associated to the exploration strategy, e.g. E in ∈-greedy exploration and θ in Boltzmann-distributed explorationA signaling method between a central node such as the central node130and a distributed node such as the distributed node110to communicate one or more training parameters associated with a training strategy for the one or more RL modules111of the distributed node110.The parameters are determined by the central node130e.g. based on:The importance of the services provided by the distributed node110.The requirements of the services provided by the distributed node110.The search policy at the central node130, for example, grid search, interpolation, Bayesian approaches, or population-based training.The observed performance of the distributed node110for a variety of KPIs.The parameters associated with the training strategy e.g. include:A discount factor for calculating the value of an action.The type of gradient, such as e.g. full batch, mini batch, . . . , and the associated one or more training parameters such as e.g. number of epochs, number of samples per epoch, . . . .An index indicating the type of learning scheme, e.g., stochastic gradient descent, Adam, etc.

To perform the action as mentioned above, the central node130may comprise the arrangement as shown inFIGS.5aandb. The central node130is configured to control an exploration strategy associated to RL in the one or more RL modules111in the distributed node110in the RAN102. The central node130may in some embodiments be configured to control a training strategy associated to the RL in the one or more RL modules111in the distributed node110.

The central node130may comprise a respective input and output interface500configured to communicate with e.g. the distributed node110, seeFIG.5a. The input and output interface500may comprise a wireless receiver (not shown) and a wireless transmitter (not shown).

The central node130may further be configured to, e.g. by means of an evaluating unit510in the central node130, evaluate a cost of actions performed for explorations in the one or more RL modules111, and a performance of the one or more RL modules111.

The central node130may further be configured to, e.g. by means of a determining unit511in the central node130, based on the evaluation, determine one or more exploration parameters associated to the exploration strategy.

The one or more exploration parameters may be adapted to be determined, e.g. by means of the determining unit511, for a specific cell or group of cells controlled by the distributed node110.

The central node130may further be configured to, e.g. by means of the determining unit511, determine the one or more exploration parameters based on any one or more out of:a performance of the RAN102andservice requirements associated to services and applications arranged to be provided by the distributed node110,importance of services arranged to be provided by the distributed node110.

The one or more exploration parameters may be adapted to comprise any one or more out of:an index adapted to indicate a type of the exploration strategy, anda value of the respective one or more exploration parameters.

The central node130may further be configured to, e.g. by means of the determining unit511, determine one or more training parameters, which one or more training parameters are adapted to be associated to the training strategy.

The central node130may further be configured to, e.g. by means of the determining unit511, determine the one or more training parameters based on any one or more out of:importance of services arranged to be provided by the distributed node110,requirements of services arranged to be provided by the distributed node110,a search policy at the central node130,observed performance of the distributed node110arranged for a variety of KPIs.

The one or more training parameters may be adapted to comprise any one or more out of:a discount factor arranged for calculating the value of an action,a type of gradient and the corresponding one or more training parameters, andan index adapted to indicate a type of learning scheme.

The central node130may further be configured to, e.g. by means of an configuring unit512in the central node130, control the exploration strategy by configuring the one or more RL modules111with the determined one or more exploration parameters to update its exploration strategy, to enforce the respective one or more RL modules111to act according to the updated exploration strategy to produce data samples for the one or more RL modules111in the distributed node110.

The central node130may further be configured to, e.g. by means of the configuring unit512, configure the one or more RL modules111with the determined one or more training parameters to update its training strategy, to enforce the respective one or more RL modules111in the distributed node110to act according to the updated training strategy to use the produced data samples to update an RL policy of the RL module.

The central node130may further be configured to, e.g. by means of the configuring unit512, any one or more out of:configure one or more RL modules111with the determined one or more exploration parameters arranged to be performed by sending the one or more exploration parameters in a first control message, andconfigure one or more RL modules111with the one or more training parameters, arranged to be performed by sending the one or more training parameters in a second control message.

The embodiments herein may be implemented through a processor or one or more processors, such as a processor550of a processing circuitry in the central node130inFIG.5a, together with computer program code for performing the functions and actions of the embodiments herein. The program code mentioned above may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the central node130. One such carrier may be in the form of a CD ROM disc. It is however feasible with other data carriers such as a memory stick. The computer program code may furthermore be provided as pure program code on a server and downloaded to the central node130.

The central node130may further comprise a memory560comprising one or more memory units. The memory560comprises instructions executable by the processor550in the central node130. The memory560is arranged to be used to store, e.g. training parameters, exploration parameters, training strategy, control messages, data samples, RL policies, information, data, configurations, and applications, to perform the methods herein when being executed in the central node130.

In some embodiments, a computer program570comprises instructions, which when executed by the at least one processor550, cause the at least one processor550of the central node130to perform the actions above.

In some embodiments, a carrier580comprises the computer program570, wherein the carrier580is one of an electronic signal, an optical signal, an electromagnetic signal, a magnetic signal, an electric signal, a radio signal, a microwave signal, or a computer-readable storage medium.

Further Extensions and Variations

The communication system3300further includes the UE3330already referred to. Its hardware3335may include a radio interface3337configured to set up and maintain a wireless connection3370with a base station serving a coverage area in which the UE3330is currently located. The hardware3335of the UE3330further includes processing circuitry3338, which may comprise one or more programmable processors, application-specific integrated circuits, field programmable gate arrays or combinations of these (not shown) adapted to execute instructions. The UE3330further comprises software3331, which is stored in or accessible by the UE3330and executable by the processing circuitry3338. The software3331includes a client application3332. The client application3332may be operable to provide a service to a human or non-human user via the UE3330, with the support of the host computer3310. In the host computer3310, an executing host application3312may communicate with the executing client application3332via the OTT connection3350terminating at the UE3330and the host computer3310. In providing the service to the user, the client application3332may receive request data from the host application3312and provide user data in response to the request data. The OTT connection3350may transfer both the request data and the user data. The client application3332may interact with the user to generate the user data that it provides.

It is noted that the host computer3310, base station3320and UE3330illustrated inFIG.7may be identical to the host computer3230, one of the base stations3212a,3212b,3212cand one of the UEs3291,3292ofFIG.6, respectively. This is to say, the inner workings of these entities may be as shown inFIG.7and independently, the surrounding network topology may be that ofFIG.6.

The wireless connection3370between the UE3330and the base station3320is in accordance with the teachings of the embodiments described throughout this disclosure. One or more of the various embodiments improve the performance of OTT services provided to the UE3330using the OTT connection3350, in which the wireless connection3370forms the last segment. More precisely, the teachings of these embodiments may improve the applicable RAN effect: data rate, latency, power consumption, and thereby provide benefits such as corresponding effect on the OTT service: e.g. reduced user waiting time, relaxed restriction on file size, better responsiveness, extended battery lifetime.

FIG.8is a flowchart illustrating a method implemented in a communication system, in accordance with one embodiment. The communication system includes a host computer, a base station such as the central node130, and a UE such as the UE120, which may be those described with reference toFIG.6andFIG.7. For simplicity of the present disclosure, only drawing references toFIG.8will be included in this section. In a first action3410of the method, the host computer provides user data. In an optional subaction3411of the first action3410, the host computer provides the user data by executing a host application. In a second action3420, the host computer initiates a transmission carrying the user data to the UE. In an optional third action3430, the base station transmits to the UE the user data which was carried in the transmission that the host computer initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In an optional fourth action3440, the UE executes a client application associated with the host application executed by the host computer.

FIG.10is a flowchart illustrating a method implemented in a communication system, in accordance with one embodiment. The communication system includes a host computer, a base station such as an AP STA, and a UE such as a Non-AP STA which may be those described with reference toFIG.6andFIG.7. For simplicity of the present disclosure, only drawing references toFIG.10will be included in this section. In an optional first action3610of the method, the UE receives input data provided by the host computer. Additionally, or alternatively, in an optional second action3620, the UE provides user data. In an optional subaction3621of the second action3620, the UE provides the user data by executing a client application. In a further optional subaction3611of the first action3610, the UE executes a client application which provides the user data in reaction to the received input data provided by the host computer. In providing the user data, the executed client application may further consider user input received from the user. Regardless of the specific manner in which the user data was provided, the UE initiates, in an optional third subaction3630, transmission of the user data to the host computer. In a fourth action3640of the method, the host computer receives the user data transmitted from the UE, in accordance with the teachings of the embodiments described throughout this disclosure.