Patent ID: 12256263

DETAILED DESCRIPTION OF EMBODIMENTS

In the following, concepts in accordance with exemplary embodiments of the invention will be explained in more detail and with reference to the accompanying drawings. The illustrated embodiments relate to controlling data user data traffic in a wireless communication network.

The wireless communication network may be based on various radio technologies, e.g., the NR radio technology, the LTE radio technology, the UMTS radio technology, or combinations of these technologies.

The illustrated concepts aim at efficiently controlling the user data traffic with respect to QoE of a user. As further explained below, this may be achieved by applying a machine learning processes in a network node. For this purpose, the network node is provided with information to enable the machine learning processes, in particular data indicating a desired or wanted QoE (in the following also denoted as wQoE) and optionally also data indicating an actual QoE (in the following also denoted as aQoE) as measured at endpoints transmitting and receiving the user data traffic, e.g., a UE (user equipment) and a service provider node. The machine learning processes may be based on an RL (reinforcement learning) algorithm. However, other machine learning algorithms, such as supervised learning or unsupervised learning, could be used as well.

FIG.1illustrates exemplary structures of the wireless communication network. In particular,FIG.1shows multiple UEs10in a cell110of the wireless communication network. The cell110is assumed to be served by an access node100, e.g., a gNB of the NR technology, an eNB of the LTE technology, or an NB of the UMTS technology. Further,FIG.1illustrates a core network (CN)120of the wireless communication network. The CN120is illustrated as including a GW (gateway)150and a controller160. The GW150is responsible for handling user data traffic of the UEs10, e.g., by forwarding user data traffic from a UE10to a network destination or by forwarding user data traffic from a network source to a UE10. Here, the network destination may correspond to another UE10, to an internal node of the wireless communication network, or to an external node which is connected to the wireless communication network. Similarly, the network source may correspond to another UE10, to an internal node of the wireless communication network, or to an external node which is connected to the wireless communication network. The controller160may in turn be responsible for controlling the user data traffic with respect to QoE of users associated with the UEs10.

As illustrated by double-headed arrows, the access node100may send DL (downlink) transmissions to the UEs, and the UEs may send UL (uplink) transmissions to the access node100. The DL transmissions and UL transmissions may be used to provide various kinds of services to the UEs, e.g., a voice service, a multimedia service, or a data service. Such services may be hosted in the CN120, e.g., by a corresponding network node. Further, such services may be hosted externally, e.g., by an AF (application function) connected to the CN120. By way of example,FIG.1illustrates a service platform180provided outside the wireless communication network. The service platform180could for example connect through the Internet or some other wide area communication network to the CN120. The service platform180may be based on a server or a cloud computing system. The service platform180may include or be associated with one or more AFs that enable interaction of the service platform180with the CN120. The service platform180may provide one or more services to the UEs10, corresponding to one or more applications. These services or applications may generate the user data traffic conveyed by the DL transmissions and/or the UL transmissions between the access node100and the respective UE10. Accordingly, the service platform180may include or correspond to the above-mentioned network destination and/or network source for the user data traffic.

It is noted that the wireless communication network may actually include more access nodes for serving multiple cells in a similar way as explained for the access node100and the cell110.

As mentioned above, the wireless communication network may be based on the NR technology.FIG.2illustrates elements of a 5G (5thGeneration) CN architecture which a used in connection with the NR technology. Specifically,FIG.2illustrates a UDR (Unified Data Repository)210, an NEF (Network Exposure Function)220, an NWDAF (Network Data Analytics Function)230, an AF (Application Function)240, a PCF (Policy Control Function)250, a CHF (Charging Function)260, an SMF (Session Management Function)270, a UPF (User Plane Function)280, and an AMF (Access Management Function)290. Further,FIG.2also illustrates interfaces (also referred to as reference points) between these node. Specifically, these interfaces include an Nudr reference point with respect to the UDR210, an Nnef reference point with respect to the NEF220, an Nnwdaf reference point with respect to the NWDAF230, an Naf reference point with respect to the AF240, an Npcf reference point with respect to the PCF250, an Nchf reference point with respect to the CHF260, an Nsmf reference point with respect to the SMF270, an N4reference point between the SMF270and the UPF280, and an Namf reference point with respect to the AMF290.

In the context of the illustrated concepts functionalities of the AF240may include interaction with the CN in order to provide one or more services. This may specifically include controlling of traffic handling with respect to QoE, by providing the CN with information on the desired QoE and optionally the actual QoE experienced by the user.

In the context of the illustrated concepts functionalities of the NEF220may include exposure of capabilities and events. Specifically, capabilities of network nodes and events may be securely exposed to 3rd party nodes, such as a 3rdparty AF240. As further explained below, the functionalities of the NEF220may for example be used when establishing a user data session for a certain AF, which requires a certain QoE. Further, the NEF220may support secure provision of information from external nodes or applications to the wireless communication network and translate between network-external and network-internal information.

In the context of the illustrated concepts functionalities of the PCF250may include providing of policy rules to control plane node(s) to enforce them. Specifically, the PCF250may support retrieving information on QoS requested for user data traffic from the NEF220and installing corresponding PCC rule/s with the corresponding QoS enforcement actions towards the SMF270.

In the context of the illustrated concepts functionalities of the UPF280may include: acting as a point of interconnect to an external data network, e.g., the Internet, packet routing and forwarding, packet inspection, (e.g. application detection based on service data flow template and optionally one or more PFDs (Packet Flow Descriptions) or one or more PDRs (Packet Detection Rules) provided by the SMF270, user plane policy rule enforcement, e.g., by gating, redirection, traffic steering, and user plane QoS handling, e.g., by rate enforcement or QoS marking.

In the context of the illustrated concepts functionalities of the SMF270may include obtaining application-specific PCC rules from the PCF250. The SMF270may also be responsible for providing and activating one or more PDRs (Packet Detection Rules) in the UPF280and/or for providing and activating one or more QERs (QoS Enforcement Rules) in the UPF280. The PDR(s) may be used to identify user data traffic of a certain application and the QER(s) may then be used to indicate the requested QoS handling to the UPF280.

Further details concerning functionalities of the illustrated nodes and reference points can for example be found in 3GPP TS 23.501 V16.1.0.

It is noted that whileFIG.2illustrates typical elements of a 5G CN, not all the illustrated elements are actually required for implementing the illustrated concepts. Further, it is noted that in other implementations, e.g., using a 4G (4thGeneration) or a 3G (3rdGeneration) CN architecture, the elements ofFIG.2could be replaced having other designations, but similar functionalities. For example, in the illustrated concepts the GW150ofFIG.1could be implemented by the UPF280, and the controller160could be implemented by the SMF270and/or the PCF250. In the case of a 4G CN architecture, the GW150could be implemented by a PGW (Packet Data Gateway) and the controller160could be implemented by a PCRF (Policy and Charging Rules Function). In the case of a 3G CN architecture, the GW150could be implemented by a GGSN (General Packet Data Service Gateway Support Node) and the controller160could be implemented by a PCRF.

FIG.3further illustrates implementation of the illustrated concepts in an architecture as illustrated inFIG.2and using RL processes in the UPF280. As illustrated inFIG.3, the UPF280forwards user data traffic T. In the illustrated example, the UPF280is provided with a UPF logic281, a QoS enforcement module282, an RL agent283, and a QoE estimation module284. The QoE estimation module estimates the QoE for the user by monitoring the user data traffic T. This may be accomplished in a user-specific and application-specific manner. The QoE may for example be estimated in terms of a MOS (Mean Opinion Square) level. This may for example be achieved by detecting characteristics of the monitored user data traffic and determining the estimated QoE from a mapping of the characteristics to MOS levels. This mapping may be based on known correlations of characteristics to MOS levels as actually measured, e.g., using user ratings or the like. The estimated QoE constitutes an environment285for the RL processes.

In the illustrated example, the RL processes involve that the RL agent283learns how to behave with respect to the environment285by selecting actions and observing results of the actions. Each of the actions correspond to one or more QoS rules enforced by the QoS enforcement module282. In the following, the actions will therefore also be referred to as QoS enforcement actions. The results of the QoS enforcement actions correspond to changes of the estimated QoE. In response to selecting a QoS enforcement action, the RL agent283receives information on a reward. The reward may be regarded as a measure of how desirable the state of the environment285is. The RL agent283may then execute an algorithm in order to identify and learn QoS enforcement actions that maximize a cumulative reward function in the long term. It is noted that the RL agent283may operate on the basis of various kinds of RL algorithms, including brute-force type algorithms, value-function type algorithms, Monte-Carlo type algorithms, temporal-difference type algorithms, and direct-policy search type algorithms.

The environment285may be modeled as a Markov decision process (MDP). The MDP may use a finite set of states, also referred to as observation space, and a finite set of QoS enforcement actions, also referred to as action space, that lead to changes between the states. Interaction of the RL agent283with the environment285is assumed to occur in discrete time steps. At a given time step, denoted by t, the environment285is in some state, denoted by St, and the QoE estimation module284provides an observation of this state St, e.g., an estimated MOS level, and the corresponding reward, denoted by Rt, to the RL agent283. The RL agent285may then choose a QoS enforcement action, denoted by At, that is available in the state St. The environment285reacts to the QoS enforcement action Atby transitioning to a new state, denoted as St+1. The new state St+1may correspond to a QoE level which differs from the former state St. Information on the new state St+1and the corresponding reward, denoted as Rt+1, are indicated to the RL agent283, and the process may continue in an iterative manner.

By iterating the above process and observing the rewards, the RL agent283may learn an optimized policy that maps states to the QoS enforcement actions in such a way that the cumulative reward function gets maximized.

The RL agent285may operate in two different modes: an exploration mode and an exploitation mode. In the exploration mode the RL agent285may select QoS enforcement actions that do not follow the optimized policy, e.g. by selecting QoS enforcement actions randomly, using heuristic algorithms to select the QoS enforcement actions, or using more complex methods such as an epsilon-greedy algorithm. In the exploitation mode the RL agent285may select QoS enforcement actions according to the optimized policy learned in the exploration mode. The RL agent285may switch between the two modes with the aim balancing the benefit of utilizing an optimized policy against the chance of even further optimizing the policy.

The QoE estimation module284calculates the rewards as a function of the desired QoE (wQoE) and the estimated QoE. In some scenarios, the calculation of the rewards may also be based on the actually measured QoE (aQoE) and/or on various other parameters. As information on the desired QoE and optionally the actual QoE may be provided by the AF240and indicated via the NEF220, PCF250, and SMF270to the UPF280. This may be accomplished upon PDU session establishment between the UE10and the wireless communication network.

In the UPF280, information on the desired QoE is also provided to the RL agent283. The RL agent283may use the desired QoE as a basis for determining the QoS enforcement action space, i.e., a set of QoS rules which can be applied by the QoS enforcement module.

An exemplary procedure for implementing the illustrated concepts may include the following sub-procedures:1. The UPF280associates to the SMF270and indicates that it supports the RL-based QoE control, i.e., in the course of negotiating capabilities between the UPF280and the SMF270.2. A user, e.g., associated with one of the UEs10, establishes a PDU session and the SMF270selects the UPF280, which supports the capability of RL-based QoE control, for this PDU session.3. The user starts a service provided by the AF240. The AF240provides the wanted QoE for the service to the NEF220. The NEF220forwards the wanted QoE to the PCF250. The PCF250generates a PCC rule extended by the wanted QoE and installs the PCC rule in the SMF270. The SMF270indicates the wanted QoE to the UPF280.4. The QoE estimation module284processes the user data traffic associated with the service and estimates the QoE and calculates the corresponding reward, e.g., based on the estimated QoE and the wanted QoE. Further, the QoE estimation module284indicates the state and the corresponding reward to the RL agent283.5. Based on the indicated state and reward, the RL agent283learns the effect of the past QoS enforcement action selections by the RL agent283.6. The RL agent selects a QoS enforcement action from the QoS enforcement action space. This selection is based on the wanted QoE, the indicated state, and the indicated reward. Further, this selection may depend a learned or pre-configured policy or on whether the RL agent is in the exploration mode or the exploitation mode.

Sub-procedures 4 to 6 may be iterated for learning an optimized policy for selection of QoS enforcement actions.

It is noted that whileFIG.3assumes implementation of the illustrated concepts in a 5G context, corresponding functionalities could also be implemented in a 3G or 4G context, e.g., by providing a 3G GGSN or a 4G PGW with similar functionalities as described for the UPF280.

FIG.4illustrates an example of processes which may be utilized in the illustrated concepts and involve the UPF280and the SMF270. In the processes ofFIG.4, the UPF280and the SMF270negotiate capabilities. As illustrated, such processes may be used to inform the SMF270or other nodes that the UPF280supports the capability of RL-based QoE control. The processes ofFIG.4may be performed when the UPF280is deployed in the wireless communication network or in response to deployment changes of the wireless communication network.

In the processes ofFIG.4, the UPF280sends an Association Setup Request401to the SMF270. The Association Setup Request may be part of a PFCP (Packet Forwarding Control Protocol) Association Setup Procedure as specified in section 6.2.6 of 3GPP TS 29.244 V16.0.0 (2019-06). The Association Setup Request401indicates that the UPF280supports the capability of RL-based QoE control. Further, the Association Setup Request401may indicate other capabilities or features of the UPF280.

The SMF270responds to the UPF280by sending an Association Setup Response402. Similar to the Association Setup Request401, the Association Setup Response may be part of the PFCP Association Setup Procedure. The Association Setup Response402indicates that the SMF270supports the capability of RL-based QoE control. Further, the Association Setup Response402may indicate other capabilities or features of the SMF270.

It is noted that processes similar to those ofFIG.4can also be initiated by the SMF270. In such cases the SMF270could send an Association Setup Request indicating that the SMF270supports the capability of RL-based QoE control, and the UPF280respond by sending an Association Setup Response indicating that the UPF280supports the capability of RL-based QoE control. Further, it is noted that in a similar manner, the capability of RL-based QoE control could also be negotiated between other nodes, e.g., between the AF240and the NEF220, between the NEF220and the PCF250, and between the PCF250and the SMF270. Further, the indicated capability may also be further propagated by the nodes.

FIG.5illustrates a further example of processes which may be utilized in the illustrated concepts and involve the UE10, the UPF280, the SMF270, the PCF250, the NEF280, and the AF240. The processes ofFIG.5may be used for providing information related to the QoE control from the AF240to the UPF280. The processes ofFIG.5may be performed when the UPF280is deployed in the wireless communication network or in response to deployment changes of the wireless communication network.

As illustrated by501, the UE10first establishes a PDU session with the wireless communication network. The PDU session is used for conveying user data traffic from the UE10. In the example ofFIG.5, it is assumed that the user data traffic includes application traffic502transmitted between the UE10and the AF240. The application traffic502may for example be generated to or by a multimedia telephony application, to or by a voice telephony application, to a video streaming application, or to or by a gaming application. The AF240is associated with a provider of the corresponding application.

As illustrated by block503, the AF240then selects the wQoE for the user data traffic of the application and starts monitoring the aQoE.

The AF240then initiates a procedure for setting up an AF session with required QoS with the NEF280. As illustrated, this involves that the AF240sends an HTTP (Hypertext Transfer Protocol) POST message504via the Nnef reference point to the NEF280. The HTTP POST message504includes the wQoE and optionally the aQoE monitored by the AF240. Further, the HTTP POST message504may include an identifier of the application and/or a provider identifier of the application. Further, the HTTP POST message504may include an IP (Internet Protocol) address of the UE10and/or a flow description, e.g., in terms of an IP 5-tuple. If present, the aQoE may be indicated in terms of an MOS level.

The NEF220then acknowledges the requested AF session setup and responds with an HTTP200OK message505to the AF240. Further, the NEF220may map the identifier of the application indicated by the HTTP POST message504to a network-internal application identifier.

The NEF220then interacts with the PCF250by sending an HTTP POST message506via the Npcf reference point to the PCF250. The HTTP POST message506includes the wQoE and optionally the aQoE. Further, the HTTP POST message506may include the network-internal identifier of the application, the provider identifier, the IP address of the UE10, and/or the flow description. The PCF250acknowledges the requested AF session setup and responds with an HTTP200OK message507to the NEF220. Further, the PCF250identifies based on the IP address of the UE10that the SMF270is responsible for handling the PDU session of the UE10. As illustrated by block508, the PCF250also determines a PCC rule for controlling the user data traffic of the application. The PCC rule also considers the QoE control, by including the wQoE and optionally the aQoE.

For installation of the PCC rule, the PCF250sends an HTTP POST message509requesting installation of the PCC rule via the Nsmf reference point to the SMF270. The SMF270then acknowledges the requested installation of the PCC rule and responds with an HTTP200OK message510to the PCF250.

The SMF270then modifies the PFCP session with the UPF280by sending a Session Modification (SM) Request511via the N4reference point to the UPF280. The SM Request511may be part of a PFCP Session Modification Procedure as specified in section 6.3.3 of 3GPP TS 29.244 V16.0.0. The SM Request511includes one or more PDRs for detecting the the application traffic in the user data traffic, a Forwarding Action Rule (FAR), and a Quality Enforcement Rule (QER). The QER includes the wQoE and optionally also the aQoE. The UPF280then acknowledges the requested session modification and responds with an SM Response512to the SMF270.

As indicated by block513, the UPF280may then detect the application traffic based on the PDR(s). If there is a match, the application traffic is processed according to the QER, taking into account the wQoE and optionally the aQoE. This processing of the application traffic may involve processes as explained in connection withFIGS.6A and6B.

It is noted that if the AF240detects a relevant change of the aQoE, the AF240may trigger an update of the aQoE towards the UPF280, by means of sending an HTTP PUT message including the updated aQoE to the NEF220, and the NEF220may then forward this information via the PCF250, and the SMF270to the UPF280. Further, it is noted that while the processes ofFIG.5involve that the wQoE and the aQoE are transmitted in the same message, it would also be possible to utilize separate messages or separate procedures for transmitting the wQoE and the aQoE. In this way, in could be taken into account that the wQoE is typically more static than the aQoE, and that updates of the aQoE could be triggered more frequently, without requiring transmission of the wQoE or other static parameters.

As illustrated, the processes ofFIGS.6A and6Binvolve the UE10, the UPF280, and the AF240. Within the UPF280, the processes involve the UPF logic281, the QoS enforcement module282, the RL agent283, and the QoE estimation module284.

In the processes ofFIG.6A, the UPF280receives the application traffic601,602from the AF240and/or the UE10. In the UPF280, the UPF logic281applies the PDR(s) to detect the application traffic601,602in the user data traffic. The PDR(s) may indicate one or more IP 5-tuples for identifying matching data packets. In response to detecting a matching data packet, as illustrated by block603, the UPF logic proceeds with the further processes ofFIGS.6A and6B.

As illustrated by block604, these processes include that the UPF logic281generates an application session identifier (asID) and stores a mapping of the IP 5-tuple matching the data packet and the asID. In this way, the asID can be identified for subsequently detected data packets matching the IP 5-tuple.

The UPF logic281then sends an application session setup message605to the QoE estimation module284. The application session setup message605indicates the asID, the wQoE, and optionally the aQoE for the application traffic. In response to receiving this information, the QoE estimation module284configures computation algorithms for calculation of the state and the reward, as indicated by block606. This configuration is specific for the application session identified by the asID and is based on the wQoE and optionally the aQoE.

Further, the UPF logic281sends an application session setup message607to the RL agent283. The application session setup message607indicates the asID and the wQoE. In response to receiving this information, the RL agent283configures the QoS enforcement action space, as indicated by block608. This configuration is based on the wQoE indicated for this application session.

The QoS enforcement action space is the set of QoS enforcement actions that are available for selection in this application session and corresponds to the action space of the RL mechanism. The QoE enforcement action space may be defined in terms of a set of QoS parameters, e.g., relating to throttling, ABR (Adaptive Bit Rate) shaping, usage of a dedicated bearer with a certain QoS or QCI (QoS Control Index). The QoS enforcement action space may be defined by defining ranges of such QoS parameters, e.g., in terms of a maximum value, a minimum value, and/or parameter step size. For example, for throttling the QoS enforcement action space could define a maximum throttling value of 1 Mbps, a minimum throttling value of 64 kbps, and a step size of to 64 kbps. When configuring the QoS enforcement action space, the wQoE may for example be considered by defining a larger step size and wider range of QoS parameters for higher values of the wQoE. That is to say, if the wQoE is high, e.g., corresponds to MOS level 5, a high step size between QoS enforcement actions can be configured, and for lower wQoE, e.g., corresponding to MOS level 3, a lower step size can be configured.

As illustrated by block609, the RL agent283then takes an QoS enforcement decision, i.e., based on the wQoE, selects a QoS enforcement action from the QoS enforcement action space. This selection may also depend on whether the RL agent283operates in the exploration mode or the exploitation mode and on a control policy. The control policy may be pre-configured or be indicated as part of the PCC rule. Further, the control policy may be a result of RL based optimization by the RL agent283or by some other RL mechanism, e.g., in another UPF operating in a field environment, laboratory environment, or in a simulated environment.

As indicated by610, the RL agent283then sets the QoS enforcement by the QoS enforcement module282in accordance with the decision of block609. For this purpose, the RL agent indicates the asID and the selected QoS enforcement action to the QoS enforcement module282.

The processes explained in connection with elements603to610ofFIG.6Aneed to be performed only for the first detected data packet that matches the PDR(s). For subsequent matching data packets the asID may be identified from the mapping stored at block604, and the processing may be controlled in accordance with the asID, i.e., the subsequent data packets may be assigned to the existing application session and processed accordingly. InFIG.6A, such subsequent data packets are illustrated by application traffic611,612from the AF240and/or the UE10.

As illustrated by block614, upon detecting the subsequent data packets of the application traffic611,612, the UPF logic281adds the corresponding asID as metadata to the application data traffic. As illustrated by614, the UPF logic281then sends the application traffic with the metadata to the QoE estimation module284.

As indicated by block615, the QoE estimation module284analyzes the application traffic, e.g., by classifying data packets and/or extracting parameters. In some scenarios, this analysis may be based on a machine learning algorithm, e.g., using the aQoE as feedback information. The QoE estimation module284may then collect the information obtained by the analysis for multiple data packets processed during a QoE estimation period. As illustrated by616, the QoE estimation module284then provides the application traffic with the metadata indicating the asID to the QoS enforcement module282.

As indicated by block617, the QoS enforcement module282then performs QoS enforcement action corresponding to the asID on the application traffic. This specifically involves enforcing the QoS rule(s) corresponding to the QoS enforcement action. The QoS enforcement module282then removes the metadata with the asID and forwards the application traffic towards its destination, i.e., to the UE10or AF240, as indicated by618and619.

The processes explained in connection with elements611to619may be performed with respect to each data packet of the detected application data traffic.

FIG.6Bfurther illustrates processes used in the RL based optimization of the control policy applied by the RL agent283for selecting the QoS enforcement actions. The processes of FIG.6B may be applied in the course of processing data packets as explained in connection with elements611to619ofFIG.6A.

Specifically, after having processed one or more data packets as explained in connection with elements611to619ofFIG.6A, the QoE estimation module284obtains a new estimate of the QoE from the information collected in the analysis of block615, as indicated by block621. For example, the QoE estimation module284may check if the QoE estimation period has ended and then evaluate the collected information. A typical duration of the QoE estimation period may be about 10 s. Alternatively or in addition, evaluation of the collected information to obtain a new estimate of the QoE could be triggered by reaching a certain number of analyzed data packets, e.g., 100 data packets. The QoE estimation module284may estimate the QoE in terms of a MOS level.

Based on the new estimate of the QoE, the QoE estimation module284then determines the new state of the environment285and calculates the corresponding reward, as indicated by block622. In some scenarios, the state can be the estimate of the QoE itself. The reward could be calculated as the difference between the estimated QoE and the wQoE. However, more complex computation models could be used as well, considering other parameters such as earlier estimates of the QoE, application traffic parameters like the current application session throughput, or other parameters extracted from analysis of data packets at block615. Further, the calculation of the reward could also use various other information available at the UPF280, e.g., network load status. For example, the reward could be lowered in response to the network load status indicating a congestion. Further, the reward could also consider estimates of the QoE from other sources, e.g., a real-time estimate of the QoE made available to the UPF280by an analytics process. Here, the reward could be lowered in response to obtaining an additional estimate indicating a low QoE or the reward could be raised in response to obtaining an additional estimate indicating a high QoE.

As illustrated by623, the QoE estimation module284then indicates the asID, the state, and the reward to the RL agent283. Based on the indicated state and reward, the RL agent283learns the effects of the past QoS enforcement decision(s), as illustrated by block624, and may adapt its control policy in view of optimizing future QoS enforcement decisions. This learning may involve that the RL agent determines an optimized mapping of states to QoS enforcement actions. The learning may aim at maximizing a cumulated reward of the QoS enforcement actions.

As illustrated by block625, the RL agent283then takes a new QoS enforcement decision, i.e., based on the wQoE, selects a QoS enforcement action from the QoS enforcement action space. This new QoS enforcement decision is based on the adapted policy rule. Further, the new QoS enforcement decision may also depend on whether the RL agent283operates in the exploration mode or the exploitation mode.

As indicated by626, the RL agent283then newly sets the QoS enforcement by the QoS enforcement module282in accordance with the decision of block625. For this purpose, the RL agent indicates the asID and the selected QoS enforcement action to the QoS enforcement module282.

The processing of the application traffic may then continue based on the newly set QoS enforcement, using processes as explained in connection with elements611to619ofFIG.6A.

In view of the above, the illustrated concepts provide a method which allows an AF node, e.g., the above-mentioned AF240, to request a wireless communication network to provide a desired QoE for user data traffic of a service or application. In this method the AF node determines the desired QoE for the user data traffic. Further, the AF node may also determine an actual QoE for the user data traffic, e.g., based on QoE measurements. These QoE measurements may be performed at end points transmitting the user data traffic, e.g., the AF node and/or at a UE. The measurements may be based on various types of QoE measurement methods, including subjective methods relying on human ratings and/or objective methods using models and metrics to approximate subjective human ratings. The AF node then transmits a request for setting up an AF session with a required QoS to a PCF node, e.g., the above-mentioned PCF250. The request is transmitted via an NEF node, e.g., the above-mentioned NEF220. The request indicates the desired QoE and optionally also the actual QoE. In order to enable service-specific handling of the QoE control, the request may also include an identifier of the service or application, such as the above-mentioned application identifier.

The PCF node then transmits a control policy to an SMF node, e.g., the above-mentioned SMF270. As mentioned above, this control policy may include a PCC rule, the identifier of the service or application, the desired QoE and optionally the actual QoE. The SMF node further indicates the control policy to an UPF node, e.g., the above-mentioned UPF node280. As mentioned above, the SMF node may indicate the control policy in terms of a QER including the desired QoE and optionally the actual QoE. Further, the SMF node may also indicate a PDR for identifying the user data traffic and an FAR for the user data traffic.

Based on the indicated control policy, the UPF node detects the user data traffic and enforces one or more QoS rules to provide the wanted QoE. The selection of the QoS rules to be enforced is based on a RL mechanism which selects QoE enforcement actions each including one or more QoS rules. The RL mechanism is based on an RL agent, e.g., the above-mentioned RL agent283, which acts on an environment provided by a QoE estimator, e.g., the above-mentioned QoE estimation module284. The QoE estimator estimates QoE of the user traffic, which may be accomplished by monitoring characteristics of the user data traffic. For the RL mechanism, the estimated QoE represents a state of the environment. The QoE estimator further calculates a reward corresponding to the state. The calculation of the reward is based on the desired QoE and optionally on the actual QoE as indicated to the UPF node.

As mentioned above, the UPF node may assign an application session identifier, such as the asID, to the detected user data traffic and add the application session identifier to the user data traffic. The RL mechanism may then utilize this application session identifier to identify the user data traffic to be processed.

At a given time step, the RL agent select a QoE enforcement action based on the control policy. The QoE enforcement action includes one or more QoS rules to be enforced by a QoS enforcer of the UPF node, e.g., the above-mentioned QoS enforcement module282. The RL agent may indicate the QoE enforcement action together with the application session identifier to the QoS enforcer. The QoS enforcer then enforces the QoS rule(s) corresponding to the selected QoE enforcement action on the user data traffic.

At a next time step, the QoE estimator estimates the QoE resulting from the QoE enforcement action, updates the state of the environment accordingly, and calculates the corresponding reward. The QoE estimator indicates the state and reward to the RL agent. Based on the state and the reward, the RL agent may update the control policy for selecting QoE enforcement actions, with the aim of learning an optimized control policy for selecting QoE enforcement actions.

The procedures described for the time step and the next time step may be iterated for subsequent time steps. Further, the learnt optimized control policy may be stored to be applied in an exploitation mode of the UPF node or by another UPF node.

FIG.7shows a flowchart for illustrating a method of controlling user data traffic in a wireless communication network. The method ofFIG.7may be utilized for implementing the illustrated concepts in a node which is responsible for forwarding the user data traffic to or from a UE connected to the wireless communication network, e.g., like described the above-mentioned UPF280or the above-mentioned GW150, which may correspond to a UPF, a PGW, or a GGSN.

If a processor-based implementation of the node is used, at least some of the steps of the method ofFIG.7may be performed and/or controlled by one or more processors of the node. Such node may also include a memory storing program code for implementing at least some of the below described functionalities or steps of the method ofFIG.7.

At step710, the node receives data indicating a desired QoE level for user data traffic of a user of the wireless communication network. The desired QoE level may be user specific. The desired QoE level may also be specific to a service or application generating the user data traffic. The above-mentioned wQoE is an example of such desired QoE level. The desired QoE level may originate from a traffic endpoint generating or receiving the user data traffic, e.g., from a node providing a service or application that generates the user data traffic, such as the above-mentioned service platform180or AF240.

In some scenarios, the node may receive the data indicating the desired QoE level in response to starting of a service or application generating the user data traffic, e.g., in a procedure for configuring service specific QoS handling of the user data traffic, e.g., as explained in connection withFIG.5.

In some scenarios, the node may indicate a capability of controlling the user data traffic in accordance with the desired QoE to at least one further node of the wireless communication network, e.g., when negotiating capabilities with the at least one further node. An example of a corresponding capability indication is explained in connection withFIG.4.

At step720, the node may further receive data indicating an actual QoE level of the user data traffic. The actual QoE level may be user specific. The actual QoE level may also be specific to a service or application generating the user data traffic. The above-mentioned aQoE is an example of such actual QoE level. The actual QoE level can for example be measured at a traffic endpoint generating the user data traffic, e.g., at a node providing a service or application that generates or receives the user data traffic, such as the above-mentioned service platform180or AF240, or at a UE10that generates or receives the user data traffic, such as the above-mentioned UE10. The measurement of the actual QoE at the traffic endpoint(s) may allow for an accurate measurement of the QoE level, e.g., based on human ratings and/or based on mechanisms that approximate human ratings.

At step730, the node determines a rule for controlling the user data traffic. This determination is based on a control policy. The rule and the control policy may be user specific. The rule and the control policy may also be specific to a service or application generating the user data traffic. The QoE enforcement decision of block609inFIG.6Aand the QoE enforcement decision of block625inFIG.6Bare examples of such determination of a rule for controlling the user data traffic. The rule determined at step730may include one or more QoS rules to be enforced by the node when handling the user data traffic.

At block740, the node obtains data indicating an estimated QoE level for the user data traffic subject to control according to the rule determined at step730. The estimated QoE level may be user specific. The estimated QoE level may also be specific to a service or application generating the user data traffic. The node may obtain the data indicating an estimated QoE level by monitoring the user data traffic and estimating the QoE based on the monitored user data traffic. That is to say, the node itself may estimate the QoE level, e.g., by using a QoE estimator like the above-mentioned QoE estimation module284. However, the node could also receive at least a part of the data indicating the estimated QoE level from another source, e.g., from another node of the wireless communication network.

At block750, the node adapts the control policy based on the data indicating the desired QoE level received at step710and the data indicating the estimated QoE level obtained at step740. In some scenarios, the node may adapt the control policy based on the data indicating the desired QoE level received at step710, the data indicating the actual QoE level received at step720, and the data indicating the estimated QoE level obtained at step740.

At block760, the node may forward the user data traffic. In particular, the node may forward the user data traffic to or from a UE connected to the wireless communication network. When forwarding the user data traffic, the node may apply the rule determined at step730.

In some scenarios, the node may adapt the control policy based on an RL algorithm. In such scenarios, computation of a reward of the RL algorithm may be based on the data indicating the desired QoE level received at step710. Further, computation of a state of the RL algorithm may be based on the data indicating the estimated QoE level obtained at step740. Further, the control rule may correspond to an action from an action space of the RL learning algorithm, such as the above-mentioned QoS enforcement actions selected by the RL agent283. In some scenarios, computation of the reward and/or of the state of the RL algorithm may be further based on the data indicating the actual QoE level received at step720.

FIG.8shows a block diagram for illustrating functionalities of a network node800which operates according to the method ofFIG.7. The network node800may for example correspond to a user plane gateway, such as the above-mentioned GW150or the above-mentioned UPF280. As illustrated, the network node800may be provided with a module810configured to receive data indicating a desired QoE level, such as explained in connection with step710. Further, the network node800may be provided with a module820configured to receive data indicating an actual QoE level, such as explained in connection with step720. Further, the network node800may be provided with a module830configured to determine a rule for controlling user data traffic, such as explained in connection with step730. Further, the network node800may be provided with a module840configured to obtain data indicating an estimated QoE, such as explained in connection with step740. Further, the network node800may be provided with a module850configured to adapt a control policy, such as explained in connection with step750. Further, the network node800may be provided with a module860configured to forward user data traffic, such as explained in connection with step760.

It is noted that the network node800may include further modules for implementing other functionalities, such as known functionalities of a user plane gateway of a wireless communication network. Further, it is noted that the modules of the network node800do not necessarily represent a hardware structure of the network node800, but may also correspond to functional elements, e.g., implemented by hardware, software, or a combination thereof.

FIG.9shows a flowchart for illustrating a method of controlling user data traffic transmission in a wireless communication network. The method ofFIG.9may be utilized for implementing the illustrated concepts in a node that interacts with one or more other nodes of the wireless communication network to enable control of the user data traffic with respect to QoE, such as the above-mentioned controller160, service platform180, NEF220, AF240, PCF250, or SMF270. In some scenarios, the node may correspond to a traffic endpoint of the user data traffic, e.g., to the above-mentioned service platform180or AF240.

If a processor-based implementation of the node is used, at least some of the steps of the method ofFIG.9may be performed and/or controlled by one or more processors of the node. Such node may also include a memory storing program code for implementing at least some of the below described functionalities or steps of the method ofFIG.9.

At step910, the node provides data to a further node of the wireless communication network. The data indicate a desired QoE level for user data traffic of a user of the wireless communication network. The further node may be a node that is responsible for forwarding the user data traffic to or from a UE connected to the wireless communication network, e.g., like described the above-mentioned UPF280or the above-mentioned GW150, which may correspond to a UPF, a PGW, or a GGSN. In some scenarios, the node may provide the data indirectly via other nodes to the further node.

The desired QoE level may be user specific. The desired QoE level may also be specific to a service or application generating the user data traffic. The above-mentioned wQoE is an example of such desired QoE level. The desired QoE level may originate from a traffic endpoint generating or receiving the user data traffic, e.g., from a node providing a service or application that generates the user data traffic, such as the above-mentioned service platform180or AF240.

At step920, the node provides further data to the further node. The further data indicate an actual QoE for the user data traffic. In some scenarios, the node may provide the data indirectly via other nodes to the further node. The actual QoE level may be user specific. The actual QoE level may also be specific to a service or application generating the user data traffic. The above-mentioned aQoE is an example of such actual QoE level. The actual QoE level can for example be measured at a traffic endpoint generating the user data traffic, e.g., at a node providing a service or application that generates or receives the user data traffic, such as the above-mentioned service platform180or AF240, or at a UE10that generates or receives the user data traffic, such as the above-mentioned UE10. The measurement of the actual QoE at the traffic endpoint(s) may allow for an accurate measurement of the QoE level, e.g., based on human ratings and/or based on mechanisms that approximate human ratings.

The data and the further data provided to the further node may enable RL-based QoE control by the further node.

At step930, the node may monitor the actual QoE level. If the node corresponds to a traffic endpoint of the user data traffic, this monitoring may be based on user data traffic generated by the node and/or based on user data traffic received by the node.

In some scenarios, the node may receive an indication of a capability of the further node to control the user data traffic in accordance with the desired QoE level. In this case, the node perform the steps910,920of providing the data and the further data to the further node in response to receiving the indication.

FIG.10shows a block diagram for illustrating functionalities of a network node1000which operates according to the method ofFIG.9. The network node1000may for example correspond to a node that interacts with one or more other nodes of the wireless communication network to enable control of the user data traffic with respect to QoE, such as the above-mentioned controller160, service platform180, NEF220, AF240, PCF250, or SMF270. In some scenarios, the node may correspond to a traffic endpoint of the user data traffic, e.g., to the above-mentioned service platform180or AF240. As illustrated, the network node1000may be provided with a module1010configured to provide data indicating a desired QoE to a further node, such as explained in connection with step910. Further, the network node1000may be provided with a module1020configured to provide further data indicating an actual QoE to the further node, such as explained in connection with step920. Further, the network node1000may be provided with a module1030configured to monitor the actual QoE, such as explained in connection with step930.

It is noted that the network node1000may include further modules for implementing other functionalities, such as known functionalities of an AF, NEF, PCF, SMF or similar node. Further, it is noted that the modules of the network node1000do not necessarily represent a hardware structure of the network node1000, but may also correspond to functional elements, e.g., implemented by hardware, software, or a combination thereof.

Further, it is noted that the functionalities described in connection withFIGS.7to10may also be combined in a system including a node operating according to the method ofFIG.9and a further node operating according to the method ofFIG.7. In such system, the data received at step710may correspond to the data provided at step910, and the data received at step720may correspond to the data provided at step920.

FIG.11illustrates a processor-based implementation of a network element1100which may be used for implementing the above-described concepts. For example, the structures as illustrated inFIG.11may be used for implementing any of the above-described nodes150,160,180,220,240,250,270,280. In some scenarios, also a system of multiple network elements1100with structures as illustrated inFIG.11may be used for implementing any of the above-described nodes150,160,180,220,240,250,270,280.

As illustrated, the network element1100includes one or more interfaces1110. These interfaces may for example be used for enabling communication with other node. The interfaces may for example be used for implementing one or more of the reference points shown inFIG.2.

Further, the network element1100may include one or more processors1150coupled to the interface(s)1110and a memory1160coupled to the processor(s)1150. By way of example, the interface(s)1110, the processor(s)1150, and the memory1160could be coupled by one or more internal bus systems of the network element1100. The memory1160may include a Read-Only-Memory (ROM), e.g., a flash ROM, a Random Access Memory (RAM), e.g., a Dynamic RAM (DRAM) or Static RAM (SRAM), a mass storage, e.g., a hard disk or solid state disk, or the like. As illustrated, the memory1160may include software1170and/or firmware1180. The memory1160may include suitably configured program code to be executed by the processor(s)1150so as to implement the above-described functionalities of a network node, such as explained in connection withFIGS.7to10.

It is to be understood that the structures as illustrated inFIG.11are merely schematic and that the network element1100may actually include further components which, for the sake of clarity, have not been illustrated, e.g., further interfaces or processors. Also, it is to be understood that the memory1160may include further program code for implementing known functionalities of a network element, e.g., known functionalities of a control plane or user plane nodes of a 3GPP network. According to some embodiments, also a computer program may be provided for implementing functionalities of the network element1100, e.g., in the form of a physical medium storing the program code and/or other data to be stored in the memory1160or by making the program code available for download or by streaming.

As can be seen, the concepts as described above may be used for efficiently controlling user data traffic with respect to QoE. In particular, a service or application provider may provide the wireless communication network with information on a desired QoE level and optionally also the actual QoE level of user data traffic, thereby enabling the wireless communication network to control the user data traffic with the aim of achieving the desired QoE level. The latter control may be accomplished in an efficient way by utilizing an RL mechanism. As a result, static configurations of QoS may be avoided and adaptation to changing conditions or individual characteristics of involved nodes or deployment scenarios may be facilitated. Further, the amount of required human intervention may be reduced. Still further, QoE control may also be enabled for encrypted user data traffic.

It is to be understood that the examples and embodiments as explained above are merely illustrative and susceptible to various modifications. For example, the illustrated concepts may be applied in connection with various wireless communication network technologies, without limitation to the NR technology. For example, when using the LTE technology, functionalities as explained for the gateway150and the UPF280could be implemented by a PGW of the LTE technology, and the PDU session could correspond to a PDN (Packet Data Network) connection. Further, when using the UMTS technology, the functionalities as explained for the gateway150and the UPF280could be implemented by a GGSN of the UMTS technology, and the PDU session could correspond to a Radio Access Bearer.

Further, it is noted that the exploration mode and exploitation mode of the RL agent could be used in different environments. For example, the exploration mode could be used in a controlled environment, e.g., a laboratory, while the exploitation mode could be used in a field environment, during operation at a deployment site. In some scenarios, it is also possible to apply a control policy learnt by an RL agent of a certain node in another node. Another possibility is to use existing production data to pre-train the RL agent. This may be utilized to avoid extensive exploration phases in a field environment.

Further, the concepts may be applied with respect to various types of machine learning algorithms, without limitation to RL algorithms. Moreover, it is to be understood that the above concepts may be implemented by using correspondingly designed software to be executed by one or more processors of an existing device or apparatus, or by using dedicated device hardware. Further, it should be noted that the illustrated nodes may each be implemented as a single device or as a system of multiple interacting devices or modules, e.g., as a cloud system.