Patent Description:
Video is one of the most important drivers of mobile telecom business today-and as with many other forms of data, companies who produce OTT video services have seen an increased demand for supporting end-to-end encryption on their platforms. With this growing expectation, and projections that mobile video traffic will grow by around fifty percent annually through <NUM> to eventually account for seventy-five percent of all mobile data traffic, there is likely to be a high demand for systems that can deliver encrypted OTT video data.

The high-quality mobile broadband service is an expectation from both the OTT service providers and the end user side, and it is considered as key business factor. Therefore, mobile network operators take special care to monitor service quality and to improve their networks in order to provide a high-quality video service from both the client and operator perspective. With that in mind, video quality metrics have become very important to ensure a high level of quality is maintained from end to end and can be obtained at the end device or based on traffic collected at the end devices. As described further below, however, the application of these methods is quite limited.

Presently, to monitor the quality of large number of OTT flows, mobile network operators apply traffic probes that collect different metrics of the OTT flows and estimate/calculate OTT data (e.g., video) key performance indicators (KPIs). A current challenge in this area is that different service providers use different video encoding techniques or protocols, which are changed from time to time. Accordingly, the video KPIs must be tuned for each of the encoding techniques used by all OTT service providers and types. Another significant challenge with the current paradigm is that the video content is now encrypted, therefore there is no possibility to sample the content directly in the network probes.

To date, terminal- or user-equipment-based techniques for monitoring a large number of OTT flows have been generally unsuccessful-often because it requires agreement of subscribers to deploy the monitoring software on their terminal (also referred to herein as user equipment (UE)). In addition, the processing capacity of the mobile terminals is limited and any OTT flow monitoring application would divert non-negligible resources from other user applications, among other factors that make such a solution impractical.

Furthermore, existing machine-learning-based techniques exclusively utilize network node events (e.g., Radio Access Network (RAN) events such as cell handover, etc.) to infer network and OTT data flow conditions. These existing techniques also require interfaces to access the network nodes and also require different types of data from multiple data sources. Moreover, these techniques are driven by buffer state models using packet probe traffic reports as input, where the models are based on predefined rules and model parameters determined by offline data analysis or laboratory measurements instead of actual measurements inside the particular communication environment. In addition, the model parameters vary significantly among OTT service providers-not to mention that the data flow packet encoding can change over time. Accordingly, it requires a gargantuan amount of effort and expert knowledge to determine and maintain the parameters for the large number of OTT service providers.

One significant limitation of the existing machine-learning techniques is that each provides only a single metric (e.g. Mean Opinion Square (MOS), or per-session KPIs) for an entire session, which does not provide a sufficient level of granularity for the network devices to determine and troubleshoot specific issues with OTT flows.

Accordingly, improvements are necessary to allow modern networks to identify fault points in an OTT data delivery path such that any issues can be more easily addressed to improve user experience.

<CIT> discloses determining video quality metrics for a video stream played at a wireless device even when a video stream is end-to-end encrypted based on using quality assessment module using machine-learning processes such as neural networks.

The present disclosure presents a real-time video quality analytics system a neural network based and method for mobile networks is proposed with an automatic learning mechanism, which can adapt automatically to video encodings used by different OTT service providers and adopt to any changes of video encoding during operation. The invention provides a method in a recurrent neural network, RNN, per claim <NUM> and its corresponding system claim <NUM>. Dependent claims provide preferred embodiments of the invention.

These and additional example embodiments will now be described in reference to the figures that follow.

The present disclosure introduces a data communication system that implements a recurrent neural network (RNN)-based real-time encrypted OTT data (e.g., video) quality analytics system automatic learning mechanism that can adapt automatically to encoding types used by different OTT service providers. The system uses fine granularity flow quality metrics for encrypted OTT data enabling troubleshooting and analytics use-cases. In examples where the OTT data is video data, video quality metrics include the initial buffering time, rebuffering or stall time ratio, video resolution.

The video quality analysis is based on relatively fine granularity (e.g. one to two seconds) packet probe records, including transport activity, throughput, and Transfer Control Protocol (TCP) metrics, along with relevant network node events (handovers, etc.) and fine granularity measurement reports. The video quality metrics are estimated real-time by a recurrent neural network (RNN), which uses packet probe records and node events as a periodic and/or an event-based input feed. Because of the relatively fine temporal granularity of the metric generation and OTT flow analysis by the RNN, the proposed solution described herein enables cross-domain root cause identification, analysis, and troubleshooting of encrypted OTT video session problems.

In an aspect of example embodiments described herein, at least one packet probe is deployed in or adjacent to the gateway node(s) of the network and configured to capture OTT-based traffic of the full network and providing packet probe reports separately for each OTT data session (also referred to herein as an OTT data flow). In an aspect, radio events and measurement reports are collected for all or part of the network and are correlated to the OTT data sessions. The RNN is further configured to implement adaptive learning based on a limited number of test terminals spread around the network. The test terminals run a terminal application that is configured to measure and report quality metrics, user experience parameter levels, and/or other relevant metrics. The weights of the RNN connections can be updated continuously based on the feedback of test terminal (also referred to herein as test user equipment). In addition, the RNN can utilize the data input from the one or more packet probes, the test terminals, and network nodes root cause analysis additional node and network events are correlated to the identified quality issues.

Unlike existing solutions, there is no need to develop and maintain models separately for each OTT service provider in the present embodiments. Instead, the example embodiments presented herein can automatically learn based on feedback from previous inbox data and adapts to encoding changes accordingly. In addition, the RNN can be configured to automatically recognize, decipher, and learn new video encoding types of OTT service providers.

Using these and other techniques described herein, the proposed solution identifies video/OTT data quality issues in effectively real time (one to two second granularity in some examples) and for all subscribers. Given these features of the present disclosure, the method can be used for customer care services, troubleshooting, network analysis and optimization, and as input to closed-loop network optimization self-organizing network (SON) features. Ultimately, the output of the system described herein can be used to localize the quality issues within the session with high accuracy such that painless and swift root cause analysis can be conducted for system optimization.

<FIG> illustrates a data communication system <NUM> that is configured to communicate over-the-top (OTT) data from one or more OTT service providers <NUM> to one or more UEs <NUM>. As shown in <FIG>, to receive OTT data <NUM> from the one or more OTT service providers <NUM>, the one or more UEs <NUM> communicate wirelessly with at least one radio access network (RAN) <NUM> via a network node <NUM> (also referred to as radio network node or device, e.g., an eNB, gNB, base station, nodeB, etc.). In addition to traversing the RAN <NUM> on its last leg of its path from its corresponding OTT service provider <NUM>, the OTT data <NUM> also traverses core network devices, including a gateway <NUM> (over S1-U interface) and, in some instances, Mobility Management Entity (MME) <NUM> over S11 and S1-MME interfaces. The gateway <NUM> can interface with the one or more Internet-based OTT service providers <NUM>.

In a further aspect of the disclosure, on its path from the OTT service providers <NUM> to the one or more UEs <NUM>, the OTT data <NUM> can encounter one or more packet probes <NUM> configured to perform packet inspection (e.g. Deep Packet Inspection (DPI)), which can include analyzing the data packets of the OTT data <NUM> to determine characteristics of the OTT data streams traversing the data communication system <NUM>. In particular, the packet probes <NUM> can be configured to parse the OTT data streams to determine a type of encryption used by the OTT service providers <NUM> in each data stream (also referred to herein as "session"), among other characteristic information.

In some instances, the packet probe or probes <NUM> can be entities or modules inside a network device (e.g., gateway <NUM>) such that the device performs multiple functions (i.e., gateway functionality and probe functionality). Alternatively, the one or more packet probes <NUM> can be stand-alone devices. Furthermore, different packet probes <NUM> may be located logically and/or physically on different communication interfaces and may be tasked with obtaining specific information pertinent to the overall OTT data traffic quality analysis. For instance, in an example embodiment shown in <FIG>, OTT data traffic <NUM> (e.g., video data traffic) is captured by a first packet probe 120a at the S1-u interface, which communicatively couples the gateway <NUM> to the network node <NUM> of the RAN <NUM> serving one or more destination UEs <NUM> that are to receive the OTT data in the probed data flow/session. This first packet probe 120a on the S1-u interface can be configured to classify the OTT traffic of the different OTT sessions/flows by destination/target IP address(es) and/or port(s). In addition, as is likewise shown in the example embodiment of <FIG>, a second packet probe 120b can be situated on the S11 interface to capture GTP-C signaling, which associates a particular video session to a particular user and/or user equipment <NUM>.

In addition, the one or more packet probes <NUM> can include different probe types that broaden the functionality of the packet probe <NUM>. For example, in some embodiments, the one or more packet probes <NUM> can include an Evolved Packet System (EPS) Encryption Algorithm (EEA) probe, which can be configured to classify Transmission Control Protocol (TCP) data, such as OTT data <NUM>, according to originating OTT service provider <NUM>, subscriber identity (i.e. of the terminating terminal <NUM>), operator identity (i.e., of the RAN serving the terminating terminal <NUM>), and the like. In some examples, an EEA correlator can be included in the one or more packet probes <NUM> to detect individual video session boundaries and to extend the transport layer information with RAN events and measurement reports. For purposes of the present disclosure, where.

In addition, the one or more packet probes <NUM> can include a General Packet Radio Service (GPRS) Tunneling Protocol (GTP) probe, which can be configured to generate fine-granularity transport layer information (called Transport layer Session Records (TSRs)) for each video session that is analyzed. Specifically, the transport layer information available in these TSR reports can include one or more of the following (which forms a non-limiting list):.

The information introduced above (or a subset thereof) can be generated by the one or more packet probes <NUM> according to a probing periodicity and/or based on one or more triggers, including event triggers (e.g., a particular parameter value reaches or drops below a threshold value for that parameter). In some examples, the packet probes can process OTT data and generate information and video session-specific performance parameter values at a temporal granularity of approximately one to two seconds. In other examples, the granularity may, for performance or feasibility reasons, be greater than two seconds or less than one second. This probing periodicity can be configured by a network operator and/or may be dynamic and negotiated among the different networks (e.g., RANs) in the system <NUM> based on current system conditions.

In a further aspect, the data communication system <NUM> includes one or more test terminals 102T that are configured to observe OTT data communication conditions at the test terminal location as well as identify certain events of consequence to the overall analysis of the OTT data delivery of the system as a whole. Accordingly, like the one or more packet probes <NUM> introduced above, the test terminals 102T are configured to generate and output data to the RNN <NUM> that is germane to the ultimate system analysis and the corresponding data generated by the RNN <NUM> that is utilized for problem point identification and potential rectification. In an aspect, set of data and/or events monitored by the test terminals 102T can include, but is not limited to the TSR parameters discussed above (e.g., in relation to the one or more probes <NUM>), radio and/or overall call measurements, certain radio events such as, but not limited to, call/link drops, cell handovers, reselection (therefore including some of the same events that can be used for RCA purposes, see below), etc..

In an aspect, the test terminals 102T can be configured to explicitly measure OTT data quality metrics, and based on these metrics, can generate a record for each OTT data session. Furthermore, in some instances, the test terminals 102T can maintain a list of known OTT service providers as well as general and OTT-service-provider-specific video encoding schemes in order to identify and process the OTT data according to such encryption schemes. Furthermore, to maximize the set of training data available to the RNN <NUM>, the test terminals 102T continuously run OTT data sessions for the existing OTT services and sites and report the information garnered from these sessions to the RNN <NUM>.

In case of change in encoding scheme for the OTT data, the test terminals can begin measuring the new traffic type and provide quality metrics for the new encoding. Furthermore, in the case of a new OTT service or OTT service provider, the terminals can explicitly be requested to run OTT data sessions for the new traffic types to ensure training samples are generated for the new traffic type in short order (e.g., within one hour or the like).

In addition, the test terminals 102T can be placed geographically in a manner that maximizes knowledge of the system footprint. Accordingly, in addition to the information output to the RNN <NUM> from the one or more packet probes <NUM>, the test terminals 102T can provide important information to be utilized by the RNN <NUM> to better understand the behavior of the system <NUM> and to more clearly and more quickly identify particular problem paths, interfaces, devices, or the like so that these problems can be addressed (or avoided through routing techniques) to improve the performance of the system as a whole. The number of test terminals 102T and test sessions may bel orders of magnitude less than the number of analyzed OTT data sessions for the whole system. As an example, with <NUM> test terminals the OTT data sessions of <NUM> million subscribers can be analyzed. This allows, for example, a system <NUM> with <NUM> test terminals 102T to analyze the OTT data sessions of <NUM> million subscribers while having a much smaller overhead footprint than if all terminals and/or OTT sessions were leveraged for this test terminal function.

In addition to information output by test terminals 102T and the one or more packet probes <NUM>, additional information can be obtained by one or more network nodes <NUM> and provided to the RNN <NUM> to improve its performance and overall analysis of the system <NUM>. In an aspect, each of the one or more network nodes <NUM> can be, in some non-limiting embodiments, nodeBs, eNodeBs (eNBs), gNBs, or other network-side devices that can obtain gainful information regarding the OTT data flow through the system <NUM>. In particular, the network nodes <NUM> can be configured to conduct radio measurements and detect network node events (i.e. terminal handovers, etc.). In an aspect, the node events and probe records from the one or more packet probes <NUM> can be are correlated by the RNN <NUM> in a per-session record in its analytics system. According to example embodiments, information that is collected by the network nodes <NUM> and correlated with probe records can include (non-limiting list):.

By correlating these OTT data quality metrics with the radio events and measurements, the RNN <NUM> can provide a set of probable or real high-level root causes of performance issues, such as a lack of coverage, interference, x2 or s1 handover issues, cell dominance problems, HARQ rate issue, and others (not limiting).

Moreover, as introduced above, data communication system <NUM> includes a recurrent neural network (RNN) <NUM> configured to obtain a variety of information from other devices in the system <NUM> and, based on the obtained data, draw conclusions about OTT data flow conditions and overall system behavior and to improve its accuracy and performance over time. For example, the RNN <NUM> can receive information (see above) the one or more packet probes <NUM>, one or more network nodes <NUM> or other network-side devices (e.g., in the core network), one or more user terminals <NUM>, including test terminals 102T, or any other device in system <NUM> (or external thereto).

Specifically, in some examples, information input of the RNN <NUM> can be structured via a sliding time window (e.g., <NUM>-<NUM> seconds), with the information including the one-second probe report(s) extended with parameters input from other sources, including the radio network measurements from the test terminals 102T, other network nodes <NUM>, and including network events including those listed above. In some examples, the sliding window is stepped each second, though stepping using other time intervals is possible in some examples.

As introduced above, the RNN <NUM> is configured to analyze the input information from the above-recited sources to analyze overall system performance and identify problem locations in the system <NUM> such that they can be rectified. <FIG> is a neural network diagram showing the logical structure of the RNN <NUM>, which includes an input layer <NUM>, a hidden layer <NUM>, and an output layer <NUM>. In an aspect, the input layer <NUM> constitutes a logical representation of the correlation module <NUM> wherein the parameters input from external entities (e.g., one or more packet probes <NUM>, network nodes <NUM>, and testing terminals 102T) are grouped according to OTT flow <NUM>, time, packet order, and the like such that the later layers are provided with properly organized information. The hidden layer <NUM> includes feedback that encapsulates the previous state of the RNN in the decision making. The output layer <NUM> corresponds to the OTT (i.e., video) quality metrics utilized for system analysis: video buffering, video stall ratio (indicating that video is frozen), video resolution, and the like for video as OTT data implementations. Furthermore, the RNN <NUM> can include an analytics system that implements a correlation function. In an aspect, the correlation function matches the received probe records/information in real time with the events corresponding to the same OTT data session but potentially generated by and obtained from one or more test terminals 102T and optionally from other nodes <NUM>. In order to capture late events and match them to a correct OTT data session, the correlation function waits for events for at least a time out period (which can be on the order of, but not limited to, <NUM>-<NUM> seconds).

Once generated, the correlated event records can be sent to an analyzer <NUM> of the analytics system in RNN <NUM>, which is configured to parse the obtained data and output performance and characteristic data of OTT sessions, devices, locations, networks, etc. of the system <NUM>. In addition, the analyzer <NUM> can aggregate the OTT data quality KPIs at different dimensions or levels of scale, such as nodes, cells, terminal types, OTT service provider, and the like. In a further aspect, the analytics system can store correlated records for different time periods, which allows the aggregated data to be drilled down to the particular dimension instances. This is used for localizing and deriving the high-level root cause the OTT data quality issues identified by the analytics system of the RNN <NUM>. For example, if OTT data quality is less than optimal for certain node instances, and radio signal parameters (RSRP) are bad for the same node instances, than the most probable root cause of the OTT data quality issue is the bad coverage in the area of the given cells.

In another feature of the present disclosure, the OTT data quality KPIs can be aggregated for terminal types and drilled down to terminal type instances. This drill-down can lead to several insights into the root causes of certain issues with OTT delivery performance. For example, if it is determined by the RNN <NUM> that OTT data quality is significantly worse for a specific terminal type than that for the other terminal types, the RNN <NUM> can indicate that the root cause of the OTT data quality issue is terminal issue such as terminal model incompatibility.

In a further aspect, certain quality metrics are generated by RNN <NUM> in real time or approximately real time based on the actual junction weight between the neurons of the RNN. In some instances, the OTT data quality KPIs are calculated continuously during the session based on the quality metrics. The OTT data quality issues are often associated with specific radio conditions, such as low signal strength, interference, or simply to a handover event. Therefore, the test terminals 102T adding these parameters to the RNN input sequence significantly increase the accuracy of the prediction of the quality metrics.

Furthermore, in parallel to the packet probe data analysis process, the RNN <NUM> and its composite nodes are continually trained based on the OTT data sessions captured by the test terminals 102T. The training samples include existing OTT data encodings that continuously strengthen the existing coupling of the neurons, and as new OTT data encodings are utilized by OTT service providers, a larger number of training sequences of the new OTT data types are also sent to the RNN <NUM> in due course. This introduction of new OTT encoding schemes into the training data input into the RNN <NUM> modifies the existing neuron coupling in a way that generates appropriate quality metrics for the new OTT data types. In some examples, in operation, the one or more packet probes can recognize new OTT encoding types or schemes and can explicitly request the test terminals to generate test samples for the new traffic type, accelerating the learning process on the part of the RNN <NUM>. As time progresses and the RNN <NUM> adapts accordingly, the weights for the unused OTT data types are gradually unlearned.

In an aspect, the output of the RNN <NUM> is generated at the rate at which the information is output by the one or more packet probes <NUM>. For instance, in some examples, should the one or more packet probes <NUM> output information every second, the RNN <NUM> can output the video quality metrics every second as well. The RNN output is used for calculating the video quality KPls, such as initial buffering time, stall time ratio, and the like. Accordingly, the RNN <NUM> is configured to continuously update its output data values informed by previous values input by the one or more packet probes <NUM> and test terminal 102T measurements. Therefore, as introduced in the context of the devices, networks, and interfaces of the system <NUM> of <FIG>, the features of the present disclosure allow a system to leverage an RNN <NUM> to analyze and improve system operation and OTT data delivery.

<FIG> illustrates an example method <NUM> performed by a device <NUM> in a recurrent neural network RNN <NUM> implemented in a data communication system <NUM>. In an aspect, at block <NUM> method <NUM> can include periodically obtaining information corresponding to one or more over-the-top OTT data sessions <NUM> in the data communication system. As introduced above, the information at block <NUM> can be obtained from one or more of a packet probe <NUM>, a test terminal 102T, and a network node <NUM>. Furthermore, the method <NUM> can include, at block <NUM>, generating at least one quality metric associated with the one or more OTT data sessions <NUM> based on the periodically obtained information.

In addition, method <NUM> can include additional or alternative aspects that are not explicitly shown in <FIG>, including those that follow. For instance, in some examples, the generating aspect of block <NUM> can further include analyzing a sliding temporal window of the obtained information, and in a further aspect, the sliding window can be temporally moved periodically. In addition, the method <NUM> can include in some instances aggregating the at least one quality metric according to defined categories and identifying a root cause of an issue with one or more OTT data sessions based on the aggregated at least one quality metric.

In some instances of method <NUM>, the information is obtained from two or more sources and can further include correlating the information received from the two or more sources to a same OOT data session and/or time. The method <NUM> can also include periodically calculating OTT data quality key performance indicators based on the generated at least one quality metric. Furthermore, periodically obtaining the information at block <NUM> can include receiving at least some of the information once every second. Additionally, in some example embodiments, the at least one quality metric comprises a data buffering level and/or a data stall time metric.

Furthermore, in some instances, the RNN <NUM> can include a hidden layer <NUM> managing feedback representing a previous state of the RNN <NUM>. In addition, in some examples, the method <NUM> can include continually training the RNN <NUM> based on the obtained information, including the feedback information obtained, managed, and/or stored at the hidden layer <NUM>. In addition, in some examples, generating the at least one quality metric is based on a junction weight of neurons of the RNN <NUM>.

<FIG> illustrates another example method <NUM> performed by a test terminal 102T in an OTT data communication system <NUM>. In an aspect, the method <NUM> can include, at block <NUM>, conducting an ongoing OTT data session <NUM> with each of a group of available OTT service providers <NUM>. In addition, at block <NUM>, method <NUM> can include, for each ongoing OTT data session <NUM>, measuring one or more OTT data quality metrics. Furthermore, at block <NUM>, method <NUM> can include periodically reporting the one or more OTT data quality metrics for each ongoing OTT data session <NUM> to a recurrent neural network RNN <NUM> associated with the OTT data communication system <NUM>.

In addition, method <NUM> can include additional or alternative aspects that are not explicitly shown in <FIG>, including those that follow. For instance, some examples of method <NUM> can include maintaining a list of encoding formats used by each of the OTT service providers <NUM>. Likewise, method <NUM> can include maintaining a list of the available OTT service providers <NUM> in some examples. Furthermore, the method <NUM> can include reporting the list of encoding formats and/or the list of available OTT service providers <NUM> to the RNN <NUM>.

In addition, in some examples, method <NUM> can include determining that a new encoding format has been introduced for an OTT data session <NUM>, determining which OTT service provider <NUM> is implementing the new encoding format, and/or adding the new encoding format to the list of encoding formats used by the determined OTT service provider <NUM>.

Furthermore, in any of the example embodiments of method <NUM> or <NUM> in <FIG> and <FIG>, at least one of the one or more OTT data sessions <NUM> can be a video data session <NUM>.

<FIG> illustrates additional details of a device <NUM> in an example RNN <NUM> in accordance with one or more embodiments. As shown, the device <NUM> includes processing circuitry (also referred to herein as a processor, one or more processors, processing unit (s) and the like) <NUM> and radio circuitry <NUM>. The radio circuitry <NUM> is configured to transmit via one or more antennas <NUM>. The processing circuitry <NUM> is configured to perform processing described above, e.g., in <FIG>, such as by executing instructions stored in memory <NUM>. The processing circuitry <NUM> in this regard may implement certain functional means or units.

<FIG> illustrates the device <NUM> that according to other embodiments implements various functional means or units, e.g., via the processing circuitry <NUM> (or processor(s) or one or more processors) in <FIG>. As shown, these functional means or units implement the example method presented in <FIG> (other modules/units for doing so may not be explicitly shown). For instance, <FIG> shows that the device <NUM> can include an obtaining module/unit <NUM> configured to periodically obtain information corresponding to one or more over-the-top OTT data sessions <NUM> in the data communication system. As introduced above, the information obtained by the obtaining module/unit <NUM> can be obtained from one or more of a packet probe <NUM>, a test terminal 102T, and a network node <NUM>. Furthermore, the device <NUM> can include a generating module/unit <NUM> configured to generate at least one quality metric associated with the one or more OTT data sessions <NUM> based on the periodically obtained information.

<FIG> illustrates additional details of a test terminal <NUM> (or, more generally, a wireless communication device or user equipment configured to provide feedback to RNN <NUM>) in accordance with one or more embodiments. As shown, the test terminal 102T includes processing circuitry <NUM> and radio circuitry <NUM>. The radio circuitry <NUM> is configured to transmit via one or more antennas <NUM>. The processing circuitry <NUM> is configured to perform processing described above, e.g., in <FIG>, such as by executing instructions stored in memory <NUM>. The processing circuitry <NUM> in this regard may implement certain functional means or units.

<FIG> illustrates network node <NUM> (e.g., a base station) that according to other embodiments implements various functional means or units, e.g., via the processing circuitry <NUM> in <FIG>. These functional means or units, e.g., for implementing the method in <FIG>, include an OTT data session conducting module/unit <NUM> configured to conduct an ongoing OTT data session <NUM> with each of a group of available OTT service providers <NUM>. In addition, the functional means or units can include a measuring module/unit <NUM> configured, for each ongoing OTT data session <NUM>, to measure one or more OTT data quality metrics. Furthermore, the functional means or units can include a reporting module/unit <NUM> configured to periodically report the one or more OTT data quality metrics for each ongoing OTT data session <NUM> to a recurrent neural network RNN <NUM> associated with the OTT data communication system <NUM>.

A terminal <NUM> (including a subset of these terminals <NUM>, test terminal 102T) according to the present disclosure is any type of device capable of communicating with a network node using radio signals. A terminal <NUM> may therefore refer to a user equipment (UE), machine-to-machine (M2M) device, a machine-type communications (MTC) device, a NB-loT device, etc. A terminal <NUM> may also be referred to as a wireless communication device, a radio device, a radio communication device, a wireless terminal, or simply a terminal - unless the context indicates otherwise, the use of any of these terms is intended to include device-to-device UEs or devices, machine-type devices or devices capable of machine-to-machine communication, sensors equipped with a wireless communication device, wireless-enabled table computers, mobile terminals, smart phones, laptop-embedded equipped (LEE), laptop-mounted equipment (LME), Universal Serial Bus (USB) dongles, wireless customer-premises equipment (CPE), etc. In the discussion herein, the terms machine-to-machine (M2M) device, machine-type communication (MTC) device, wireless sensor, and sensor may also be used. It should be understood that these devices may be a terminal <NUM>.

In an loT scenario, a terminal <NUM> as described herein may be, or may be comprised in, a machine or device that performs monitoring or measurements, and transmits the results of such monitoring measurements to another device or a network. Particular examples of such machines are power meters, industrial machinery, or home or personal appliances, e.g. refrigerators, televisions, personal wearables such as watches etc. In other scenarios, a wireless communication device as described herein may be comprised in a vehicle and may perform monitoring and/or reporting of the vehicle's operational status or other functions associated with the vehicle.

Note that the terminal <NUM> (e.g., UE) as described above may perform the processing herein by implementing any functional means or units. In one embodiment, for example, the terminal <NUM> comprises respective circuits or circuitry configured to perform the steps shown in Figures 6A-6E. The circuits or circuitry in this regard may comprise circuits dedicated to performing certain functional processing and/or one or more microprocessors in conjunction with memory. In embodiments that employ memory, which may comprise one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc., the memory stores program code that, when executed by the one or more processors, carries out the techniques described herein.

A computer program comprises instructions which, when executed on at least one processor of equipment (e.g., UE or radio network equipment), cause the equipment to carry out any of the respective processing described above. A computer program in this regard may comprise one or more code modules corresponding to the means or units described above.

In this regard, embodiments herein also include a computer program product stored on a non-transitory computer readable (storage or recording) medium and comprising instructions that, when executed by a processor of equipment, cause the equipment to perform as described above. The example aspects of the disclosure presented below may be included in and/or may add features to one or more embodiments presented in the disclosure above. In some cases, the example aspects of the disclosure presented below form aspects of one or more embodiments that are at least partially distinct from the one or more embodiments presented above. In any case, this Appendix forms part of the present Application.

<FIG> schematically illustrates a telecommunication network connected via an intermediate network to a host computer. With reference to <FIG>, in accordance with an embodiment, a communication system includes a telecommunication network A-<NUM>, such as a 3GPP-type cellular network, which comprises an access network A-<NUM>, such as a radio access network, and a core network A-<NUM>. The access network A-<NUM> comprises a plurality of base stations A-12a, A-12b, A-12c, such as NBs, eNBs, gNBs or other types of wireless access points (e.g. network nodes <NUM>), each defining a corresponding coverage area A-13a, A-13b, A-13c. In an aspect, any of base stations A-12a, A-12b, A-12c, or any other base stations described herein may be considered to be a network node, for instance, if such as network node is described above in the present Application. Each base station A-12a, A-12b, A-12c is connectable to the core network A-<NUM> over a wired or wireless connection A-<NUM>. A first user equipment (UE) A-<NUM> (i.e. terminal <NUM>) located in coverage area A-13c is configured to wirelessly connect to, or be paged by, the corresponding base station A-12c. A second UE A-<NUM> in coverage area A-13a is wirelessly connectable to the corresponding base station A-12a. While a plurality of UEs A-<NUM>, A-<NUM> are illustrated in this example, the disclosed embodiments are equally applicable to a situation where a sole UE is in the coverage area or where a sole UE is connecting to the corresponding base station A-<NUM>. In an aspect, any of these UEs, or any other UE described herein may be considered to be configured to perform the aspects of any UE, user terminal, client device, or mobile device described above in the present Application.

The telecommunication network A-<NUM> is itself connected to a host computer A-<NUM>, which may be embodied in the hardware and/or software of a standalone server, a cloud-implemented server, a distributed server or as processing resources in a server farm. The host computer A-<NUM> may be under the ownership or control of a service provider or may be operated by the service provider or on behalf of the service provider. The connections A-<NUM>, A-<NUM> between the telecommunication network A-<NUM> and the host computer A-<NUM> may extend directly from the core network A-<NUM> to the host computer A-<NUM> or may go via an optional intermediate network A-<NUM>. The intermediate network A-<NUM> may be one of, or a combination of more than one of, a public, private or hosted network; the intermediate network A-<NUM>, if any, may be a backbone network or the Internet; in particular, the intermediate network A-<NUM> may comprise two or more sub-networks (not shown).

The communication system of <FIG> as a whole enables connectivity between one of the connected UEs A-<NUM>, A-<NUM> and the host computer A-<NUM>. The connectivity may be described as an over-the-top (OTT) connection A-<NUM>. The host computer A-<NUM> and the connected UEs A-<NUM>, A-<NUM> are configured to communicate data and/or signaling via the OTT connection A-<NUM>, using the access network A-<NUM>, the core network A-<NUM>, any intermediate network A-<NUM> and possible further infrastructure (not shown) as intermediaries. The OTT connection A-<NUM> may be transparent in the sense that the participating communication devices through which the OTT connection A-<NUM> passes are unaware of routing of uplink and downlink communications. For example, a base station A-<NUM> may not or need not be informed about the past routing of an incoming downlink communication with data originating from a host computer A-<NUM> to be forwarded (e.g., handed over) to a connected UE A-<NUM>. Similarly, the base station A-<NUM> need not be aware of the future routing of an outgoing uplink communication originating from the UE A-<NUM> towards the host computer A-<NUM>.

Example implementations, in accordance with an embodiment, of the UE, base station and host computer discussed in the preceding paragraphs will now be described with reference to <FIG>, which is shown below. <FIG> is a generalized block diagram of a host computer communicating via a base station with a user equipment over a partially wireless connection.

In a communication system B-<NUM>, a host computer B-<NUM> comprises hardware B-<NUM> including a communication interface B-<NUM> configured to set up and maintain a wired or wireless connection with an interface of a different communication device of the communication system B-<NUM>. The host computer B-<NUM> further comprises processing circuitry B-<NUM>, which may have storage and/or processing capabilities. In particular, the processing circuitry B-<NUM> may comprise one or more programmable processors, application-specific integrated circuits, field programmable gate arrays or combinations of these (not shown) adapted to execute instructions. The host computer B-<NUM> further comprises software B-<NUM>, which is stored in or accessible by the host computer B-<NUM> and executable by the processing circuitry B-<NUM>. The software B-<NUM> includes a host application B-<NUM>. The host application B-<NUM> may be operable to provide a service to a remote user, such as a UE B-<NUM> connecting via an OTT connection B-<NUM> terminating at the UE B-<NUM> and the host computer B-<NUM>. In providing the service to the remote user, the host application B-<NUM> may provide user data which is transmitted using the OTT connection B-<NUM>.

The communication system B-<NUM> further includes a base station B-<NUM> provided in a telecommunication system and comprising hardware B-<NUM> enabling it to communicate with the host computer B-<NUM> and with the UE B-<NUM>. The hardware B-<NUM> may include a communication interface B-<NUM> for setting up and maintaining a wired or wireless connection with an interface of a different communication device of the communication system B-<NUM>, as well as a radio interface B-<NUM> for setting up and maintaining at least a wireless connection B-<NUM> with a UE B-<NUM> located in a coverage area (not shown in <FIG>) served by the base station B-<NUM>.

The communication interface B-<NUM> may be configured to facilitate a connection B-<NUM> to the host computer B-<NUM>. The connection B-<NUM> may be direct or it may pass through a core network (not shown in <FIG>) of the telecommunication system and/or through one or more intermediate networks outside the telecommunication system. In the embodiment shown, the hardware B-<NUM> of the base station B-<NUM> further includes processing circuitry B-<NUM>, which may comprise one or more programmable processors, application-specific integrated circuits, field programmable gate arrays or combinations of these (not shown) adapted to execute instructions. The base station B-<NUM> further has software B-<NUM> stored internally or accessible via an external connection.

The communication system B-<NUM> further includes the UE B-<NUM>, which has already been referred to above. Its hardware B-<NUM> may include a radio interface B-<NUM> configured to set up and maintain a wireless connection B-<NUM> with a base station serving a coverage area in which the UE B-<NUM> is currently located. The hardware B-<NUM> of the UE B-<NUM> further includes processing circuitry B-<NUM>, which may comprise one or more programmable processors, application-specific integrated circuits, field programmable gate arrays or combinations of these (not shown) adapted to execute instructions. The UE B-<NUM> further comprises software B-<NUM>, which is stored in or accessible by the UE B-<NUM> and executable by the processing circuitry B-<NUM>.

The software B-<NUM> includes a client application B-<NUM>. The client application B-<NUM> may be operable to provide a service to a human or non-human user via the UE B-<NUM>, with the support of the host computer B-<NUM>. In the host computer B-<NUM>, an executing host application B-<NUM> may communicate with the executing client application B-<NUM> via the OTT connection B-<NUM> terminating at the UE B-<NUM> and the host computer B-<NUM>. In providing the service to the user, the client application B-<NUM> may receive request data from the host application B-<NUM> and provide user data in response to the request data. The OTT connection B-<NUM> may transfer both the request data and the user data. The client application B-<NUM> may interact with the user to generate the user data that it provides.

It is noted that the host computer B-<NUM>, base station B-<NUM> and UE B-<NUM> illustrated in <FIG> may be identical to the host computer A-<NUM>, one of the base stations A-12a, A-12b, A-12c and one of the UEs A-<NUM>, A-<NUM> of <FIG>, respectively.

In <FIG>, the OTT connection B-<NUM> has been drawn abstractly to illustrate the communication between the host computer B-<NUM> and the use equipment B-<NUM> via the base station B-<NUM>, without explicit reference to any intermediary devices and the precise routing of messages via these devices. Network infrastructure may determine the routing, which it may be configured to hide from the UE B-<NUM> or from the service provider operating the host computer B-<NUM>, or both. While the OTT connection B-<NUM> is active, the network infrastructure may further take decisions by which it dynamically changes the routing (e.g., on the basis of load balancing consideration or reconfiguration of the network).

The wireless connection B-<NUM> between the UE B-<NUM> and the base station B-<NUM> is in accordance with the teachings of the embodiments described throughout this disclosure. One or more of the various embodiments improve the performance of OTT services provided to the UE B-<NUM> using the OTT connection B-<NUM>, in which the wireless connection B-<NUM> forms the last segment. More precisely, the teachings of these embodiments can improve one or more of data rate, latency, and/or power consumption associated with one or more devices and/or communications of/performed in communication system B-<NUM>, and thereby can provide benefits for OTT user data communication, such as one or more of reduced user waiting time, relaxed restriction on file size, better responsiveness, and/or extended battery lifetime.

There may further be an optional network functionality for reconfiguring the OTT connection B-<NUM> between the host computer B-<NUM> and UE B-<NUM>, in response to variations in the measurement results. The measurement procedure and/or the network functionality for reconfiguring the OTT connection B-<NUM> may be implemented in the software B-<NUM> of the host computer B-<NUM> or in the software B-<NUM> of the UE B-<NUM>, or both.

In embodiments, sensors (not shown) may be deployed in or in association with communication devices through which the OTT connection B-<NUM> passes; the sensors may participate in the measurement procedure by supplying values of the monitored quantities exemplified above, or supplying values of other physical quantities from which software B-<NUM>, B-<NUM> may compute or estimate the monitored quantities. The reconfiguring of the OTT connection B-<NUM> may include message format, retransmission settings, preferred routing etc.; the reconfiguring need not affect the base station B-<NUM>, and it may be unknown or imperceptible to the base station B-<NUM>.

In certain embodiments, measurements may involve proprietary UE signaling facilitating the host computer's B-<NUM> measurements of throughput, propagation times, latency and the like. The measurements may be implemented in that the software B-<NUM>, B-<NUM> causes messages to be transmitted, in particular empty or 'dummy' messages, using the OTT connection B-<NUM> while it monitors propagation times, errors etc..

Figures <NUM>, <NUM>, <NUM>, and <NUM> (below) are flowcharts illustrating methods implemented in a communication system including a host computer, a base station and a user equipment.

In a first step C-<NUM> of the method, the host computer provides user data.

In an optional substep C-<NUM> of the first step C-<NUM>, the host computer provides the user data by executing a host application. In a second step C-<NUM>, the host computer initiates a transmission carrying the user data to the UE.

In an optional third step C-<NUM>, the base station transmits to the UE the user data which was carried in the transmission that the host computer initiated, in accordance with the teachings of the embodiments described throughout this disclosure. In an optional fourth step C-<NUM>, the UE executes a client application associated with the host application executed by the host computer.

In a first step D-<NUM> of the method, the host computer provides user data.

In a second step D-<NUM>, the host computer initiates a transmission carrying the user data to the UE. In an optional third step D-<NUM>, the UE receives the user data carried in the transmission.

In an optional first step E-<NUM> of the method, the UE receives input data provided by the host computer. Additionally or alternatively, in an optional second step E-<NUM>, the UE provides user data. In an optional substep E-<NUM> of the second step E-<NUM>, the UE provides the user data by executing a client application. In a further optional substep E-<NUM> of the first step E-<NUM>, the UE executes a client application which provides the user data in reaction to the received input data provided by the host computer. Regardless of the specific manner in which the user data was provided, the UE initiates, in an optional third substep E-<NUM>, transmission of the user data to the host computer. In a fourth step E-<NUM> of the method, the host computer receives the user data transmitted from the UE, in accordance with the teachings of the embodiments described throughout this disclosure.

Claim 1:
A method (<NUM>) in a recurrent neural network, RNN (<NUM>) implemented in a data communication system (<NUM>), comprising:
periodically obtaining (<NUM>), by a device (<NUM>) in said RNN, information corresponding to one or more over-the-top, OTT, data sessions (<NUM>) in the data communication system, the information being obtained from a test terminal (102T), one or more of a packet probe (<NUM>),and optionally a network node (<NUM>);
continually training, by the device (<NUM>), the RNN (<NUM>) based on the OTT data sessions captured by the test terminal (102T);
recognizing new OTT encoding types or schemes by the packet probe (<NUM>) and requesting the test terminal (120T) to generate training samples for new traffic types; and
generating (<NUM>), by the device (<NUM>), at least one quality metric associated with the one or more OTT data sessions (<NUM>) based on the periodically obtained information.