Patent Description:
Over-The-Top (OTT) video monitoring is required to detect and diagnose the root cause of video streaming problems as soon as they begin and identify their origin-whether inside or outside of networks. This is critical as video exceeds <NUM>% of today's network traffic and will rise to over <NUM>% in <NUM> networks by <NUM>. OTT video streaming issues are a primary source of network-related customer churn, a multimillion-dollar problem for network operators (also can be referred to interchangeably as service providers, wireless providers, etc.). Of course, customer churn is undesirable and network performance is a key differentiator.

Consumers blame their video QoE issues primarily on service providers, even though approximately half of all video quality issues originate outside of the service providers' networks and their control. Specifically, Applicant has performed a survey in <NUM> that showed about half of video quality issues are due to issues in a CDN network or a customer network (e.g., Wi-Fi), not in the service provider. OTT QoE is difficult to monitor in real-time or to correlate with an easier measurement of Quality of Service (QoS). Further, it can take significant time and personnel to troubleshoot and resolve OTT quality issues. As a result, service providers have high volumes of churn and are unable to effectively monetize OTT video services and leverage QoE as an essential differentiator (and blame shifter). In turn, revenue losses and higher customer acquisition costs increase as the provider's reputation suffers-even if the issues stem from a customer device, a poor home Wi-Fi network, or a content provider's data center issue.

Mobile network operators, internet service providers, and content delivery providers need to detect, segment, classify and diagnose the root cause of video streaming problems as soon as they begin. It is important for them to pinpoint whether problems like freezing, buffering, or lagging come from their network, the video platform, the user's device, or a video provider such as YouTube™ or Netflix™.

Legacy video monitoring solutions are generally expensive, compute-intensive, and tailored to content delivery applications. The industry needs real-time video monitoring that helps service providers overcome network issues, where the majority of issues originate. Many tools are currently available to obtain video streaming QoE test results, such as, for example, SamKnows, Apache JMeter™, Flood. io, UbikLoadPack, PureLoad, Wowza, StreamTest which all provide load testing services. For example, SamKnows offers several QoE tests, with different emphasis including gaming, social media, video conferencing, video streaming, CDN measurements and web browsing. Their video streaming QoE test includes data from Netflix, YouTube, iPlayer, Hulu, and Multicast IPTV. SamKnows' measurements for video streaming consider the bit rate that is reliably streamed, the startup delay (the time taken to download two seconds of video), the Transmission Control Protocol (TCP) connection time, the number of stalls and their duration, and the downstream throughput that is achieved.

Applicant has also proposed a video streaming QoE test. In particular, an "Over-The-Top Media Service Testing And QoE Issues Isolation" method and system are described in <CIT>, and a "VIDEO STREAMING OVER HAS TESTING WITH TRAFFIC PATTERN MODEL" method and system are described in <CIT>. These methods reliably measure the OTT Video QoE of a selected quality (resolution, frame rate, etc.). These methods select a reference or source video Uniform Resource Locator (URL) to perform the QoE assessment. In some embodiments, videos hosted by YouTube are used to reflect the real user experience (same delivery path, caching, etc.). However, YouTube automatically detects requests coming from network computers and the YouTube Terms of service prevent repeated requests made to YouTube's Application Programming Interface (API). It is therefore not possible to perform a significant number of requests to YouTube's API without breaking usage policies and being blocked. As described herein, YouTube is an example content provider (also referred to as video provider, OTT provider, etc.). Of course, this same problem, i.e., significant repeated requests, exists with other content providers (e.g., Netflix, Hulu, Disney+, Tik Tok, etc.).

Document <CIT> describes fault isolation in the OTT content broadband networks. Document <CIT> describes methods and systems for controlling quality of a media session.

There is a need to obtain reliable QoE test results without querying the video source (i.e., content provider) directly.

The present disclosure includes systems and methods to adjust the QoE measurement obtained from a testing Content Delivery Network (CDN) entity to reflect accurately the QoE that would have been obtained from a reference video source from a content provider, for example, YouTube.

The invention is defined by a method according to claim <NUM>, a processing device according to claim <NUM> and a computer code comprising instructions according to claim <NUM>.

The present disclosure includes systems and methods to adjust the QoE measurement obtained from a testing Content Delivery Network (CDN) entity to reflect accurately the QoE that would have been obtained from another CDN or content provider, for example, YouTube. Again, YouTube as referenced herein is an example content provider. Those skilled in the art will appreciate other content providers are also contemplated herewith. That is, the present disclosure includes QoE measurements from a first source (e.g., a CDN) and extrapolating that result to reflect what it would have been from a second source (e.g., the content provider, another CDN, etc.). A use case of this approach is to provide QoE measurements in a manner that does not overwhelm a content provider and create issues (service violation, blocking, etc.).

YouTube and other content providers deliver video services and other content with Content Delivery Network (CDN) entities to improve performance and locally manage requests (caching, firewall, etc.). As is known in the art, a CDN is a distributed network of proxies and associated data centers, to provide better availability and performance by distributing content spatially relative to end users. It would therefore be possible to use CDNs, such as Cloudflare, Amazon CloudFront, etc., to stream OTT video instead of streaming directly from YouTube or another content provider. Unfortunately, the measured QoE would then not precisely reflect the YouTube video delivery performance. The QoE results might be slightly different since the video are not streamed from the same origin.

<FIG> is a network diagram of an example network <NUM> that includes a service provider network <NUM> and a user device <NUM> connected thereto. Also, there are two content sources <NUM>, <NUM> connected to the service provider network <NUM>. The content sources <NUM>, <NUM> can be CDN entities hosting content, video providers, content providers, OTT providers, etc. For example, the content source <NUM> can be YouTube while the content source <NUM> can be a CDN entity hosting content from YouTube. The network <NUM> is presented for illustration purposes and those skilled in the art will recognize various embodiments are contemplated. Also, various components are omitted for simplicity. There can be many more content sources <NUM>, <NUM>. Also, the user device <NUM> can be located in another network (not shown) such as home Wi-Fi or the like. Again, as mentioned above, QoE issues can be either in the service provider network <NUM> or external to the service provider network <NUM>, such as at the content sources <NUM>, <NUM> and associated networks or at the user device <NUM> and associated network.

<FIG> is a flowchart of a process <NUM> to adjust the QoE measurement obtained from a first CDN entity to reflect accurately the QoE that would have been obtained from another CDN or content provider. With reference to <FIG> as well, the process <NUM> includes a first QoE measurement <NUM> between the user device <NUM> and the content source <NUM> (step <NUM>), a determination of a QoE adjustment factor <NUM> between the content source <NUM> and the content source <NUM> (step <NUM>), and a determination of a second QoE measurement <NUM> based on the first QoE measurement <NUM> and the QoE adjustment factor <NUM>. The QoE measurements <NUM>, <NUM> can utilize any approach including the aforementioned OTT video QoE approaches. The QoE adjustment factor <NUM> is utilized to allow a QoE measurement from the user device <NUM> to the content source <NUM> (e.g., CDN) to accurately substitute for a QoE measurement from the user device <NUM> to the content source <NUM> (e.g., content provider such as YouTube).

One use case for this approach is to avoid traffic surges to content providers for QoE testing. The solution to reflect accurately the QoE from a reference CDN such as a content provider (e.g., YouTube) when testing with another CDN is to learn the performance difference between the reference CDN, and the testing CDN as a function of time and location, and then adjust the measured QoE accordingly. Of course, the approach described herein can be used to determine a QoE measurement for any other use case.

To achieve this accurate substitution, the present disclosure includes the QoE adjustment factor <NUM> including a determination of this value and ongoing monitoring to assure its accuracy. <FIG> is a flowchart of a QoE adjustment factor <NUM> process <NUM> that includes a learning process <NUM>, an adjustment process <NUM>, and a monitoring process <NUM>. The process <NUM> can be implemented in the network <NUM>, such as by various QoE measurement between entities and region. As described herein, a region can be a portion or subset of the service provider network <NUM>. Also, as described herein, a reference CDN is used to refer to a content provider, CDN, video provider, etc. The testing CDN is used to refer to a CDN that an actual QoE measurement is performed to, such as with a user device <NUM>. The reference CDN is the location where the actual QoE measurement is extrapolated to using the QoE adjustment factor <NUM>.

<FIG> is a flowchart of details of the learning process <NUM>. The learning process <NUM> takes periodic QoE measurements from both the reference CDN and the testing CDN. The number of measurements is set low so as not to be blocked by the reference CDN (e.g., YouTube). The QoE measurements are accompanied with timestamp and location information. The objective of the learning process <NUM> is to develop the QoE adjustment factor <NUM>. The QoE adjustment factor <NUM> includes different values (a delta) for different time intervals (e.g., every hour or any other value). The main steps of the learning process are the following for each region associated to the reference CDN and testing CDN:.

Note, in another embodiment, it can be possible to measure QoE between the content sources <NUM>, <NUM>.

This approach takes into account the intrinsic periodicity of <NUM>-day that typically characterize network behavior. As traffic increases rapidly in the morning, network performance measures deteriorate according to the same trend. Then, at the end of the day, when traffic drops significantly, performance measures return to their ideal values. The learning process should last at least one week, ideally much more to properly train the model. Also, additional parameters could be added to improve the model prediction such as holidays, special events, etc..

The output of the learning process <NUM> is a time-series which includes a QoE delta for each time interval. Then, the adjustment process <NUM> uses the predictive model of the corresponding region to adjust the measured QoE from the testing CDN, to reflect an estimate of the measured QoE from the reference CDN.

<FIG> is a flowchart of details of the adjustment process <NUM>. The main steps of the adjustment process <NUM> are the following for each region associated to the reference and testing CDN, and for each QoE measured with a testing CDN:.

And the QoE measurement adjusted corresponds to a QoE measurement to the reference CDN.

<FIG> is a flowchart of details of the monitoring process <NUM>. The monitoring process <NUM> ensures than the predictive model remains accurate over time while the network <NUM> behavior might change. The main steps of the monitoring process <NUM> are the following for each region associated to the reference and testing CDN:.

The main technical advantages of the various processes are offering a non-linear adjustment of QoE measurements better reflecting real network performance. It also automatically adjusts itself over time as network behavior evolve. In an embodiment, the proposed implementation leverages a neural network such as a multivariate Long-Short-Term-Memory (LSTM) with auto-encoder. A transformer or other supervised learning techniques could also be used.

A simulation of the learning and adjustment processes <NUM>, <NUM> is described as follows.

Synthetic reference CDN and testing CDN QoE measurements were generated with a time interval of <NUM> hour for a period of <NUM> year. Each series of measurements has a distribution of random QoE measurement values according to an Absolute Category Rating (ACR) score scale (QoE reference: mean = <NUM>, std = <NUM>; QoE reference: mean = <NUM>, std = <NUM>). Also, a daily variation of +/- <NUM> points on the ACR score scale is introduced on each series of measurements following a sine wave. The resulting QoE measurement time series are shown in <FIG>. The QoE score distribution of QoE reference and QoE testing measurements is shown in <FIG>.

The learning process steps <NUM>, <NUM><NUM> created a QoE delta time series with timestamp and QoE delta (Table provided in <FIG>). The time of day value and day of the week value have been one-hot encoded. This becomes the inputs to the multivariate Long-Short-Term-Memory (LSTM) with auto-encoder at step <NUM> (e.g., first <NUM>% is used to train, last <NUM>% to test).

The adjustment process steps <NUM>, <NUM> use the predictive model to get the QoE adjustment factor corresponding to the time of day, the day of the week. The table of <FIG> shows the QoE adjustment factor obtained from the predictive model for each timestamp. <FIG> shows the difference between the predicted QoE adjustment factor and the actual difference between the QoE reference and QoE testing (QoE delta).

Step <NUM> applies the equation: QoE testing adjusted = QoE testing + QoE adjustment factor, as shown in <FIG>.

The QoE score distribution of QoE reference, QoE testing and QoE testing adjusted measurements is shown in <FIG>. Note that this simulation demonstrates the functionality of, not the performance of the method and system. Using synthetic QoE generated from random processes degrades the performance of the machine-learning system. By construction, a random variable cannot be learned.

<FIG> is a graphical representation of the systems which interact together to provide mobile network operators, internet service providers and content delivery providers with reliable data and information about their networks. This component is accessible via a web browser, for example through a JavaScript™ script running on a webpage. The webpage can be accessed by a user from any type of internet-connected device, including a personal computer running Windows or Mac, any Android, Apple, Google, or other devices, including phones, tablets, watches, etc. In addition to these video monitoring tests, the operators and providers can receive data from physical and transport and datacom test and measurement devices as well as from other service assurance monitoring devices. Additionally, all of these data sources can communicate with a data aggregator and analytics module which finds correlations and issues which can then be reported in a more succinct and action-related compilation to the operators and providers.

<FIG> is a graphical representation of the components of a user-accessible webpage for testing a video streaming service using the method and system presented herein. On the lefthand side, the user selects a video to watch and with which the test will be conducted. A resolution for the video (SD, HD, <NUM>, for example) is selected. The play button triggers the test. In this example, the test is calibrated for YouTube, meaning that it is correlated to closely match the real YouTube viewing experience without impacting on the user's privacy and without breaking YouTube's terms of reference.

On the right-hand side, the live test results are shown. A graph of the performance over the duration of the test is shown. On the Y-axis, stars are lit up when the score level reaches a sufficient level. An overall video test score is shown below the graph, using a <NUM>-star rating. Results are further expressed in terms of Start Delay, Waiting Time and Playback Stalls using sliding bars and a numerical value.

The embodiments described above are intended to be exemplary only. The scope of the invention is therefore intended to be limited solely by the appended claims.

<FIG> is a flowchart of a process <NUM> to adjust the QoE measurement obtained from a testing Content Delivery Network (CDN) entity to reflect accurately the QoE that would have been obtained from another entity. The process <NUM> can be realized as a method having steps, via a processing system including at least one processor and memory with instructions that, when executed, cause the at least one processor to implement the steps, and a non-transitory computer-readable medium having instructions stored thereon for programming at least one processor to perform the steps.

The process <NUM> includes obtaining (step <NUM>) a first Quality of Experience, QoE, measurement <NUM> from a first content source <NUM>; determining (step <NUM>) a QoE adjustment factor <NUM> based on a predictive model; and adjusting (step <NUM>) the first QoE measurement <NUM> using the QoE adjustment factor <NUM> to obtain a second QoE measurement <NUM>, the second QoE measurement <NUM> reflects a QoE estimate from a second content source <NUM>.

The first content source <NUM> can a Content Delivery Network, CDN, entity, and the second content source <NUM> can be an Over-the-Top, OTT, content provider. The QoE adjustment factor includes a delta value for a corresponding time interval and day, such that the delta value is applied to the first QoE measurement <NUM> to estimate the second QoE measurement <NUM>.

The predictive model is determined based on a learning process <NUM> to obtain delta values for time intervals and days, and, optionally, a monitoring process <NUM>, that adjusts the delta values over time. The learning process <NUM> can take periodic QoE measurements from both the first content source <NUM> and the second content source <NUM> including timestamp and location information. The learning process <NUM> can be performed for a plurality of regions in a network, both from the first content source <NUM> and from the second content source <NUM>.

The learning process <NUM> can include steps of obtaining QoE measurements from the first content source <NUM> and from the second content source <NUM> at various time intervals and days; learning a performance difference between the first content source <NUM> and the second content source <NUM> as a function of time and location; and adjusting the predictive model based on the performance difference.

The learning process <NUM> can include steps of obtaining <NUM> QoE measurements from the first content source <NUM> and from the second content source <NUM> at various time intervals and days; determining <NUM> differences between the QoE measurements; and creating <NUM> a time series with the differences. The learning process <NUM> can include steps of taking QoE measurements per time interval from both from the first content source <NUM> and from the second content source <NUM>; calculating a difference between QoE measurements for each time internal: QoE delta = QoE reference - QoE testing; creating a QoE delta time series with timestamp and QoE delta; and building a predictive model from the QoE delta time series with parameters including a time value and QoE delta value. The time value can be at least one of a time of day value and a day of the week value.

The monitoring process <NUM> ensures than the predictive model remains accurate over time while network behavior changes. The monitoring process <NUM> can include steps of periodically repeating portions of the learning process and comparing values in the predictive model and updating the predictive model based thereon. The QoE adjustment factor <NUM> can be based on a region of a plurality of regions.

<FIG> is a block diagram of a processing device <NUM>, which may be used to implement any of the processes and functionality described herein. The processing device <NUM> may be a digital computer that, in terms of hardware architecture, generally includes a processor <NUM>, input/output (I/O) interfaces <NUM>, a network interface <NUM>, a data store <NUM>, and memory <NUM>. It should be appreciated by those of ordinary skill in the art that <FIG> depicts the processing device <NUM> in an oversimplified manner, and a practical embodiment may include additional components and suitably configured processing logic to support known or conventional operating features that are not described in detail herein. The components (<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>) are communicatively coupled via a local interface <NUM>. The local interface <NUM> may be, for example, but not limited to, one or more buses or other wired or wireless connections, as is known in the art. The local interface <NUM> may have additional elements, which are omitted for simplicity, such as controllers, buffers (caches), drivers, repeaters, and receivers, among many others, to enable communications. Further, the local interface <NUM> may include address, control, and/or data connections to enable appropriate communications among the aforementioned components.

The processor <NUM> is a hardware device for executing software instructions. The processor <NUM> may be any custom made or commercially available processor, a Central Processing Unit (CPU), an auxiliary processor among several processors associated with the processing device <NUM>, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions. When the processing device <NUM> is in operation, the processor <NUM> is configured to execute software stored within the memory <NUM>, to communicate data to and from the memory <NUM>, and to generally control operations of the processing device <NUM> pursuant to the software instructions. The I/O interfaces <NUM> may be used to receive user input from and/or for providing system output to one or more devices or components.

The network interface <NUM> may be used to enable the processing device <NUM> to communicate on a network. The network interface <NUM> may include, for example, an Ethernet card or adapter or a Wireless Local Area Network (WLAN) card or adapter. The network interface <NUM> may include address, control, and/or data connections to enable appropriate communications on the network. A data store <NUM> may be used to store data. The data store <NUM> may include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, and the like)), nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, and the like), and combinations thereof.

Moreover, the data store <NUM> may incorporate electronic, magnetic, optical, and/or other types of storage media. In one example, the data store <NUM> may be located internal to the processing device <NUM>, such as, for example, an internal hard drive connected to the local interface <NUM> in the processing device <NUM>. Additionally, in another embodiment, the data store <NUM> may be located external to the processing device <NUM> such as, for example, an external hard drive connected to the I/O interfaces <NUM> (e.g., SCSI or USB connection). In a further embodiment, the data store <NUM> may be connected to the processing device <NUM> through a network, such as, for example, a network-attached file server.

The memory <NUM> may include any of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)), nonvolatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.), and combinations thereof. Note that the memory <NUM> may have a distributed architecture, where various components are situated remotely from one another but can be accessed by the processor <NUM>. The software in memory <NUM> may include one or more software programs, each of which includes an ordered listing of executable instructions for implementing logical functions. The software in the memory <NUM> includes a suitable Operating System (O/S) <NUM> and one or more programs <NUM>. The operating system <NUM> essentially controls the execution of other computer programs, such as the one or more programs <NUM>, and provides scheduling, input-output control, file and data management, memory management, and communication control and related services. The one or more programs <NUM> may be configured to implement the various processes, algorithms, methods, techniques, etc. described herein.

It will be appreciated that some embodiments described herein may include or utilize one or more generic or specialized processors ("one or more processors") such as microprocessors; Central Processing Units (CPUs); Digital Signal Processors (DSPs): customized processors such as Network Processors (NPs) or Network Processing Units (NPUs), Graphics Processing Units (GPUs), or the like; Field-Programmable Gate Arrays (FPGAs); and the like along with unique stored program instructions (including both software and firmware) for control thereof to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of the methods and/or systems described herein. Alternatively, some or all functions may be implemented by a state machine that has no stored program instructions, or in one or more Application-Specific Integrated Circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic or circuitry. Of course, a combination of the aforementioned approaches may be used. For some of the embodiments described herein, a corresponding device in hardware and optionally with software, firmware, and a combination thereof can be referred to as "circuitry configured to," "logic configured to," etc. perform a set of operations, steps, methods, processes, algorithms, functions, techniques, etc. on digital and/or analog signals as described herein for the various embodiments.

Claim 1:
A method (<NUM>) implemented by a processing device (<NUM>) having one or more processors (<NUM>), the method (<NUM>) comprising steps of:
obtaining (<NUM>) a first Quality of Experience, QoE, measurement (<NUM>) from a first content source (<NUM>);
determining (<NUM>) a QoE adjustment factor (<NUM>) based on a predictive model, based on a learning process (<NUM>) to obtain delta values for time intervals and days, comprising
obtaining (<NUM>) QoE measurements from the first content source (<NUM>) and from the second content source (<NUM>) at various time intervals and days;
determining (<NUM>) differences between the QoE measurements;
creating (<NUM>) a time series with the differences, and
building (<NUM>) said predictive model from said time series,
and
adjusting (<NUM>) the first QoE measurement (<NUM>) using the QoE adjustment factor (<NUM>) to obtain a second QoE measurement (<NUM>), the adjusting (<NUM>) comprising using the predictive model corresponding to a current time and day to get corresponding QoE adjustment factor, and adding the QoE adjustment factor to the first QoE measurement,
the second QoE measurement (<NUM>) reflecting a QoE estimate from a second content source (<NUM>) different from the first content source (<NUM>).