Patent ID: 12262097

DETAILED DESCRIPTION

Embodiments of the present disclosure are described herein. It is to be understood, however, that the disclosed embodiments are merely examples and other embodiments can take various and alternative forms. The figures are not necessarily to scale; some features could be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the embodiments. As those of ordinary skill in the art will understand, various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combinations of features illustrated provide representative embodiments for typical applications. Various combinations and modifications of the features consistent with the teachings of this disclosure, however, could be desired for particular applications or implementations.

A unified end-to-end method or system may be used to solve QoE monitoring, optimization, and management problems for the multimedia delivery chain as a whole. The principle is to “begin with the end in mind,” because the QoE of end users determines the ultimate overall performance of a media delivery system. The design and resource allocation in the multimedia distribution system, regardless of whether it is for an individual component at the head-end, media data center, network, access server, user device, or the whole system, should be evaluated, compared, and optimized for their impact on end users' QoE. Highly accurate, efficient, and versatile QoE metrics may be allocated to monitor QoE at the transition points throughout the multimedia delivery network, aggregate the QoE measurements from these points in a common middleware, and regularize the measurements to create a unified assessment of the entire system as well as its individual components. The additional benefits of the present invention include quality degradation identification, failure discovery, localization and recovery, degraded-reference and reduced-reference QoE assessment, latency measurement, and optimal encoding, streaming and network resource allocation, as some possibilities.

FIG.1illustrates an overall framework for solving quality and latency measurement, optimization and management problems for the multimedia delivery chain100. As shown, a media source102provides a media stream that V that passes through multiple stages of operations104before the media stream reaches the end viewer106. Each of the operations104may cause some quality degradation and delay; thus, the user experience of the end viewer106may be affected by all of the operations104. Let Vn-1and Vnbe the input and output media stream of the n-th operation.

Monitoring points108may be used to monitor aspects of the media stream at the various locations V. For instance, Quality-of-delivery (QoD) and quality-of-experience (QoE) measures may be applied to each of the mid-stage media streams V. The QoD measures are concerned about the performance of the delivery process, such as bit error rate, package loss rate, network delay, video freeze, audio silence, errored seconds, black frames, loss of audio-video synchronization, etc. The QoE measures are concerned with the perceptual quality of the media content by examining the media fully decoded at the measurement/monitoring point and evaluating how humans would rate the quality when experiencing the media stream with their perceptual systems.

A good objective QoE measure should accurately predict human perception of the media content and should be fast. The QoE measure should also have other critical properties, such as cross-content type, cross-resolution, cross-viewing device, cross-frame rate, and cross dynamic range consistent scoring. An example of such a QoE measure is the SSIMPLUS index.

Absolute QoE of a video, as used herein, relates to the mapping of an objective QoE measure to the scale of human perceptual QoE, i.e., the average score given by human subjects when expressing their visual QoE when watching the playback of a video content. For example, a score may be defined on a scale of 0-100, which is evenly divided to five quality ranges of bad (0-19), poor (20-39), fair (40-59), good (60-79), and excellent (80-100), respectively. Absolute QoE is in contrast to relative QoE and quality degradation measures, where the quality change before and after a video operation is measured.

In addition to QoD and QoE measures, the monitoring points108may extract other features from the media streams V. Examples of the features may include statistical quantities such as the mean and standard deviation of the signal intensity and spatial and temporal information content measures from the media stream on a per moment basis.

The QoD, QoE measures and the extracted features from multiple monitoring points108(which could be a subset of all possible monitoring points108) are transmitted to and aggregated at a common middleware110, which could locate at a public or private cloud, or at dedicated a data storage and processing server. At the middleware110, the QoD and QoE scores collected from multiple monitoring points108are synchronized, compared, and refined to create correlated and consistent scores. Correlated scoring means that all the QoD or QoE scores throughout the delivery chain100should use the same scale for scoring, and all scores collected should be aligned and refined when necessary to be mapped to the same scale, such that all QoD or QoE scores are comparable throughout the video delivery chain100and distribution network. The features extracted, together with the QoD and QoE measures collected from multiple monitoring points108, are used to perform latency assessment for each point along the media delivery chain100. All the QoD, QoE and latency scores are stored in a database, based on which, reports are generated and may be visualized using a user interface. The reporting may be performed per time span (e.g., per second, per minutes, per hour, per day, per week, per month, per year, etc.) and may also be per geo-location, per media asset, per program, per asset type, etc.

By comparing the correlated QoD, QoE, and latency scores, problems in the media delivery process are identified and localized. When the problems are significant, alerts are generated. By combining the QoD, QoE measures, the extracted features, and the alerts generated, optimization methods may be applied that generate suggestions on each of the operation points108in the delivery chain100for actions that may be performed to improve the delivery process. The alerts and optimization suggestions may be transmitted to each of the operations104, and the operations104may be adjusted correspondingly. The alerts and optimization suggestions may be different for different operations104. For example, for encoders and transcoders, suggestions may be made on the bit rates that should be used for each content and for each of the transcoded derivative video profiles. When multiple media sources (e.g., primary, backup, and disaster recovery) or multiple encoders (e.g., H.264 vs. HEVC) are available for a media channel, suggestions may be made on which options are available and on which source or encoder to select that will generate the best viewer experience. Depending on the stage and location of the alerts being generated, the alerts may be classified into layers, for example, the video assets (a video asset refers to a distinct video content, for example, a movie or other video clip in video-on-demand application, or a TV channel, a TV program, or an ad video clip in live streaming environment) layer, the derivative video profile layer, per viewer stream layer, etc. When significant errors or failures occur at multiple points in the media delivery chain100, an alert may be generated to reflect the cause of the failure, leading to a minimal number of alerts being generated. For example, at the transcoder, when the media quality of all transcoded derivatives together with the input stream are not available, or all have very low QoD or QoE, then a major alert on the input steam is generated, as opposed to many alerts generated for each of the derivative videos. In the content delivery network, alerts may be generated to identify delivery problems or to improve resource allocation, e.g., to avoid certain notes in the network, or to find better paths/routes in the content delivery network for the next step of video delivery.

At any monitoring point108along the multimedia delivery chain100, when the QoE score is lower than a threshold value, or when the latency measure is longer than a threshold value, an alert may be generated to identify a QoE degradation or long-latency problem. When multiple alerts are generated at multiple points along a multimedia delivery chain100, the critical QoE or latency problem may be localized at the monitoring point108where the first alert is generated. When a multimedia delivery system consists of multiple multimedia delivery chains100, each for a media channel, program or service, their correlated QoE, latency measures, and alerts generated at a plurality of monitoring points108may be cumulated. The cumulated data may be grouped and/or divided for the whole enterprise, for each market region, for each media data center, for each service levels, for each type of encoders, for each type of programs, and for each time segment. The overall performance for a given time segment of each group and division may be measured and reported by computing the average or weighted average QoE, average or weighed average latency, and the alerting rate for the time segment. An example of alert template setup is given inFIG.11. Examples of generated alert points are given as red and yellow dots inFIGS.15,19,20,21and22. Examples of classifying the channels/programs/services into critical, warning and stable categories are also shown inFIGS.15,19,20,21and22. Examples of dividing by different media centers are given inFIGS.16and17. An example of comparing a set of encoders is given inFIG.18. An example of reporting the performance for different regions is given inFIG.23. An example of reporting the performance for different media service centers is given inFIGS.24and25.

FIG.2illustrates a practical example of an end-to-end system200for unified QoE monitoring, optimization, and management of video content. In the illustrated example of the video delivery chain100, the encoder204, the transcoder206, the packager208, the origin210, the content delivery network212, and the home viewing devices214such as TV, tablet and cell phones are examples of the operations in the video delivery chain100that may create video quality degradations and latencies. The source video feed may be in the format of many video formats, for example, SDI, transport stream, multicast IP, or mezzanine files from content producers/providers. For home TV, there are often set-top boxes that replay the received video streams to TV, e.g. through HDMI cables. The monitoring points108may be before the set-top box, at the set-top box decoder/player, and post set-top box through the HDMI cables.

An instance of video content may include, as some examples, live video feeds from current events, prerecorded shows or movies, and advertisements or other clips to be inserted into other video feeds. The video content may include just video in some examples, but in many cases the video further includes additional content such as audio, subtitles, and metadata information descriptive of the content and/or format of the video. As shown, the system200includes one or more sources202of instances of video content. In general, when a video distributor receives source video, the distributor passes the video content through a sophisticated video delivery chain such as shown, including a series of content sources202, encoders204, transcoders206, packagers208, origins210, content delivery networks212, and consumer devices214to ultimately present the video content.

More specifically, one or more encoders204may receive the video content from the sources202. The encoders204may be located at a head-end of the system200. The encoders204may include electronic circuits and/or software configured to compress the video content into a format that conforms with one or more standard video compression specifications. Examples of video encoding formats include MPEG-2 Part 2, MPEG-4 Part 2, H.264 (MPEG-4 Part 10), HEVC, Theora, RealVideo RV40, VP9, and AV1. In many cases, the compressed video lacks some information present in the original video, which is referred to as lossy compression. A consequence of this is that decompressed video may have a lower quality than the original, uncompressed video.

One or more transcoders206may receive the encoded video content from the encoders204. The transcoders206may include electronic circuits and/or software configured to re-encode the video content from a source format, resolution, and/or bit depth into an instance of video content with a different format, resolution, and/or bit depth. In many examples, the transcoders206may be used to create, for each received instance of video content, a set of time-aligned video streams, each with a different bitrate and frame size. This set of video streams may be referred to as a ladder or compression ladder. It may be useful to have different versions of the same video streams in the ladder, as downstream users may have different bandwidth, screen size, or other constraints. In some cases, the transcoders206may be integrated into the encoders204, but in other examples the encoders204and transcoders206are separate components.

One or more packagers208may have access to the ladders for each of the instances of video content. The packagers208may include hardware and/or software configured to create segmented video files to be delivered to clients that then stitch the segments together to form a contiguous video stream. The segmented video may include video fragments, as well as a manifest that indicates how to combine the fragments. The packager208may sometimes be integrated into the encoder204and/or transcoder206that first creates the digital encoding of the instance of video content, but often it is a separate component. In one example, the transcoders206and packagers208may be located in a media data center between the head-end and the content delivery network212.

The packagers208may provide the packaged video content to one or more origins210to the content delivery network212. The origins210refer to a location of the content delivery network212to which video content enters the content delivery network212. In some cases, the packagers208serve as origins210to the content delivery network212, which in other cases, the packagers208push the video fragments and manifests into the origins210. The content delivery network212may include a geographically-distributed network of servers and data centers configured to provide the video content from the origins210to destination consumer devices214. The consumer devices214may include, as some examples, set-top boxes connected to televisions or other video screens, tablet computing devices, and/or mobile phones. Notably, these varied devices214may have different viewing condition (including illumination and viewing distance, etc.), spatial resolution (e.g., SD, HD, full-HD, UHD, 4K, etc.), frame rate (15, 24, 30, 60, 120 frames per second, etc.), dynamic range (8 bits, 10 bits, and 12 bits per pixel per color, etc.). The consumer device214may execute a video player to play back the video content received to the devices214from the content delivery network212.

As far as quality assurance is concerned, the user experience measured at the very end of the chain is what matters. However, only measuring QoE at the very end may be insufficient to help localize problems that could occur at any point along the video distribution chain of the system200. Therefore, to ensure the video is faithfully and smoothly delivered to the consumer device214, a quality assurance approach may include inspector components deployed at the consumer device214and also at each of the transition points along the video distribution chain.

The devices of the system (e.g., the encoders204, the transcoders206, the packagers208, the origins210, the content delivery network212, and the consumer devices214) may each be configured to provide information with respect to the QoE of the video content being experienced. In an example, the user experience may be measured using an objective full-reference perceptual video quality-of-experience (QoE) algorithm. The algorithm may perform an accurate, device-adaptive, cross-resolution, cross-content QoE score predictive of what an average human viewer would say about the quality of the video being viewed. In an example, the score may be defined on a scale of 0-200, which is evenly divided to five quality ranges of bad (0-19), poor (20-39), fair (40-59), good (60-79), and excellent (80-200), respectively.

An example QoE algorithm may be the SSIMPLUS metric based on the application of structured similarly (SSIM) determination techniques to the analysis of video content. SSIM is a perceptual metric that quantifies image quality degradation caused by processing such as data compression or by losses in data transmission. SSIM is a full reference metric that utilizes two images, a reference image and a processed image, and determines a perceptual difference between the images. Further aspects of SSIM are discussed in the paper: Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,”IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600-612, April 99004, which is incorporated by reference herein.

However, as compared to SSIM, the QoE score utilized herein further accounts for additional factors in the determination of the quality score, such as resolution of the target device and screen size of the target device. For instance, a video feed may receive a higher score on a smaller device, as the quality impairments to that feed may be less perceptible to the end user. By accounting for these additional aspects in the determination of the quality score, the QoE score utilized herein may allow for production of scores that are scaled to indicate comparable end-user perceived quality across devices.

The system200therefore provides for unified end-to-end QoE monitoring, optimization, and management with reference to the end user's QoE. All the QoE monitoring points may produce instantaneous scoring that reflects the end user's QoE up to the monitoring point in the video delivery chain. The QoE scores described herein are accurate, consistent, and directly comparable, such that the monitoring solutions of the entire video distribution network speaks the same language. Such a unified end-to-end solution lays the groundwork for subsequent operations. First, operation engineers will gain instantaneous awareness about how video QoE degrades along the chain, such that problems can be immediately identified, localized, and resolved. Second, design engineers may closely observe the QoE of the input and output of individual components, and perform better design and optimization, and be confident about the impact of their new design and optimization on the final user QoE. Third, managing executives will have a clear picture about how video quality evolves throughout the video delivery system and over long-time scales. Meanwhile, when longtime large-scale data has been collected, big data analytics can be performed, so as to make intelligent strategic decisions to manage user QoE.

Thus, perceptual end-user QoE may be measured at every transition point of the video delivery chain. Using QoE scoring, the QoE measurement may be made consistent throughout the delivery chain, both with comparable scoring for the same video along the delivery chain and also comparable scoring across different video content. Scores from all points throughout the video delivery chain may be aggregated, synchronized, and further adjusted to improve their consistency. Full-reference (FR), no-reference (NR) and degraded-reference (DR) scoring strategies may also be used at various different stages of the delivery chain.

Moreover, QoE measurement at the client-side (end-user device) may be used to combine presentation quality (measured at server side) and statistics collected by the video player (buffering, rebuffering, significant quality jumps due to profile switching, etc.). Problems may be identified and localized using the end-to-end QoE measurement for failure recovery and significant quality drops. The activation of backup streams and failure recovery streams may, accordingly, be controlled using these results.

Additionally, the objective QoE measures may be used to optimize components in the video delivery. These may include, but are not limited to: (i) optimizing the performance of the encoders204/transcoders206; (ii) finding best encoding profiles (ladder) for the encoders204/transcoders206; (iii) finding the best bitrates for each video service (e.g., TV channel), each program, and each period of time (hour, day, month, etc.); and (iv) optimizing the streaming process for each individual consumer. For adaptive video streaming, this may include choosing the best video profile at each time segment to maximize the overall QoE on a per-client per-view basis.

Regarding intelligence, data may be offered for analysis including: (i) QoE statistics on the performance of the network as a whole; (ii) QoE statistics per-stream, per-service (channel), per-region, per-title (asset), per-encoder, per-resolution, per-user, per-user device type, per-program (for TV), per-hour/day/week/month/year; (iii) relationships between QoE statistics and user engagement statistics; and (iv) resource allocation strategies based on QoE statistics.

The video player executed by the consumer devices214may also be configured to provide information with respect to the playback of video content back to the content delivery network212. In an example, the consumer devices214may provide back information indicative of player and network analytics. This information may include, for instance, indications of dropped frames or packets, player connection speeds, buffer size, etc.

In an example, the video player of the consumer devices214may provide network metrics, as well as the information with respect to the QoE of the video content, from the consumer device214to the packager208. As one possibility, this data flow may be performed back through the content delivery network212, through the origins210, back to the packagers208, although other approaches are possible.

Responsive to receipt of the information from the consumer device214, the packager208may be configured to make real-time adjustments to the version of video content being provided to the consumer device214. By real-time, such adjustment may advantageously be made in the sub-second timeframe, or preferably even faster. In an example, the packager208may determine, based on the information received from the consumer device214, that a different version of the video content may be more applicable to be sent to the consumer device214. More specifically, based on the QoE score received from the consumer device214, the packager208may determine that the consumer device214may still be able to provide an adequate QoE score using a lower-bandwidth version of the video content. Accordingly, the packager208may change the video streaming to use a different transcoded version of the video content from the ladder that is of lower bandwidth, but that still will provide adequate QoE to the consumer. In doing so, the system200may be able to provide the same or substantially the same quality of video to the consumer, while at the same time making the best use of available bandwidth and other resources along the system200. Notably, the ability to regulate bandwidth while accounting for quality is unavailable in systems that lack QoE score information that is received from the consumer device214.

The video player may also be configured to provide other information as well, such as an identifier of the consumer device214or the consumer at the consumer device214that may be used to correlate information about use of the consumer device214while protecting the privacy of the consumer. The video player may also provide behavioral information, such as information about when playback of the video content was initiated or completed, or when in the context the video was paused, fast-forwarded, rewound, or turned off. The video player may also provide screenshots or other information with respect to the video content provided to the consumer via the consumer device214. Such data may be useful in determining why the QoE scoring identified a particular score at the consumer device214.

In an example, the privacy identifiers, behavior data, and screenshots may be provided to a hosted cloud218for later analysis. In an example, one or more customers220or other data analysis services may utilize the hosted data to identify issues with the video content, such as time indexes in the content where users tend to drop off.

FIG.3illustrates a reduced-reference framework300for QoE measurement. A degraded-reference (DR) QoE measure targets at computing the absolute QoE at the output of a video operation104-kbased on not only the output video being evaluated, but also the input video Vk-1to the operation104-kas a reference, whereas the reference video is not of pristine or distortion-free quality, but its quality has been degraded in previous operations. As shown, the degraded-reference (DR) approach may be used to generate correlated QoE scores at two points along the multimedia delivery chain100, where there may be one or multiple operations104between the media source102and the testing point.

An example of such an operation104-kis a video transcoder. A no-reference (NR) QoE measure302may be applied first to obtain an NR QoE measure for the input source. This is regarded as the absolute QoE of the input source304. A full-reference (FR) QoE measure306may take both the source input Vk-1and the test output Vnto create an FR perceptual fidelity measure308for the relative QoE between input and output. A DR QoE measure310is then performed that take the absolute QoE measure of the source input and the relative QoE measure between the input and output, and create an estimate of the absolute QoE score312for the output.

FIG.4illustrates full-reference (FR) and reduced-reference (RR) frameworks400for latency measurement. A RR measure assesses a test video using another video as reference, where the reference video may have pristine or better quality then the test video and may be at an earlier stage of the video delivery chain before the test video. However, the reference video is not fully available in terms of full pixel content, but only certain features, namely RR features, are available when assessing the test video. Such RR features may be statistical features of the reference video content, such as the mean, standard deviation, higher-order moments, quality measure, or statistical model parameters extracted from the reference video. An RR approach can be used for both QoE and latency measurement by comparing the features from the reference and test videos. These FR and RR methods may be used to compute the latency between any two points along the multimedia delivery chain100, as illustrated, where there may be one or multiple operations between the media source and the testing points.

The first method, the FR latency measure402, decodes the video into pixels at both the source input Vk-1and test output Vn, and aligns the two videos along the temporal direction to the frame level. Once the videos are temporally aligned, the temporal offset can then be combined with the video frame rate (in terms of frames per second), to create an estimate of the latency404.

The second method, the RR latency measure406, does not apply temporal alignment to fully decoded video pixels. Instead, feature extractions408,410are applied to both the source input Vk-1and test output Vnvideos, respectively, to create video features412,414, respectively. The video features412,414may include statistical quantities such as the mean and standard deviation of the signal intensity and spatial and temporal information content on a per moment (e.g., per frame or per second) basis. The features412,414are regarded as reduced-reference (RR) features of the videos and are transmitted to a common location (e.g., to the server or in a public or private cloud). The RR latency measure406is then applied by aligning the received features412,414, and the alignment offset is used to estimate the latency416between the source input Vk-1and test output Vn.

The described system200may provide for further applications as well.FIGS.5-10illustrate examples of a live monitor502application.FIG.5illustrates a welcome screen500of the live monitor502. As noted in the welcome screen500, four regions, 92 markets, and 4,635 services.

FIG.6illustrates a breakdown of markets600for a selected region as shown by the live monitor502. As shown, the breakdown is based on a selection of the East region from the welcome screen500.

FIG.7illustrates a breakdown of services700for a selected market as shown by the live monitor502. For instance, the breakdown is based on a selection of the Charleston market. As shown, the selection of information is for a data mode, as opposed to a video mode as shown inFIG.10.

FIG.8illustrates an example breakdown for a service800selected from a market as shown by the live monitor502. As shown, the breakdown is for the CW East HD. Moreover, information can be seen in the breakdown retrieved from various monitoring points108.

FIG.9illustrates an example expansion of information900for a monitoring point108for a selected service as shown by the live monitor502. As shown, further details of the monitoring of the service are provided for the monitoring point “2”. This information includes, for example QoE scores, network information, video parameters, and any alerts.

FIG.10illustrates a breakdown of services1000for a selected market in video mode as shown by the live monitor502. As compared to what is shown inFIG.7, each service is represented by a video feed rather than by data descriptive of the video. This allows for a quick visual inspection, such that the user can understand how the video feeds are being provided.

FIGS.11-25illustrate examples of a multiport live monitor502application. In general, the live monitor502application may utilize the QoE information captured from the monitors of the system200to accurately predict the end viewer's display-adaptive QoE. These prescriptive QoE metrics may result in cost savings (e.g., bandwidth reductions in cases where a lower-bitrate stream would still provide adequate QoE) and provide further data for use in making technology decisions (e.g., which encoders to use). Cross-content video quality measurements may also be performed, and correlated QoE may be used to make true measurements and recommendations. The high performance of the system200ensures scalability for lifetime value (LTV) workflows, and the system200is also adaptive for numerous workflows and monitoring points108across the system200.

The live monitor502may further make use of additional diagnostic journey tools to further strengthen triaging and issue resolution. These tools may include a video freeze on first monitoring point that alerts a user to a component at which video may have frozen. The enhanced video freeze algorithm may more accurately detect video freezes via more accurate slate detection, monochrome detection. The live monitor502may also implement temporal alignment reasoning, and alert-driven tiles on a market page that provide information in a useful format. NOC/Ops workflows may be supported through dashboards, tiles, and alerts. The live monitor502may also include enhanced support for quality on delivery (QoD) only workflow (e.g., when QoE data is unavailable) to broaden the available implementation scenarios.

As another possible application of the system200, a video-on-demand (VOD) monitor may be implemented. The VOD monitor may support AB testing for configuration and purchasing decision for on demand content, as tuning encoders204can drive down costs while maintaining video quality. The VOD monitor may also provide a sandbox environment for encoder204and transcoder206output troubleshooting. The VOD monitor may include both a UX workflow and a RESTful API to automate processes and quality controls. Using the QoE data from the QoE monitors throughout the system200, the VOD monitor may accurately predict the end viewer's display-adaptive QoE.

As a specific optimization process to optimize the QoE or optimize the rate-distortion performance for the best QoE using the lowest bit rate, smart adaptive streaming decisions, or what we call stream smart, may be done in two approaches.

FIG.11illustrates an example of alert template1100setup in a multiport live monitor502application. The alert template1100may include various criteria that may be selected to trigger the alert, e.g., criteria related to video, audio, and/or closed captioning of the streamed content.

FIGS.12A,12B, and12Cillustrate an example1200of QoE assessment presentation, including both statistics and QoE curves, in a multiport live monitor application. As shown, various data from monitoring points108for a channel is displayed in the multiport live monitor application.

FIG.13illustrates an example1300of quality assessment reporting in a multiport live monitor502application. As shown, a QoE score for a stream is shown at several test points over time.

FIG.14illustrates an example1400of quality assessment reporting setup in a multiport live monitor502application. The setup may allow the user to provide information regarding services on which to be reported.

FIG.15illustrates an example1500of market region multiple services quality monitoring in a multimedia delivery network. As shown, the quality monitoring indicates scoring for a city of the network, as well as indications of which channels have alerts. For instance, channel1is indicated as having a critical error, while channels2and3are each indicates as having warnings. The remining channels are indicated as being stable.

FIG.16illustrates an example1600of quality monitoring at multiple media service centers in a multimedia delivery network. For instance, quality monitoring of video, audio, closed captions, and ads is indicated for a particular channel.FIG.17illustrates an additional example1700of quality monitoring at multiple media service centers in a multiport live monitor502application. In the example1700, quality of experience is illustrated in further detail for the video.

FIG.18illustrates an example1800of multiple encoder comparison in a multiport live monitor502application. As shown, information with respect to multiple sources, encoder204outputs, and packager208outputs are shown.

FIG.19illustrates an example1900of service performance classification (critical, warning, and stable) in a multimedia delivery network. The monitoring points108where alerts are generated are labeled as critical or warning, depending on the alert level.FIG.20illustrates another example2000of service performance classification (critical, warning, and stable) in a multimedia delivery network. The monitoring points108where alerts are generated are labeled as critical or warning, depending on the alert level.

FIG.21illustrates an example2100of service performance classification with quality degradation reasoning in a multimedia delivery network. Here also, the monitoring points108where alerts are generated are labeled as critical or warning, depending on the alert level.

FIG.22illustrates another example2200of service performance classification with quality degradation reasoning in a multimedia delivery network. The monitoring points108where alerts are generated are labeled as critical or warning, depending on the alert level. Moreover, the critical alert is expanded (e.g., responsive to input by an operator of the application), to show the video for the channel alerting as critical.

FIG.23illustrates an example of national and region level reporting in a multi-service multimedia delivery network. As shown, a listing of the regions available for monitoring are presented. Responsive to selection of one of the regions, information with respect to that region is displayed.

FIG.24illustrates an example2400of multiple media service center reporting in a multi-service multimedia delivery network. As shown the reporting relates to a specified channel in a region, where historical information about the video, audio, closed captioning, and ads are provided.

FIG.25illustrates another example2500of multiple media service center reporting in a multi-service multimedia delivery network. As compared to the example2400, in the example2500video output at the source, encoder204out, packager208out, and set-top box (e.g., end viewer106) are shown to provide context for the indicated information.

FIG.26illustrates a first approach2600to performing smart adaptive streaming decisions. In this approach2600, the post packager video QoE for each video profile (with different resolution and bit rates) is measured at2602by comparing the video source with the encoded/transcoded video derivatives. Such derivatives may be obtained by directly accessing post encoder204/transcoder206streams, or at a point post the packager208by de-packaging the video and then decoding the video for each profile.

After the QoE measurement2602, the QoE scores are aggregated into per segment/per profile scores by referencing to the post-packager manifest file. These scores are then embedded/attached to the manifest file as shown at2604. This step can be done in different ways, either by following the video streaming standard format (e.g., MPEG-DASH), or by producing a separate file. The modified manifest file after per-segment per-profile QoE data embedding is then written back to the video packages and stored at the origin210or other storage location in the network212. The length of the segment depends on the adaptive streaming method being used between the video server and the viewer device, and is typically in the order of second, e.g., 2 seconds, or 10 seconds.

At the viewer device player214, the QoE measures embedded in the manifest file is decoded, and used to help make smart adaptive streaming decisions2606on a per-segment basis for the video player, which will then request to fetch the best profile for the next segment that will maximize the final QoE or will optimize the rate-distortion performance of the player. Such final QoE is not only impacted by the per-profile per-segment QoE measure after transcoding and packaging, but also by the video freezing and quality/profile switching events, and the interactions between them. The combined end-user QoE measure for each time moment may be stored to provide a historical recording of end user QoE on a per viewer, per viewing session basis.

FIG.27illustrates a second approach2700to performing smart adaptive streaming decisions2606. In the second approach, the post packager video QoE for each video profile (with different resolution and bit rates) is measured by comparing the video source with the encoded/transcoded video derivatives. Such derivatives may be obtained by directly accessing post encoder204/transcoder206streams, or at the post-packager208point by de-packaging the video and then decoding the video for each profile.

Such QoE measurement results are transmitted to a common middleware110, which may be in a public or private cloud or at a video server. The post-packager manifest file is also transmitted to the middleware110. For each of the player on end viewers' devices214, player/viewer status is transmitted to the middleware110on a per-segment basis. The status information may include the manifest file, the video stream ID, the segment ID, the network bandwidth estimation, the buffer condition, the viewing device display type, the device parameters of the viewing session (e.g., the size of the viewing window). Such player/viewer status may be transmitted directly to the middleware110, or be transmitted to the network212, and the network212will relay such information to the middleware110.

In the middleware110, the per-segment per-profile QoE scores are first synchronized with the player/viewer status information received for each viewing device214by comparing the stream and segment ID. Smart adaptive streaming decisions2606on a per-segment basis are then made for the video player214. Such stream smart decisions2606either are transmitted directly back to the player214or are sent to the network212, which relays the decision to the player214. The player214will then request to fetch the best profile for the next segment that will maximize the final QoE or will optimize the rate-distortion performance of the player214. Such final QoE is not only impacted by the per-profile per-segment QoE measure after transcoding and packaging, but also by the video freezing and quality/profile switching events, and the interactions between them. When multiple players214are using the middleware110to make stream smart decisions, optimal resource allocations may be made to adjust the stream smart decision making2606so that the overall average QoE of all viewers is achieved, or the QoE of the viewers are adjusted based on the levels of services the viewers subscribed to. The final QoE measures of all viewers are collected and stored in the database, from which reports are generated and may be visualized using a user interface. The reporting may be performed per time span (e.g., per second, per minutes, per hour, per day, per week, per month, per year, etc.) and may also be per geo-location, per media asset, per program, per asset type, etc.

FIG.28illustrates an example user interface2800of a stream smart video player for unified QoE monitoring, optimization, and management of video content. The user interface2800may allow the user to view aspects of the delivery of video content to the consumer devices214. For instance, the user interface2800may display multiple versions of an instance of media content for visual comparison by a user. As shown, video of a soccer match is shown at 4K resolution as compared to at 1440p.

The user interface2800may allow the user to view various metrics with respect to the display of the different versions of the video content. For instance, as shown bitrates for each of the video feeds are displayed in the lower portion of the user interface2800. The user may also be able to select to display other metrics, such as quality (e.g., SSIMPLUS score), network usage, buffer size, the selected profile per video segment, or other profile information with respect to the video content.

Additionally, the displayed metrics may be provided in terms of various different approaches. For instance, the metrics may be buffer-based, such that if buffer contains a healthy amount of video already downloaded then the video is being provided adequately. Or, the metrics may be bitrate-based, such that if the network212has high bandwidth then a higher bitrate encoding profile may be selected for the next time segment. Or, the metrics may also be provided in terms of a combination of these measures, or by another measure such as to optimize start time.

Using the video player, smart decisions may be able to be made regarding the playout. For instance, the video player may provide for optimization based on a desired QoE score, but also optimized to the lowest bitrate within the ladder. For instance, the video player may choose a lowest bitrate selection for an instance of video content that still meets a minimum QoE score but also that minimizes transmission, storage, and other network costs. As another example, decisions may be made using the data to adjust encoder204or transcoder206settings to improve end-user QoE. For instance, the stream smart application may use machine learning, brute force, or information with respect to the type of content being displayed (e.g., sports event, cartoon, etc.) to suggest alternate settings for encoding of the ladder that may provide for better QoE.

As some additional capabilities, the stream smart application may allow for the offline computation of quality gains and/or bandwidth savings. Moreover, a lab tool version of the stream smart application may allow for the simulation of network212parameters and user experiences to allow for the evaluation of QoE in various experimental network212conditions. As another possibility, the stream smart application may incorporate an AB testing feature to allow a user to observe differences in settings.

As yet another possibility, the functionality of the stream smart application may be implemented in existing players, such as the video player executed by the consumer devices214. Additionally, a pixel-level QoE measurement may be made at the player executed by the consumer devices214. As mentioned above, some QoE measures, such as SSIM and SSIMPLUS, are pixel based and measure perceptual difference between images at the pixel level. This functionality may be implemented at the player level to allow a user of the player214to directly observe which areas of the video content are deemed to show degradation.

FIG.29illustrates an example VOD monitor2900. As shown, the VOD monitor2900may further provide additional capabilities, such as grayscale quality maps2902to provide pixel level graphic visualization of content impairments, expanded format support including HLS and MPEG-DASH, and an enterprise solution variant establishing a cloud solution to address production volumes and pass/fail use cases. In an example, the data of the grayscale quality maps2902may be used as a factor in the QoE score.

As yet another possible application of the system200, an advertisement monitor may be implemented to perform a high-volume ad capture to assess QoE of ads, as well as to provide for instant visual validation of ads delivered by a video wall add-on displaying a wall of the captured ads.

FIGS.30-36illustrate examples of the video wall3004.FIG.30illustrates an example3000of the video wall3004in a view mode. In the view mode, some general information regarding steaming is provided on a pane view3002. The video wall3004is provided with a listing of chosen screens that may be monitored for quality. The example3000also illustrates a view/edit control3006, which may be used to toggle between the view shown in the example3000and the editable view as shown inFIG.31. The example3000also includes a change wall control3008that, when selected allows the user to switch between stored video walls3004that are saved to the system, as well as to create a new video wall3004.

FIG.31illustrates an example3100of the video wall3004in an edit mode. The edit mode may be displayed responsive to toggling of the view/edit control3006. As shown, the pane view3002now allows the user to change which streams are displayed on the video wall3004.

FIG.32illustrates an example3200of the video wall3004in the change wall mode. The change wall mode may be entered responsive to selection of the change wall control3008. As shown, a selection of various video walls3004is available, such as a national wall, a south region wall, an east region wall, a west region wall, and a central region wall. A selection is also available to create a new wall.

FIG.33illustrates an example3300of editing the video wall3004. As shown, the user interface in the example3300allows the user to select or drag services from the pane view3002, which may be added as video tiles in the video wall3004. Once the user has completed customizing the video wall3004, the user may select the save & view control3010to save the video wall2004and revert to the view mode.

FIG.34illustrates an example3400of the video wall2004showing a selector3012for cycling channels. For instance, using the selector3012, the user may set a channel view on the video wall3004to loop among a set of channels, rather than using screen real-estate for each channel.

The advertisement monitor may further provide additional capabilities, such as an enhanced ad asset acquisition and identification approach; an API and IR; support for locking, cycling and scheduling of ad capture and monitoring; unique settings for global, ad zone, alerts, and configuration pages; updated market, service, alerts, and report pages to support unique ad monitoring requirements; support for video walls; a real-time dashboard; and security and privacy support to ensure the safety of customer information.

Moreover, additional reporting insights may be incorporated as an additional application of the system200. For instance, a reporting insights platform may include additional aspects incorporated into the live monitor application. For instance, the reporting insights may provide support for a diagnostic journey through forensics analysis and deeper diving into audio, video, alerts, and closed captioning data; a reporting foundation for compliance reporting; and a foundation to support service level agreement (SLA)/service level operator (SLO) level analyses. For instance, issues with malfunctioning components may be identified, alerted on, and reported by time, where the reporting may relate to impacted services, service-level alerts, or stream-level alerts. An example may be an issue may be identified as being pre-encoder204and therefore an issue with the source, while another issue may be determined to be the encoder204according to monitoring of the encoder out vs the source out to the encoder204.

As yet a further application of the system200, a viewer QoE aware, content adaptive, per-title or per-asset encoding optimization, or “encoding smart” application may be utilized to allow system operators to provide for optimization for a specific title (or for a genre or type of content with similar characteristics). The application may include an engine that (for each title or asset) combines target quality, viewing conditions, and business rules to: (i) recommend a full encoding ladder; (ii) recommend an optimized bitrate for each ABR profile; (iii) suggest modifying the number of profiles; (iv) run in “Bitrate Saving” and “Quality Improvement” modes. The per-title optimization or encoding smart application may generate testing reports after analysis of the title to show the savings and improvements against given video set for the given optimization mode. The application may also provide a recommended ladder for the tested title/asset. To accomplish these aspects, the per-title optimization or encoding smart application may learn encoder204behavior for various settings and content types to improve the recommendation accuracy. Optimizations may be content aware as well.

FIG.35illustrates an example process3500for generating correlated quality-of-experience (QoE) and latency measures at a plurality of monitoring points108along a multimedia delivery chain including multiple video operations. In an example, the process3500may be performed by the unified end-to-end quality and latency monitoring, optimization and management system as described in detail herein.

At operation3502, each of the plurality of monitoring points108computes an absolute QoE measure defined on a human perceptual quality scale for media content. For instance, the absolute QoE score should relate to an average score given by human subjects when expressing their visual QoE when watching the playback of the media content.

At operation3504, each of the plurality of monitoring points108performs one or more of content extraction or feature extraction on the media content. Examples of the features to be extracted may include statistical quantities such as the mean and standard deviation of the signal intensity and spatial and temporal information content measures from the media stream on a per moment basis.

At operation3506, each of the plurality of monitoring points108transmits, to the middleware110, the respective QoE measure and results of the one or more of content extraction or feature extraction. Accordingly, the middleware110aggregates the QoE and other extracted results.

At operation3508, the middleware110computes and updates an absolute QoE measure for each of the plurality of monitoring points. At the middleware110, the QoD and QoE scores collected from multiple monitoring points108are synchronized, compared, and refined to create correlated and consistent scores. Correlated scoring means that all the QoD or QoE scores throughout the delivery chain100should use the same scale for scoring, and all scores collected should be aligned and refined when necessary to be mapped to the same scale, such that all QoD or QoE scores are comparable throughout the video delivery chain100and distribution network.

At operation3510, the middleware110computes and updates latencies between multiple monitoring points108using the results from each of the plurality of monitoring points108. Thus, the features extracted together with the QoD and QoE measures collected from multiple monitoring points108, may be used to perform latency assessment for each point along the media delivery chain100. In one example, the latency may be computed by decoding the media content into frames at both the source input point and the test output point; aligning the media content at the source input point and the media content at the test output point along a temporal direction at a frame level; identifying a temporal offset in frames between the media content at the source input point and the media content at the test output point; and accounting for a frame rate of the media content to compute the latency. In another example, the latency may be computed by applying one or more feature extractions of reduced-reference (RR) features to the media content at the source input point and to the media content at the test output; aligning the RR features to identify an alignment offset between the media content at the source input point and to the media content at the test output; and determining the latency according to the alignment offset.

By comparing the correlated QoD, QoE, and latency scores, problems in the media delivery process are identified and localized. When the problems are significant, alerts are generated. By combining the QoD, QoE measures, the extracted features, and the alerts generated, optimization methods may be applied that generate suggestions on each of the operation points108in the delivery chain100for actions that may be performed to improve the delivery process. After operation3510, the process3500ends.

FIG.36illustrates an example process3600for optimizing streaming over a multimedia delivery chain100for use by a video player214. In an example, as with the process3500, the process3600may be performed by the unified end-to-end quality and latency monitoring, optimization and management system as described in detail herein.

At operation3602, the multimedia delivery chain100measures post-packager video QoE measures for a plurality of video profiles by comparing a video source with a plurality of encoded/transcoded video derivatives of the video source, each derivative having a different resolution and/or framerate. In one or more examples, the plurality of encoded/transcoded video derivatives of the video source may be obtained by accessing post encoder/transcoder streams of the video source. In one or more examples, the plurality of encoded/transcoded video derivatives of the video source may be obtained at a post-packager point by de-packaging and decoding video for each of the plurality of video profiles.

At operation3604, the multimedia delivery chain100aggregates the QoE measures into per segment scores according to a post-packager manifest file defining segments of the video source. At operation3606, the multimedia delivery chain100embeds the scores into the manifest file. At operation3608, the multimedia delivery chain100sends the manifest file including the QoE measures to a video player. In one or more examples, the manifest file is a Moving Picture Experts Group-Dynamic Adaptive Streaming over Hypertext Transfer Protocol (MPEG-DASH) media presentation description. In one or more examples, the segments are of a length on the order of seconds.

At operation3610, the multimedia delivery chain100sends a next video segment to the video player, responsive to a request from the video player for one of the plurality of video profiles chosen to one or more of maximize QoE at the video player or optimize rate-distortion performance of the video player. In one or more examples, QoE measures from the video player are collected; and a report is displayed of the QoE measures of the video player as collected, the report indicating the QoE measures according to one or more of time span, geo-location, media asset, program, or asset type. After operation3610, the process3600ends.

FIG.37illustrates an example process3700for optimizing streaming over the multimedia delivery chain100for use by the common middleware110. In an example, as with the processes3500and3600, the process3700may be performed by the unified end-to-end quality and latency monitoring, optimization and management system as described in detail herein.

At operation3702, the common middleware110receives per-segment per-profile QoE measures. The common middleware110receives the information being in communication with a plurality of viewer devices214. The QoE measures are measured for a plurality of video profiles by comparing a video source with a plurality of encoded/transcoded video derivatives of the video source, wherein each derivative has a different resolution and/or framerate.

At operation3704, the common middleware110receives a post-packager manifest file. At operation3706, the common middleware110receives from the plurality of viewer devices214, viewer status information on a per-segment basis. In one or more examples, the viewer status information includes one or more of: a manifest file, a video stream ID, a segment ID, a network bandwidth estimation, a buffer condition, a viewing device display type, or physical device parameters of the viewer device.

At operation3708, the common middleware110synchronizes the per-segment per-profile QoE measures with the viewer status information for each of the plurality of viewer devices214. At operation3710, the common middleware110makes one or more adaptive streaming decisions for the plurality of viewer devices214on a per-segment basis.

At operation3712, the common middleware110sends streaming update messages to one or more of the plurality of viewer devices214according to the adaptive streaming decisions, to cause one or more of the plurality of viewer devices214to fetch a best profile for a next segment to maximize QoE at the respective viewer devices214or to optimize rate-distortion performance of the respective viewer devices214. In one or more examples, the common middleware110optimizes resource allocation across the plurality of viewer devices to improve overall average QoE of the plurality of viewer devices. In one or more examples, the common middleware110optimizes resource allocation across the plurality of viewer devices based on levels of services to which the plurality of viewer devices are subscribed. In one or more examples, QoE measures from the video player are collected; and a report is displayed of the QoE measures of the video player as collected, the report indicating the QoE measures according to one or more of time span, geo-location, media asset, program, or asset type. After operation3712, the process3700ends.

The processes, methods, or algorithms disclosed herein can be deliverable to/implemented by a processing device, controller, or computer, which can include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, the processes, methods, or algorithms can be stored as data and instructions executable by a controller or computer in many forms including, but not limited to, information permanently stored on non-writable storage media such as ROM devices and information alterably stored on writeable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. The processes, methods, or algorithms can also be implemented in a software executable object. Alternatively, the processes, methods, or algorithms can be embodied in whole or in part using suitable hardware components, such as Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.

While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in the specification are words of description rather than limitation, and it is understood that various changes can be made without departing from the spirit and scope of the disclosure. As previously described, the features of various embodiments can be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments could have been described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those of ordinary skill in the art recognize that one or more features or characteristics can be compromised to achieve desired overall system attributes, which depend on the specific application and implementation. These attributes can include, but are not limited to cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. As such, to the extent any embodiments are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and can be desirable for particular applications.