Abstract:
Systems and methods are disclosed for testing video quality by generating a stress tracker test pattern with one or more moving zone plates and one or more stamps; determining compression quality scores for encoder resources spent at predetermined levels of compression (stress); and analyzing the test pattern and generating a Compression Stress Response profile.

Description:
BACKGROUND 
       [0001]    The present invention relates to a video content processing and content delivery systems. 
         [0002]    Many applications require quality evaluation of video images. Such evaluations can be subjective or objective. Subjective quality evaluation techniques for video images are fully specified in the ITU-T Recommendation BT.500. The Recommendation provides a methodology for numerical indication of the perceived quality from the users&#39; perspective of received media after compression and/or transmission. 
         [0003]    The score is typically expressed as a single number in the range 1 to 5, where 1 is lowest perceived quality, and 5 is the highest perceived. 
         [0004]    Currently there are two main types of objective video degradation measurement processes: 
         [0005]    1. Full reference methods (FR), where the whole original video signal is available 
         [0006]    2. No-reference methods (NR), where the original video is not available at all 
         [0007]    Devices and processes of both types can be used in off-line systems (file-based environment) as well as in on-line systems (live video transmission). 
         [0008]    Video content re-purposing and delivery system Quality Control (QC) should be fully automatic because checking thousands of channels and hundreds of formats semi-automatically is not an economically viable option. 
         [0009]    The most widely used FR video quality metric during the last 20 years is Peak Signal-to-Noise Ratio (PSNR). PSNR is used approximately in 99% of scientific papers, but only in 20% of marketing materials. 
         [0010]    The validity of PSNR metric is limited and often disputed. This also applies to all PSNR derivatives, such as Structural similarity (SSim) and many others. 
         [0011]    Significant drawback of all PSNR-based tools is that they require perfect spatial, temporal and color space alignment of two pictures A and B used for comparison:
       A=Original picture, presumed to be of very good (pristine) quality,   B=Output picture, typically distorted by video processor of some sort Common examples of the video processor under test are:   Video scalers and format converters, including color space converters   Compression codecs, such as MPEG2, H264, etc.       
 
         [0016]    Certain software tools, such as ClearView A/V Analyzers by US-based Video Clarity, allow playout, capture and direct visualization of A-B pictures and further calculation of PSNR values or more sophisticated error metrics. 
         [0017]    http://www.videoclarity.com/CVSoftwareOM.html, http://www.videoclarity.com/PDF/ClearViewDataSheet.pdf 
         [0018]    However, in case of even small A vs. B discrepancies in frame sizes, color spaces, time-line positions, etc., these tools are in fact not applicable—because the total contribution of these “secondary” factors to the integral sum(abs(A-B)) error is typically much larger then the strength of the artefacts to be measured. 
         [0019]    All attempts to automatically estimate these discrepancies and automatically compensate their effect (i.e. auto-equalize A with B) have been rather unsuccessful. 
         [0020]    On the other hand, there are well-known objective techniques, such as time-code insertion for time-line position reading, and automatic measurement of video processor parameters based on artificial test patterns. 
         [0021]    However, these techniques have been so far not used in compression artefacts measurements tools available on the market—mainly because of the out-dated assumption that the purpose of video compression codec is to produce output picture as close as possible to the primary reference, i.e. to the original picture—byte by byte, dot by dot. 
         [0022]    In fact, modern multi-format content delivery system processes original high quality content (i.e. primary reference, typically coming from a single source) and delivers it as a set of streamed or downloaded pictures in a variety of frame sizes, aspect ratios, frame rates and even color spaces. 
         [0023]    In such system the set of output (delivered) images should look on the screens of the appropriate players as close as possible to a set of best available secondary references, i.e. to optimally converted versions of the original picture presented in a variety of formats. 
         [0024]      FIG. 1  illustrates a prior art video compression quality measurement system block diagram. It should be noted that prior art systems typically use external sources of test materials and/or test patterns and external devices to measure the quality loss due to the encoding of video content. 
         [0025]    Referring initially to  FIG. 1 , input video content package typically contains descriptive metadata  102  as well as main video content data  104 —typically in uncompressed format. 
         [0026]    In test mode this input video is replaced by the test stream  106 , which may represent static or dynamic test pattern, or even short video clip—so called “reference video”. 
         [0027]    Via input selector  108  input video data  110  are fed to the compression encoder  112 , controlled by Media Assets Management System  114  and/or Operator (Compressionist), providing coding preset  116  based among other factors on the incoming metadata  102 . 
         [0028]    Encoder  112  outputs compressed video stream  118  going to the Content Delivery Network  120 . 
         [0029]    Reference decoder  122  converts compressed stream  118  to the decompressed data  124 , thus allowing calculation of differential (“A-B”) video stream  128  in the block  126 . 
         [0030]    Stream  128 , which represents compression artefacts (errors), goes into the block  130 , which calculates compression quality estimate (quality score) in accordance to some commonly accepted algorithm (metric). 
         [0031]    The result is Quality Report  132  document (set of compression quality scores). 
         [0032]    Major drawback of such architecture is its incapacity to handle any modification of picture parameters except the compression itself. 
         [0033]    Another well-known vulnerability of all existing compression quality measurement systems is a lack of commonly accepted test sequences suitable for modern multi-format Content Delivery Networks. 
         [0034]    Popular video test materials, such as live clips, are usually adequate only for some specific applications and cover only small range of frame sizes and bitrates. 
         [0035]    Thus, fundamentally different Video Quality Control technologies are needed. 
         [0036]    Scientific approach should be based on the development of artificial, repeatable and scalable “Compression Stress Tracker” test pattern covering much wider range of video formats. 
         [0037]    In any case, reliable information about global spatial, temporal, and color space parameters of the delivered video must be available prior to actual compression artefacts assessment. 
         [0038]    For this purpose some video QC systems use descriptive technical metadata, but they are prone to human mistakes and often missing. 
         [0039]    The most reliable way to provide the necessary information about the delivered video consists in the automated measurement of pre-inserted reference markers or “stamps”. 
         [0040]    For correct operation of the video quality analyzer it is highly desirable to have such stamps in the incoming video and use them as a “helper” for accurate compression artefacts measurements. 
       SUMMARY 
       [0041]    Systems and methods are disclosed for testing video quality by generating a stress tracker test pattern with one or more moving zone plates and one or more stamps; determining compression quality scores for encoder resources spent at predetermined levels of compression (stress); and analyzing the test pattern and generating a Compression Stress Response profile. 
         [0042]    In one aspect, a system to perform automated analysis of video quality of a video processor or complete content delivery system, encompassing among others blocks video scalers, encoders, transcoders and decoders/players, and including (1) “clean zone” insertion means, which put into video images at least one area of pre-defined size and position, consisting of pre-defined static or dynamic test pattern, thus creating first component of primary reference video sequence, (2) “compression stress zone” insertion means, which put into original primary reference video images at least one area of pre-defined size and position, consisting of pseudo-random textures, the textures luminance and chrominance contrast and/or texture size varying along the time-line in accordance with the pre-defined set of stress levels, thus creating second component of primary reference video sequence; together said components form complete compression stress test sequence. 
         [0043]    In another aspect, a system to perform automated analysis of video quality of a video processor or complete content delivery system, encompassing among others blocks video scalers, encoders, transcoders and decoders/players, and including (1) “reference stamps” insertion means, which put into original, typically uncompressed, video images a set of pre-defined area stamps, including predefined content code (clip number) stamp, time-code stamps, spatial position (geometry) stamps, and color space stamps, thus creating primary reference video sequence, (2) means for automatic input video format detection and conversion of delivered video data into uncompressed format, (3) means for automatic measurements of parameters of all stamps contained within the delivered images, (4) means for creation or retrieval of secondary reference video sequence matching delivered video images in size, spatial position, aspect ratio, time-line position and color space, (5) means for error image calculation providing a difference between delivered video sequence and secondary reference sequence, (6) means for conversion of the said differential images into objective statistical values, which calculate these values separately for stress zone and clean zone, and separately for each stress level, thus creating measured stress response time profile, (7) means for conversion of said objective statistical values into reported objective score values correlated with traditional subjective image quality scores. 
         [0044]    Main video, underlying reference stamps, could be a stress test sequence, or another artificial test pattern, or any live clip, or any combination of these types suitable for particular video quality testing task. 
         [0045]    The system can be used for a plethora of video quality tests, e.g. for benchmarking of scalers and/or compression codecs. 
         [0046]    Moreover, in one embodiment where the video processors are based on multi-thread parallel calculation schemes, the processing of the short stamped reference test stream may happen simultaneously with the main (unstamped) video content processing. 
         [0047]    And all parallel threads can be controlled by the same settings; thus the impairments of main video stream, e.g. color space errors or compression distortions, can be assessed by objective measurement of the corresponding impairments of the accompanying test stream. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0048]    Embodiments of the invention described herein will become apparent from the following detailed description considered in connection with the accompanying drawings, which disclose several embodiments of the invention. It should be understood, however, that the drawings are designed for the purpose of illustration and not as limits of the invention. 
           [0049]      FIG. 1  illustrates a prior art video quality measurement system block diagram. 
           [0050]      FIG. 2  shows an exemplary Stress Tracker Test Pattern with Moving Zone Plate and Stamps. 
           [0051]      FIG. 3  shows exemplary snapshots of “Golfer” live clip with Stamps. 
           [0052]      FIG. 4  shows an exemplary Stress Tracker test sequence timeline. 
           [0053]      FIG. 5  shows an exemplary variant of Stress Tracker Test with static picture in the Clean Zone. 
           [0054]      FIG. 6  shows one embodiment of a Video Compression Quality Meter system block diagram. 
       
    
    
     DETAILED DESCRIPTION 
       [0055]      FIG. 2  shows example of Stress Tracker Test Pattern with Moving Zone Plate and Stamps. 
         [0056]    This test pattern allows calculation of compression quality scores for several levels of “stress”, which means here the amount of compression encoder resources spent. 
         [0057]    In combination with the appropriate meter/analyzer this test pattern allows building of 
         [0058]    Compression Stress Response Profile. Such profiles are critical for benchmarking, acceptance tests and comparison of various encoding presets. 
         [0059]    In the example shown the test pattern consists of flat gray background  202 , one Clean Zone, two Stress Zones and two sets of Reference Stamps. For better noise immunity all stamps of the set are repeated twice—at the top and bottom of the image. 
         [0060]    Pattern Code Stamp  204  represents in binary format (9 bit in this example) an ID code of the pattern used. This allows automatic recognition of the incoming video ID and automatic selection of the matching secondary reference data. 
         [0061]    Color Reference Stamp  206  contains several shades of Gray and calibrated Green patch, plus digital burst of the highest possible frequency. These components provide for automatic detection and measurement of any color space modifications introduced by video data processing within the Content Delivery Network. 
         [0062]    Frame Number Stamp  208  (16 bit binary in this example) serves for automatic recognition of the incoming video frame time-line position within a playout loop and automatic selection of the matching secondary reference video frame. 
         [0063]    Four Geometry Reference Stamp  208  (in this example, four white crosses on black background) provide for automatic measurement of image geometry modifications introduced by video data processing within the Content Delivery Network (e.g. aspect ratio conversion) and automatic selection of the matching secondary reference video frame geometry. 
         [0064]    Light Gray rectangle  212  designates the boundary of Clean Zone, containing Zone Plate Sprite  214  moving along the elliptic trajectory  216 . 
         [0065]    Current Stress Level Indicator  218  serves as a visual guide; it is not used for any automatic calculations. 
         [0066]    Stress Zone  220  contains pseudo-random YUV texture, which stepwise increases its contrast along the time-line, and its right boundary  222  expands rightwards along the time-line. 
         [0067]    Stress Zone  224  contains another (uncorrelated) pseudo-random YUV texture, which also increases its contrast and its left boundary  226  expands leftwards along the time-line. 
         [0068]    It should be noted, that encoding of stress zones textures requires significant encoder resources, which may result in the significant distortion of all test pattern components, including those situated in the Clean Zone, in particular the distortion of the Zone Plate Sprite  214 . Analysis of Zone Plate spectrum provides valuable additional information about the quantization scales controls and buffer occupancy controls chosen by the encoder in response to the stress. 
         [0069]      FIG. 3  shows example of “Golfer” Live Clip with Stamps. 
         [0070]    Stamps shown are similar to those described for  FIG. 2 , but this test is not subdivided in zones. This example illustrates that Stamps can be used in combination with traditional compression artefacts estimation methodology based on live clips. Main advantage of this test vs. traditional tests, not containing stamps, is its suitability to work even after image geometry modification, frame size and/or color space modifications. 
         [0071]      FIG. 4  shows example of Stress Tracker Test Sequence Timeline. 
         [0072]    Size and contrast of Stress Zone textures increment in several steps along the time-line from zero to maximum. 
         [0073]    In the example shown it means ten steps, i.e. ten different levels of stress. 
         [0074]    Total duration of video loop is typically set between 50 and 100 seconds, allowing enough time for the encoder to optimize its behavior during each of ten steps. 
         [0075]      FIG. 5  shows variant of Stress Tracker Test with Static Picture in the Clean Zone. 
         [0076]    The advantage of this variant vs. Zone Plate variant, shown on  FIG. 2 , is larger number of colors in the palette and less demanding distribution of spatial frequencies. 
         [0077]    Another advantage of this variant is that the static central part can be captured off LCD screen by any still camera or video camera without the need for frame rate synchronization. 
         [0078]      FIG. 6  shows the block diagram of one embodiment of the Video Compression Quality Meter system block diagram. 
         [0079]    The embodiment of  FIG. 6  is particularly advantageous in digital video distribution systems, especially to the hardware and software systems and devices used for multi-format content production, post-production, re-purposing and delivery. It is particularly efficient with application to Content Delivery Networks (CDN). 
         [0080]    Referring now to  FIG. 6 , input live video  602  is converted by Stamp Inserter  604 , driven by Stamp Generator  606 , into stamped video data  608 . 
         [0081]    These data are captured for further use in local storage device  610  and also fed to the input selector  612 . Selector  612  allows optional replacement of the incoming live video by pre-captured version of the video stream in question, or by a locally stored test pattern or by another video clip available in the storage  610 . 
         [0082]    From selector  612  primary reference video data stream  614  goes into compression encoder  616 , controlled by Media Assets Management System  618  and/or Operator (Compressionist), providing a coding preset  620  based among other factors on the incoming metadata  622 . 
         [0083]    Compressed video stream  624  via Content Delivery Network  626  comes to the reference decoder  628 . Decompressed video  630  is not necessarily suitable for comparison with the primary reference video  614 , for example because of the different frame sizes. 
         [0084]    Stamps, contained in video stream  630  are measured/decoded in the Reference Stamp Meter  632 , which controls the Secondary Reference Generator  636 . 
         [0085]    This important block converts a stored copy  634  of primary reference video, replayed from storage  610 , into Secondary Reference Video  638 , suitable for comparison with decoded video  630 . 
         [0086]    If necessary, the Secondary Reference Generator  636  can apply (online or offline) spatial scaling (including image geometry modification), color correction and color space conversion. It is also capable of finding in the storage  610  a video frame with pattern ID and time-line position matching those of the current frame of video stream  630 . 
         [0087]    Block  640  performs calculation of differential (“A-B”) video stream  642 , which represents compression artefacts (errors), in the format matching the format of the delivered images at the CDN  626  output. 
         [0088]    Differential stream  642  goes into the block  644 , which calculates compression quality estimate (quality score) in accordance to some commonly accepted algorithm (metric). 
         [0089]    The result is Quality Report  646  document (set of compression quality scores). 
         [0090]    Unlike prior art system, the system of  FIG. 6  can measure compression artefacts and other distortions in much wider range of conditions—with different frame sizes and even in presence of short-term skips/freezes of the delivered video stream. 
         [0091]    Because the reference stamps are mainly static and occupy only a small fraction of total image area, their presence does not significantly affect the payload of compression codec. 
         [0092]    Thus, the quality measurements are not significantly biased by the presence of the stamps. 
         [0093]    The secondary reference video sequence may be created in advance and stored within the video quality analyzer or created on-the-fly in parallel with the process of delivered content capture, once the parameters of input content package are known. 
         [0094]    It is desirable, so not absolutely necessary, that the secondary reference video sequence contains reference stamps identical to those inserted into incoming video. 
         [0095]    If present, stamp areas are used in the quality measurement the same way as other image areas, i.e. in absence of significant errors they are not visible in the differential images. 
         [0096]    Correct operation of video quality analyzer depends on its capability to retrieve or create appropriate secondary reference video stream. 
         [0097]    It should be noted that retrieval or generation of down-converted secondary reference video (co-timed, scaled and color-corrected version of the primary reference video) usually requires only a fraction of the available resources. 
         [0098]    However, the system may work even without the inserted stamps. In such case manual scaling, time offset and color corrections controls may replace automatic controls, though it may require much more time and video quality measurement accuracy may suffer.