Multimedia Group

TEST PLAN

Draft Version 1.5b

April 212, 2005

Contact: D. Hands Tel: +44 (0)1473 648184

Fax: +44 (0)1473 644649

E-Mail: david.2.hands@bt.com

Editorial History

  ------------- ----------- -------------------------------------------------
  **Version**   **Date**    **Nature of the modification**

  1.0           July 25,    Initial Draft, edited by H. Myler
                2001        

  1.1           28 January, Revised First Draft, edited by David Hands
                2004        

  1.2           19 March,   Text revised following VQEG Boulder 2004 meeting,
                2004        edited by David Hands

  1.3           18 June     Text revised during VQEG meeting, Rome 16-18 June
                2004        2004

  1.4           22October   Text revised during VQEG meeting, Seoul meeting
                2004        October 18-22, 2004

  1.5           18 March    Text revised during MM Ad Hoc Web Meeting, March
                2005        10-18, 2005

  1.5a          22 April    Text revised to include input from GC, IC and CL
                2005        
  ------------- ----------- -------------------------------------------------

Summary

1\. Introduction [6](#introduction)

2\. List of Definitions [8](#list-of-definitions)

3\. List of Acronyms [11](#list-of-acronyms)

4\. Subjective Evaluation Procedure
[13](#subjective-evaluation-procedure)

4.1. The ACR Method with Hidden Reference Removal
[13](#the-acr-method-with-hidden-reference-removal)

4.1.1. General Description [13](#general-description)

4.1.2. Application across Different Video Formats and Displays
[14](#application-across-different-video-formats-and-displays)

4.1.3. Display Specification and Set-up
[14](#display-specification-and-set-up)

4.1.4. Subjects [14](#__RefHeading___Toc101866004)

4.1.5. Viewing Conditions [15](#__RefHeading___Toc101866008)

4.1.6. Test Data Collection [15](#__RefHeading___Toc101866011)

4.2. Data Format [15](#__RefHeading___Toc101866013)

4.2.1. Results Data Format [15](#__RefHeading___Toc101866016)

4.2.2. Subjective Data Analysis [16](#__RefHeading___Toc101866018)

5\. Test Laboratories and Schedule [17](#test-laboratories-and-schedule)

5.1. Independent Laboratory Group (ILG)
[17](#__RefHeading___Toc101866021)

5.2. Proponent Laboratories [17](#__RefHeading___Toc101866023)

5.3. Test Schedule [17](#__RefHeading___Toc101866025)

6\. Sequence Processing and Data Formats
[19](#sequence-processing-and-data-formats)

6.1. Sequence Processing Overview [19](#sequence-processing-overview)

6.1.1. Camera and Source Test Material Requirements
[19](#camera-and-source-test-material-requirements)

6.1.2. Software Tools [19](#software-tools)

6.1.3. De-Interlacing [20](#de-interlacing)

6.1.4. Cropping & Rescaling [20](#cropping-rescaling)

6.1.5. Rescaling [20](#rescaling)

6.1.6. File Format [20](#file-format)

6.1.7. Source Test Video Sequence Documentation
[21](#source-test-video-sequence-documentation)

6.2. Test Materials [21](#test-materials)

6.2.1. Selection of Test Material (SRC)
[21](#selection-of-test-material-src)

6.3. Hypothetical Reference Circuits (HRC)
[21](#hypothetical-reference-circuits-hrc)

6.3.1. Video Bit-rates [22](#video-bit-rates)

6.3.2. Simulated Transmission Errors
[22](#simulated-transmission-errors)

6.3.3. Live Network Conditions [24](#live-network-conditions)

6.3.4. Pausing with Skipping and Pausing without Skipping
[24](#pausing-with-skipping-and-pausing-without-skipping)

6.3.5. Frame Rates [25](#frame-rates)

6.3.6. Pre-Processing [25](#pre-processing)

6.3.7. Post-Processing [25](#post-processing)

6.3.8. Coding Schemes [26](#coding-schemes)

6.3.9. Distribution of Tests over Facilities
[26](#distribution-of-tests-over-facilities)

6.3.10. Processing and Editing Sequences
[26](#processing-and-editing-sequences)

6.3.11. Randomization [26](#randomization)

6.3.12. Presentation Structure of Test Material
[26](#presentation-structure-of-test-material)

7\. Objective Quality Models [27](#objective-quality-models)

7.1. Model Type [27](#model-type)

7.2. Model Input and Output Data Format
[27](#model-input-and-output-data-format)

7.3. Submission of Executable Model
[28](#submission-of-executable-model)

7.4. Registration [28](#registration)

7.5. Results Analysis [29](#__RefHeading___Toc101866067)

7.5.1. Averaging [30](#__RefHeading___Toc101866068)

7.5.2. Averaging Without Extreme Values.
[30](#__RefHeading___Toc101866070)

8\. Objective Quality Model Evaluation Criteria
[32](#objective-quality-model-evaluation-criteria)

8.1. Evaluation Procedure [32](#evaluation-procedure)

8.2. Data Processing [32](#data-processing)

8.2.1. Mapping to the Subjective Scale
[32](#mapping-to-the-subjective-scale)

8.2.2. Averaging Process [33](#averaging-process)

8.2.3. Aggregation Procedure [33](#__RefHeading___Toc101866095)

8.3. Evaluation Metrics [33](#evaluation-metrics)

8.3.1. Pearson Correlation Coefficient
[33](#pearson-correlation-coefficient)

8.3.2. Root Mean Square Error [34](#root-mean-square-error)

8.3.3. Outlier Ratio [35](#__RefHeading___Toc101866102)

8.4. Statistical Significance of the Results
[35](#statistical-significance-of-the-results)

8.4.1. Significance of the Difference between the Correlation
Coefficients
[35](#significance-of-the-difference-between-the-correlation-coefficients)

8.4.2. Significance of the Difference between the Root Mean Square
Errors
[36](#significance-of-the-difference-between-the-root-mean-square-errors)

8.4.3. Significance of the Difference between the Outlier Ratios
[36](#significance-of-the-difference-between-the-outlier-ratios)

8.5. Generalizability [37](#__RefHeading___Toc101866109)

8.6. Complexity [37](#__RefHeading___Toc101866112)

9\. Recommendation [38](#recommendation)

10\. Bibliography [40](#__RefHeading___Toc101866115)

# Introduction

This document defines the procedure for evaluating the performance of
objective perceptual quality models submitted to the Video Quality
Experts Group (VQEG) formed from experts of ITU-T Study Groups 9 and 12
and ITU-R Study Group 6. It is based on discussions from various
meetings of the VQEG Multimedia working group (MM), on 6-7 March in
Hillsboro, Oregon at Intel and on 27-30 January 2004 in Boulder,
Colorado at NTIA/ITS.

The goal of the MM group is to recommend a quality model suitable for
application to digital video quality measurement in multimedia
applications. Multimedia in this context is defined as being of or
relating to an application that can combine text, graphics, full-motion
video, and sound into an integrated package that is digitally
transmitted over a communications channel. Common applications of
multimedia that are appropriate to this study include video
teleconferencing, video on demand and Internet streaming media. The
measurement tools recommended by the MM group will be used to measure
quality both in laboratory conditions using a FR method and in
operational conditions using RRNR methods.

In the first stage of testing, it is proposed that video only test
conditions will be employed. Subsequent tests will involve audio-video
test sequences, and eventually true multimedia material will be
evaluated. It should be noted that presently there is a lack of both
audio-video and multimedia test material for use in testing. Video
sequences used in VQEG Phase I remain the primary source of freely
available (open source) test material for use in subjective testing. The
VQEG does desire to have copyright free (or at least free for research
purposes) material for testing. The capability of the group to perform
adequate audio-video and multimedia testing is dependent on access to a
bank of potential test sequences.

The performance of objective models will be based on the comparison of
the MOS obtained from controlled subjective tests and the MOSp predicted
by the submitted models. This testplan defines the test method or
methods, selection of test material and conditions, and evaluation
metrics to examine the predictive performance of competing objective
multimedia quality models.

The goal of the testing is to examine the performance of proposed video
quality metrics across representative transmission and display
conditions. To this end, the tests will enable assessment of models for
mobile/PDA and broadband communications services. It is considered that
FR-TV and RRNR-TV VQEG testing will adequately address the higher
quality range (2 Mbit/s and above) delivered to a standard definition
monitor. Thus, the Recommendation(s) resulting from the VQEG MM testing
will be deemed appropriate for services delivered at 2 Mbit/s or less
presented on mobile/PDA and computer desktop monitors.

It is expected that subjective tests will be performed separately for
different display conditions (e.g. one specific test for mobile/PDA;
another test for desktop computer monitor). The performance of submitted
models will be evaluated for each type of display condition. Therefore
it may be possible for one model to be recommended for one display type
(e.g., mobile) and another model for another display format (e.g.,
desktop monitor).

The objective models will be tested using a set of digital video
sequences selected by the VQEG MM group. The test sequences will be
processed through a number of hypothetical reference circuits (HRC\'s).
The quality predictions of the submitted models will be compared with
subjective ratings from human viewers of the test sequences as defined
by this testplan.

A final report will be produced after the analysis of test results.

# List of Definitions 

[Intended frame rate]{.underline} is defined as the number of video
frames per second physically stored for some representation of a video
sequence. The [intended]{.underline} [frame rate]{.underline} may be
constant or may change with time. Two examples of *constant*
[intended]{.underline} [frame rates]{.underline} are a BetacamSP tape
containing 25 fps and a VQEG FR-TV Phase I compliant 625-line YUV file
containing 25 fps; these both have an [absolute frame rate]{.underline}
of 25 fps. One example of a *variable* [absolute frame rate]{.underline}
is a computer file containing only new frames; in this case the
[intended]{.underline} [frame rate]{.underline} exactly matches the
[effective frame rate]{.underline}. The content of video frames is not
considered when determining [intended]{.underline} [frame
rate]{.underline}.

[Anomalous frame repetition]{.underline} is defined as an event where
the HRC outputs a single frame repeatedly in response to an unusual or
out of the ordinary event. [Anomalous frame repetition]{.underline}
includes but is not limited to the following types of events: an error
in the transmission channel, a change in the delay through the
transmission channel, limited computer resources impacting the decoder's
performance, and limited computer resources impacting the display of the
video signal. \[Note: [Anomalous frame repetition]{.underline} is
allowed in the MM test plan, except for the first 1s and the final 1s of
the video sequence. This exception was requested due to potential
interactions between [anomalous frame repetition]{.underline} and the
agreed upon subjective testing methodology.\]

[Constant frame skipping]{.underline} is defined as an event where the
HRC outputs frames with updated content at an [effective frame
rate]{.underline} that is fixed and less than the [source frame
rate]{.underline}. \[Note: [Constant frame skipping]{.underline} is
allowed in the MM test plan and processed video sequences shall have an
intended frame rate equal to the source frame rate.\]

[Effective frame rate]{.underline} (EFR) is defined as the number of
unique frames (i.e., total frames -- repeated frames) per second.

[Frame rate]{.underline} is the number of (progressive) frames displayed
per second (fps).

[Live Network Conditions]{.underline} are defined as errors imposed upon
the digital video bit stream as a result of live network conditions.
Examples of error sources include packet loss due to heavy network
traffic, increased delay due to transmission route changes, multi-path
on a broadcast signal, and fingerprints on a DVD. [Live network
conditions]{.underline} tend to be unpredictable and unrepeatable.

[Pausing with skipping (formerly frame skipping)]{.underline} is defined
as events where the video pauses for some period of time and then
restarts with some loss of video information. In [pausing with
skipping]{.underline}, the temporal delay through the system will vary
about an average system delay, sometimes increasing and sometimes
decreasing. One example of [pausing with skipping]{.underline} is a pair
of IP Videophones, where heavy network traffic causes the IP Videophone
display to freeze briefly; when the IP Videophone display continues,
some content has been lost. Another example is a videoconferencing
system that performs [constant frame skipping]{.underline} or [variable
frame skipping]{.underline}. A processed video sequence containing
[pausing with skipping]{.underline} will be approximately the same
duration as the associated original video sequence. \[Note: [pausing
with skipping]{.underline} is allowed in the MM test plan. Pausing with
skipping is disallowed for the first 1s and the final 1s of the video
sequence.\]

P[ausing without skipping (formerly frame freeze)]{.underline} is
defined as any event where the video pauses for some period of time and
then restarts without losing any video information. Hence, the temporal
delay through the system must increase. One example of [pausing without
skipping]{.underline} is a computer simultaneously downloading and
playing an AVI file, where heavy network traffic causes the player to
pause briefly and then continue playing. A processed video sequence
containing [pausing without skipping]{.underline} events will always be
longer in duration than the associated original video sequence. \[Note:
[pausing without skipping]{.underline} is not allowed in the MM test
plan.\]

[Refresh rate]{.underline} is defined as the rate at which the computer
monitor is updated.

[Simulated transmission errors]{.underline} are defined as errors
imposed upon the digital video bit stream in a highly controlled
environment. Examples include simulated packet loss rates and simulated
bit errors. Parameters used to control [simulated transmission
errors]{.underline} are well defined.

[Source frame rate]{.underline} (SFR) is the [intended]{.underline}
[frame rate]{.underline} of the original source video sequences. The
source frame rate is constant. For the MM testplan the SFR may be either
25 fps or 30 fps.

[Transmission errors]{.underline} are defined as any error imposed on
the video transmission. Example types of errors include [simulated
transmission errors]{.underline} and [live network
conditions]{.underline}.

[Variable frame skipping]{.underline} is defined as an event where the
HRC outputs frames with updated content at an [effective frame
rate]{.underline} that changes with time. The temporal delay through the
system will increase and decrease with time, varying about an average
system delay. A processed video sequence containing [variable frame
skipping]{.underline} will be approximately the same duration as the
associated original video sequence. \[Note: [Variable frame
skipping]{.underline} is allowed in the MM test plan and processed video
sequences shall have an intended frame rate equal to the source frame
rate.\]

# List of Acronyms

ACR-HRR Absolute Category Rating with Hidden Reference Removal

ANOVA ANalysis Of VAriance

ASCII ANSI Standard Code for Information Interchange

CCIR Comite Consultatif International des Radiocommunications

CODEC COder-DECoder

CRC Communications Research Centre (Canada)

DVB-C Digital Video Broadcasting-Cable

FR Full Reference

GOP Group Of Pictures

HRC Hypothetical Reference Circuit

IRT Institut Rundfunk Technische (Germany)

ITU International Telecommunication Union

MM MultiMedia

MOS Mean Opinion Score

MOSp Mean Opinion Score, predicted

MPEG Moving Picture Experts Group

NR No (or Zero) Reference)

NTSC National Television Standard Code (60 Hz TV)

PAL Phase Alternating Line standard (50 Hz TV)

PS Program Segment

QAM Quadrature Amplitude Modulation

QPSK Quadrature Phase Shift Keying

RR Reduced Reference

SMPTE Society of Motion Picture and Television Engineers

SRC Source Reference Channel or Circuit

SSCQE Single Stimulus Continuous Quality Evaluation

VQEG Video Quality Experts Group

VTR Video Tape Recorder

# Subjective Evaluation Procedure

## The ACR Method with Hidden Reference Removal

This section describes the test method according to which the VQEG
multimedia (MM) subjective tests will be performed. We will use the
absolute category scale (ACR) \[Rec. P.910\] for collecting subjective
judgments of video samples. ACR is a single-stimulus method in which a
processed video segment is presented alone, without being paired with
its unprocessed ("reference") version. The present test procedure
includes a reference version of each video segment, not as part of a
pair, but as a freestanding stimulus for rating like any other. During
the data analysis the ACR scores will be subtracted from the
corresponding reference scores to obtain a DMOS. This procedure is known
as "hidden reference removal."

### General Description

The selected test methodology is the single stimulus Absolute Category
Rating method with hidden reference removal (henceforth referred to as
ACR-HRR). This choice has been selected due to the fact that ACR
provides a reliable and standardized method (ITU-R Rec. 500-11, ITU-T
P.910) that allows a large number of test conditions to be assessed in
any single test session.

In the ACR test method, each test condition is presented singly for
subjective assessment. The test presentation order is randomized
according to standard procedures (e.g. Latin or Graeco-Latin square, or
via random number generator). The test format is shown in Figure 1. At
the end of each test presentation, human judges (\"subjects\") provide a
quality rating using the ACR rating scale below.

5 Excellent

4 Good

3 Fair

2 Poor

1 Bad

![](media/image1.wmf){width="6.694444444444445in"
height="1.3652777777777778in"}

Figure 1 -- ACR basic test cell.

Instructions to the subjects provide a more detailed description of the
ACR procedure. The instruction script appears in Annex I.

### Application across Different Video Formats and Displays

The proposed MM test will examine the performance of objective
perceptual quality models for different video formats (Rec. 601, CIF and
QCIF). Section 4.1.3 defines format and display types in detail. Video
applications targeted in this test include internet video, mobile video,
video telephony, and streaming video.

Presently, VQEG MM assumes a rolling program of tests. The audio-video
tests are expected to involve three separate stages. Stage 1 will assess
video quality only; the current Test Plan covers Stage 1. Stage 2 will
assess audio quality only. Stage 3 will assess overall audio-video
quality. The specification and selection of video cards for Stage 1 is
still to be decided.

The instructions given to subjects request subjects to maintain a
specified viewing distance from the display device. The viewing distance
has been agreed as:

-   QCIF: nominally 6-10 picture heights (H), and let the viewer choose
    within physical limits (natural for PDAs).

-   CIF: 6-8H and let the viewer choose within physical limits.

-   Rec. 601: 6H

H=Picture Heights (picture is defined as the size of the video window)

We note regarding the Stage 2 and Stage 3 audio and audio-video tests,
that the room must be acoustically isolated and conform to relevant
international standards (e.g. ITU-T Rec. P.800. and ITU-R Rec. BS.1116).
Use of headphones will be investigated and perhaps included or mandated
in the test (e.g., Stax diffused field equalized Headphones). The
specification and selection of audio cards is to be decided.

### Display Specification and Set-up

Given that the subjective tests will use LCD displays, it is necessary
to ensure that each test laboratory selects appropriate display
specification and common set-up techniques are employed. This Test Plan
requires that LCD displays meet the following specifications:

The LCD shall be set-up using the following procedure:

-   Use the autosetting to set the default values for luminance,
    contrast and colour shade of white.

-   Adjust the brightness according to Rec. ITU-T P.910, but do not
    adjust the contrast (it might change balance of the colour
    temperature).

-   Set the gamma to 2.2.

```{=html}
<!-- -->
```
-   Set the colour temperature to 6500 K (default value on most LCDs).

The LCD display shall be a high-quality monitor for which it can be
verified that different displays of same model and brand name use the
same panel inside (i.e., either from the display manufacturer or through
the TCO-testing labs, e.g. \[Editor's note: TBD; Minimum response time
should be ??(e.g. 16ms) 17 inch ?)\]). The LCD display that is selected
shall have a pixel pitch similar to that currently available on PDAs and
mobile phones. It is preferred that all subjective tests use the same
LCD monitor panel. This will facilitate data analysis using data from
different tests.

[]{#__RefHeading___Toc101866004 .anchor}Subjects

Each test will require at least 24 subjects. It is recommended that as
many subjects as possible participate in each test in order to improve
the statistical power of the resulting data. It is preferred that each
subject be given a different ordering of video sequences where possible.
Otherwise, the viewers will be assigned to sub-groups, which will see
the test sessions in different orders. At least two different orderings
of test sequences are required per subjective test. \[Note: This section
will need to be re-written if we get software to randomly select
sequences of scenes for each individual subject.\]

Only non-expert viewers will participate. The term non-expert is used in
the sense that the viewers' work does not involve video picture quality
and they are not experienced assessors. They must not have participated
in a subjective quality test over a period of six months. All viewers
will be screened prior to participation for the following:

-   normal (20/20) visual acuity with or without corrective glasses (per
    Snellen test or equivalent).

-   normal colour vision (per Ishihara test or equivalent).

-   familiarity with the language sufficient to comprehend instruction
    and to provide valid responses using semantic judgment terms
    expressed in that language.

[]{#__RefHeading___Toc101866008 .anchor}Viewing Conditions

Each test session will involve only one subject per display assessing
the test material. Subjects will be seated directly in line with the
center of the video display at a distance of 6H (six picture heights).
The test cabinet will conform to ITU-T Rec. P.910 requirements. \[Note:
This section will be updated if a chin rest is required.\]

[]{#__RefHeading___Toc101866011 .anchor}Test Data Collection

The responsibity for the collection and organization of the data files
containing the votes will be shared by the ILG Co-Chairs and the
proponents. The collection of data will be supervised by the ILG and
distributed to test participants for verification.

[]{#__RefHeading___Toc101866013 .anchor}Data Format

[]{#__RefHeading___Toc101866016 .anchor}Results Data Format

The following format is designed to facilitate data analysis of the
subjective data results file.

The subjective data will be stored in a Microsoft Excel spreadsheet
containing the following columns in the following order: lab, test,
type, subject #, month, day, year, session, resolution, rate, age,
gender, order, scene, HRC, ACR Score. Missing data values will be
indicated by the value -9999 to facilitate global search and replace of
missing values. Each Excel spreadsheet cell will contain either a number
or a name. All names (e.g., test, lab, scene, hrc) must be ASCI strings
containing no white space (e.g., space, tab) and no capital letters.
Where exact text strings are to be used, the text strings will be
identified below in single quotes (e.g., 'original'). Only data from
valid viewers (i.e., viewers who pass the visual acuity and color tests)
will be forwarded to the ILG and other proponents.

Below are definitions for the Excel spreadsheet columns:

[Lab:]{.underline} Name of laboratory's ornaizationorganization (e.g.,
CRC, Intel, NTIA, NTT, etc.). This abbreviation must be a single word
with no white space (e.g., space, tab).

[Test:]{.underline} Name of the test. Each test must have a unique name.

[Type:]{.underline} Name of the test category. \[Editor's note: exact
text strings will be specified after individual test categories have
been finalized.\]

[Subject #:]{.underline} Integer indicating the subject number. Each
laboratory will start numbering viewers at a different point, to ensure
that all viewers receive unique numbering. Starting points will be
separated by 1000 (e.g., lab1 starts numbering at 1000, lab2 starts
numbering at 2000, etc). Subjects' names will *not* be collected or
recorded.

[Month:]{.underline} Integer indicating month \[1..12\]

[Day:]{.underline} Integer indicating day \[1..31\]

[Year:]{.underline} Integer indicating year \[2004..2006\]

[Session:]{.underline} Integer indicating viewing session

[Resolution:]{.underline} One of the following three strings: 'rec601',
'cif' or 'qcif'.

[Rate:]{.underline} A number indicating the frames per second (fps) of
the original video sequence.

[Age:]{.underline} Integer number that indicates the subject's age.

[Gender:]{.underline} 'f' for female, 'm' for male

[Order:]{.underline} An integer indicating the order in which the
subject viewed the video sequences \[or trial number, if scenes are
ordered randomly\].

[Scene:]{.underline} Name of the scene. All scenes from all tests must
have unique names. If a single scene is used in multiple tests (i.e.,
digitally identical files), then the same scene name must be used. Names
shall be eight characters or fewer.

[HRC:]{.underline} Name of the HRC. For reference video sequences, the
exact text 'reference' must be used. All processed HRCs from all tests
must have unique names. If a single HRC is used in multiple tests, then
the same HRC name must be used . HRC names shall be eight characters or
fewer.

[ACR Score:]{.underline} Integer indicating the subject's ACR score (1,
2, 3, 4, or 5).

See Annex II for an example.

[]{#__RefHeading___Toc101866018 .anchor}Subjective Data Analysis

Each subject\'s results will be checked for completeness. An observer is
discarded if the number of failed votes exceeds one in one of the
sessions. Additionally, the observers will be screened after the test as
specified in sec. 2.3.1 of Annex 2 "Screening for DSIS, DSCQS and
alternative methods except SSCQE method" of recommendation ITU-R
BT.500-10. The post-test screening will be applied to all subjects in a
given lab that see the same test sequences---regardless of ordering.
Section 2.3.1, Note 1, says, "\...use of this procedure should be
restricted to cases in which there are relatively few observers (e.g.,
fewer than 20), all of whom are non-experts."

Difference scores will be calculated for each processed video sequence
(PVS). A PVS is defined as a SRCxHRC combination. The difference scores,
known as Difference Mean Opinion Scores (DMOS) will be produced for each
PVS by subtracting the score from that of the hidden reference score for
the SRC used to produce the PVS. Subtraction will be done per subject.
Difference scores will be used to assess the performance of each full
reference and reduced reference proponent model, applying the metrics
defined in Section 8.

For evaluation of no-reference proponent models, the absolute (raw)
subjective score will be used. Thus, for each ACR rating, only the
absolute rating for the SRCxHRC (PVS) will be calculated. Based on each
subject's absolute rating for the test presentations, an absolute mean
opinion score will be produced for each test condition. These MOS will
then be used to evaluate the performance of NR proponent models using
the metrics specified in Section 8. \[Ed. Note: This section to be
revised after discussion with proponents submitting No Reference
models.\]

# Test Laboratories and Schedule

Given the scope of the MM testing, both independent test laboratories
and proponent laboratories will be given subjective test
responsibilities. All laboratories will report to VQEG (MMTEST
Reflector) the test environment they plan to use prior to conducting the
subjective test. \[Ed. Note: The template for such reporting has been
provided by P. Corriveau. An audio-call will be arranged to discuss and
decide upon this issue.\]

[]{#__RefHeading___Toc101866021 .anchor}Independent Laboratory Group
(ILG)

The independent laboratory group is composed of IRCCyN (France), FUB
(Italy), FT (France), CRC (Canada), INTEL (USA), Acreo (Sweden), and
Verizon (USA). A proposal from France Telecom has been received where FT
would become an ILG lab.

[]{#__RefHeading___Toc101866023 .anchor}Proponent Laboratories

A number of proponents also have significant expertise in and facilities
for subjective quality testing. Proponents indicating a willingness to
participate as test laboratories are BT, Genista, NTIA, NTT, Opticom,
SwissQual, Psytechnics, TDF, Toyama University, KDDI, and Yonsei. \[Ed
Note: Precise details of how proponent laboratories will create test
material and distribute results from their tests have yet to be
specified.\] It is clearly important to ensure all test data is derived
in accordance with this testplan. Critically, proponent testing must be
free from charges of advantage to one of their models or disadvantage to
competing models. \[Ed. Note: Details of this proposal are to be worked
out by next meeting. Proponents Working Group is established to work out
these details. WG =NTIA, BT, SwissQual, Yonsei, Psytechnics, NTT,
Genista, Opticom, KDDI, TDF, Toyama University, I2R\].

The maximum number of subjective experiments run by any one proponent
laboratory is 3 times the lowest non-zero number run by any other
proponent laboratory, per image size.

The maximum number of non-secret PVSs included in overall test by any
one proponent laboratory 20%.

[]{#__RefHeading___Toc101866025 .anchor}Test Schedule

TABLE 1: Below is the List of Actions and the Associated Schedule

  --------------------- ------------------ ------------ -------------------
  Action                Done by            Source       Destination

  Testplan completed    September, 2005    VQEG         VQEG Reflector, ITU
  and approved                                          

  Call for proponents   May 2004 (DONE)    WP6Q SG9, SG Proponents
  to submit models                         12           
  (ITU-R, ITU-T)                                        

  Final submission of   End of testplan +  Proponents   ILG
  executable model      6 months                        

  Fee payment[^1]       End of testplan +  Proponents   ILG
                        4 months                        

  Declaration by        End of testplan +  Proponents   VQEG
  proponents of         1 month                         
  intention to submit                                   
  model(s); proponents                                  
  identify type of                                      
  model to be submitted                                 
  (incl. image format)                                  

  List of proponent     Fee payment + 1    VQEG         VQEG
  models submitted for  week               co-chairs    
  evaluation                                            

  Delivery of HRC video TBD                Proponents   ILG
  material \[Ed Note:                                   
  depends on the                                        
  procedure for                                         
  preparing test                                        
  material.\]                                           

  Delivery of selected  Final submission   ILG          Proponents
  test material to be   of executable                   
  used in subjective    models + 1 month                
  tests \[Ed Note:                                      
  depends on the                                        
  procedure for                                         
  preparing test                                        
  material.\]                                           

  Completion of Formal  3 months after     Test sites   Test sites
  Subjective Tests      test sites have                 
                        received test                   
                        material                        

  Delivery of objective 3 months after     Proponents   Proponents and ILG
  data                  proponents have                 
                        received test                   
                        material                        

  Verification of       1 month after      Proponents   ILG
  submitted models      subjective and                  
                        objective data                  
                        becomes available               

  Statistical analysis  1 month after      VQEG         VQEG
  (according to         subjective and                  
  statistics defined in objective data                  
  Section 8 of the      becomes available               
  testplan)             \[Ed Note: Check                
                        this time with                  
                        stat. experts\]                 

  Final report          After VQEG         VQEG         WP6Q SG9 SG12
                        approves final                  
                        report                          

  Draft of final report 1 month after      VQEG         VQEG
                        statistical                     
                        analysis has been               
                        completed                       

  VQEG/JRG MMQA meeting Soon after draft   VQEG         VQEG
  to discuss final      of final report                 
  report                becomes available               
  --------------------- ------------------ ------------ -------------------

The ILG will verify that the submitted models (1) run on the ILG's
computers and (2) yield the correct output values when run on the test
video sequences. Due to their limited resources, the ILG may encounter
difficulties verifying executables submitted too close to the model
submission deadline. Therefore, proponents are strongly encouraged to
submit a prototype model to the ILG well before the verification
deadline, to work out platform compatibility problems well ahead of the
final verification date. Proponents are also strongly encouraged to
submit their final model executable 14 days prior to the verification
deadline date, giving the ILG two weeks to resolve problems arising from
the verification procedure.

The ILG requests that proponents kindly estimate the run-speed of their
executables on a test video sequence and to provide this information to
the ILG.

\[Ed. Note: This section will be revised pending finalization of the
test procedure.\]

# Sequence Processing and Data Formats

Separate subjective tests will be performed for different video sizes.
One set of tests will present video in QCIF (176x 144 pixels). One set
of tests will present CIF (352x288 pixels) video. One set of tests will
present VGA (640x480). In the case of 601 video source, aspect ratio
correction will be performed on the video sequences prior to writing the
AVI files (SRC) or processing the PVS.

.Note that in all subjective tests 1 pixel of video will be displayed as
1 pixel native display. No upsampling or downsampling of the video is
allowed at the player.

Presently, VQEG has access to a set of video test sequences. For
audio-video tests this database needs to be extended to include new
source material containing both audio and video.

## Sequence Processing Overview

The test material will be selected from a common pool of video
sequences. If the test sequences are in interlace format then a
standard, agreed de-interlacing method will be applied to transform the
video to progressive format. All source material should be 25 or 30
frames per second progressive. The de-interlacing algorithm will
de-interlace Rec. 601 (or other, e.g., HDTV) formatted video into a
progressive format, e.g., VGA, CIF, or QCIF. Algorithms will be proposed
on the VQEG reflector and approved before processing takes place.
Uncompressed AVI files will be used for subjective and objective tests.
Tools are being sought to convert from the various coding schemes to
uncompressed AVI. The progressive test sequences used in the subjective
tests should also be used by the models to produce objective scores.

It is important to minimize the processing of video source sequences.
Hence, we will endeavor to find methods that minimize this processing
(e.g., to perform de-interlacing and resizing in one step).

### Camera and Source Test Material Requirements

The standard definition source test material should be in Rec. 601,
DigiBeta, Betacam SP, or DV25 (3-chip camera) format or better. Note
that this requirement does not apply to Categories 4 and 8 (Section 6.2)
where the best available quality reference will be used. HD source test
material should be taken from a professional grade HD camera (e.g., Sony
HDR-FX1) or better.  Original HD video sequences that have been
compressed should show no impairments after being re-sampled to VGA,
CIF, and QCIF.

VQEG MM expresses a preference for all test material to be open source.
At a minimum, source material must be available for use within VQEG MM
proponents and ILG for testing (e.g., under non-disclosure agreement if
necessary).

### Software Tools

\[Editor\'s note: the following consensus does not cover tool use for
color space conversion (YUV to RGB) where needed.\] Transformation of
the source test sequences (e.g., from Rec. 601 525-line to CIF) shall be
performed using Avisynth 2.5.5, VirtualDub 1.6.4, and ffdshow 20050303.
Within VirtualDub, video sequences will be saved to AVI files using
Video Compression option \"ffdshow Video Codec\", configured with the
\"Uncompressed\" decoder and the UYVY color space.\[Ed. Note: Arthur W
will put the agreed version of the software tools on the mmtest ftp
server and BT will provide instruction for setup and usage.\]

### De-Interlacing

De-interlacing will be performed when original material is interlaced,
using the de-interlacing function "KernelDeint" in Avisynth.

### Cropping & Rescaling

Table 2 lists recommend values region of interests to be used for
transforming images. These source regions should be centered vertically
and horizontally. These source regions are intended to be applied
*prior* to rescaling and avoid use of over scan video in most cases.
These regions are known to correctly produce square pixels in the target
video sequence. Other regions may be used, provided that the target
video sequence contains the correct aspect ratio.

TABLE 2. Recommended Source Regions for Video Transformation

  -------------------------- --------------------------- -----------------
  **From**                   **To**                      **Source Region**

  525-line: 720x486 Rec. 601 VGA: 640x480 square pixel   704x480

  525-line: 720x486 Rec. 601 CIF: 352x288 square pixel   646x480

  525-line: 720x486 Rec. 601 QCIF: 176x144 square pixel  646x480

  625-line: 720x576 Rec. 601 VGA: 640x480 square pixel   702x576

  625-line: 720x576 Rec. 601 CIF: 352x288 square pixel   644x576

  625-line: 720x576 Rec. 601 QCIF: 176x144 square pixel  644x576

  1080i: 1920x1080           VGA: 640x480 square pixel   1440x1080

  1080i: 1920x1080           CIF: 352x288 square pixel   1320x1080

  1080i: 1920x1080           QCIF: 176x144 square pixel  1320x1080

  720p: 1280x720             VGA: 640x480 square pixel   960x720

  720p: 1280x720             CIF: 352x288 square pixel   880x720

  720p: 1280x720             QCIF: 176x144 square pixel  880x720
  -------------------------- --------------------------- -----------------

### Rescaling

Video sequences will be resized using Avisynth's 'LanczosResize'
function.

### File Format

All source and processed video sequences will be stored in Uncompressed
AVI in UyVy..

Source material with a source frame rate of 29.97 fps will be manually
assigned a source frame rate of 30 fps prior to being inserted into the
common pool of video sequences.

\[Editor's note: insert file format details here.\]

### Source Test Video Sequence Documentation

Preferably, each source video sequence should be documented. The exact
process used to create each source video sequence should be documented,
listing the following information:

-   Camera specifications

-   Source region of interest (if the default values were not used)

-   Use restrictions (e.g., "open source")

This documentation is desirable but not required.

## Test Materials

The test material will be representative of a range of content and
applications. The list below identifies the type of test material that
forms the basis for selection of sequences.

1\) video conferencing \[Ed Note: (available for research purposes only,
NTIA (Rec 601 60Hz); BT to provide more (Rec 601 50Hz), Yonsei (CIF and
QCIF), FT (Rec 601 50Hz, D1)), NTT (Rec 601 60Hz, D1) \]

2\) movies, movie trailers \[Ed Note: (VQEG Phase II??), Opticom
(trailer equivalent, restricted within VQEG) \]

3\) sports, \[Ed Note: (available, + 15-20 mins from Yonsei, + Comcast),
KDD-I (7 min D1 and D2, other scenes also available), NTIA (Comcast) \]

4\) music video, \[Ed Note:Intel? \]

5\) advertisement, \[Ed Note: (Logitech? Arthur will try to contact
Logitech to get permissions to use their material) \]

6\) animation \[Ed Note: (graphics Phase I, cartoon Phase II; Opticom
will send material to Yonsei)\]

7\) broadcasting news \[Ed Note: (head and shoulders and outside
broadcasting). (available -- Yonsei; SVT, possible Comcast) \]

8\) home video \[Ed Note: (FUB possibly, BT possibly, INTEL), NTIA\]
\[Ed Note: To be decided, whether to separately evaluate models for low
quality reference.\]

\[Ed Note: All test material should be sent to Yonsei first and then it
will be put on the ftp server by NTIA. Ideally the material should be
converted before being sent to Yonsei\]

The source video will only be used in the testing if an expert in the
field considers the quality to be good or excellent on an ACR-scale.

### Selection of Test Material (SRC)

## Hypothetical Reference Circuits (HRC)

The subjective tests will be performed to investigate a range of HRC
error conditions. These error conditions may include, but will not be
limited to, the following:

-   Compression errors (such as those introduced by varying bit-rate,
    codec type, frame rate and so on)

-   Transmission errors

-   Post-processing effects

-   Live network conditions

The overall selection of the HRCs will be done such that most, but not
necessarily all, of the following conditions are represented.

### Video Bit-rates

-   PDA/Mobile: 16kbs to 320 kbs (e.g., 16, 32, 64, 128, 192, 320)

-   PC1 (CIF): 128kbs to 704kbs (e.g. 128, 192, 320, 448, 704)

-   PC2 (VGA):320kbs to 4Mbs (e.g. 320, 448, 704, \~1M, \~1.5M, \~2M,
    3M,\~4M)

### Simulated Transmission Errors

A set of test conditions (HRC) will include error profiles and levels
representative of video transmission over different types of transport
bearers:

-   Packet-switched transport (e.g.,2G or 3G mobile video streaming,
    PC-based wireline video streaming)

-   Circuit-switched transport (e.g., mobile video-telephony)

**Packet-switched transmission**

HRCs will include packet loss with a range of packet loss ratios (PLR)
representative of typical real-life scenarios.

In **mobile video streaming**, we consider the following scenarios:

1.  Arrival of packets is delayed due to re-transmission over the air.
    Re-transmission is requested either because packets are corrupted
    when being transmitted over the air, or because of network
    congestion on the fixed IP part. Video will play until the buffer
    empties if no new (error-checked/corrected) packet is received. If
    the video buffer empties, the video will pause until a sufficient
    number of packets is buffered again. This means that in the case of
    heavy network congestion or bad radio conditions, video will pause
    without skipping during re-buffering, and no video frames will be
    lost. This case is not implemented in the current test plan as
    stated in Section 6.3.4.

2.  Arrival of packets is delayed, and the delay is too large: These
    packets are discarded by the video client.

> Note: A radio link normally has *in-order delivery*, which means that
> if one packet is delayed the following packets will also be delayed.
>
> Note: If the packet delay is too long, the radio network might drop
> the packet.

3.  Very bad radio conditions: Massive packet loss occurs.

4.  Handovers: Packet loss can be caused by handovers. Packets are lost
    in bursts and cause image artifacts.

> Note: This is valid only for certain radio networks and radio links,
> like GSM or HSDPA in WCDMA. A dedicated radio channel in WCDMA uses
> soft handover, which not will cause any packet loss.

Typical radio network error conditions are:

-   Packet delays between 100 ms and 5 seconds.

In **PC-based wireline video streaming**, network congestion causes
packet loss during IP transmission.

In order to cover different scenarios, we consider the following models
of packet loss:

-   Bursty packet loss- The packet loss pattern can be generated by a
    link simulator by a bit or block error model, such as the
    Gilbert-Elliott model.

-   Random packet loss

-   Periodic packet loss.

Note: The bursty loss model is probably the most common scenario in a
'normal' network operation. However, periodic or random packet loss can
be caused by a faulty piece of equipment in the network. Bursty, random,
and periodic packet loss models are available in commercially-available
packet network emulators.

Choice of a specific PLR is not sufficient to characterize packet loss
effects, as perceived quality will also be dependent on codecs,
contents, packet loss distribution (profiles) and which types of video
frames were hit by the loss of packets. For our tests, we will select
different levels of loss ratio with different distribution profiles in
order to produce test material that spreads over a wide range of video
quality. To confirm that test files do cover a wide range of quality,
the generated test files (i.e., decoded video after simulation of
transmission error) will be:

1.  Viewed by video experts to ensure that the visual degradations
    resulting from the simulated transmission error spread over a range
    of video quality over different contents;

2.  Checked to ensure that degradations remain within the limits stated
    by the test plan (e.g., in the case where packet loss causes loss of
    complete frames, we will check that temporal misalignment remains
    with the limits stated by the test plan).

**Circuit-switched transmission**

HRCs will include bit errors and/or block errors with a range of bit
error rates (BER) or/and block[^2] error rates (BLER) representative of
typical real-world scenarios. In circuit-switched transmission, e.g.,
video-telephony, no re-transmission is used. Bit or block errors occur
in bursts.

In order to cover different scenarios, the following error levels can be
considered:

\[Editor\'s Note: We need to check the relation between air interface
block error rates and bit stream errors.\]

Air interface block error rates: Normal uplink and downlink: 0.3%,
normally not lower. High value uplink: 0.5%, high downlink: 1.0%. To
make sure the proponents' algorithms will handle really bad conditions
up to 2%-3% block errors on the downlink can be used.

Bit stream errors: Block errors over the air will cause bits to not be
received correctly over the air. A video telephony (H.223) bit stream
will experience CRC errors and chunks of the bit stream will be lost.

Tools are currently being sought to simulate the types of error
transmission described in this section.

Proponents are asked to provide examples of level of error conditions
and profiles that are relevant to the industry. These examples will be
viewed and/or examined after electronic distribution (only open source
video is allowed for this).

### Live Network Conditions

Simulated errors are an excellent means to test the behavior of a system
under well defined conditions and to observe the effects of isolated
distortions. In real live networks however usually a multitude of
effects happen simultaneously when signals are transmitted, especially
when radio interfaces are involved. Some effects like e.g. handovers,
can only be observed in live networks.

The term \"live network\" specifies conditions which make use of a real
network for the signal transmission. This network is not exclusively
used by the test setup. It does not mean that the recorded data
themselves are taken from live traffic in the sense of passive network
monitoring. The recordings may be generated by traditional intrusive
test tools, but the network itself must not be simulated.

Live network conditions of interest include radio transmission (e.g.,
mobile applications) and fixed IP transmission (e.g., PC-based video
streaming, PC to PC video-conferencing, best-effort IP-network with
ADSL-access). Live network testing conditions are of particular value
for conditions that cannot confidently be generated by network simulated
transmission errors (see section 6.3.4). Live network conditions should
exhibit distortions representative of real-world situations that remain
within the limits stated elsewhere in this test plan.

Normally most live network samples are of very good or best quality. To
get a good proportion of sample quality levels, an even distribution of
samples from high to low quality should be saved after a live network
session.

Note: Keep in mind the characteristics of the radio network used in the
test. Some networks will be able to keep a very good radio link quality
until it suddenly drops. Other will make the quality to slowly degrade.

Samples with perfect quality do not need to be taken from live network
conditions. They can instead be recorded from simulation tests.

Live network conditions as opposed to simulated errors are typically
very uncontrolled by their nature. The distortion types that may appear
are generally very unpredictable. However, they represent the most
realistic conditions as observed by users of e.g. 3G networks.

Recording PVSs under live network conditions is generally a challenging
task since a real hardware test setup is required. Ideally, the capture
method should not introduce any further degradation. The only
requirement on capture method is that the captured sequences conform to
the file requirements in section 6.1.6 and 7.2.

For applications including radio transmissions, one possibility is to
use a laptop with e.g. a built-in 3G network card and to download
streams from a server through a radio network. Another possibility is
the use of drive test tools and to simulate a video phone call while the
car is driving. In order to simulate very bad radio coverage, the
antenna may be wrapped with some aluminum foil (Editors note: This
strictly a simulation again, but for the sake of simplicity it can be
accepted since the simulated bad coverage is overlayed with the effects
from the live network).

In order to prepare the PVSs the same rules apply as for simulated
network conditions. The only difference is the network used for the
transmission.

### Pausing with Skipping and Pausing without Skipping

Pausing without skipping events will not be included in the current
testing.

Pausing with skipping events will be included in the current testing.
Anomalous frame repetition is not allowed during the first 1s or the
final 1s of a video sequence. Note that where pausing with skipping and
anomalous frame repetition is included in a test then source material
containing still sections should form part of the testing.

If it is difficult or impossible to determine whether a video sequence
contains pausing without skipping or pausing with skipping, the video
sequence will be given the benefit of doubt and considered to contain
pausing with skipping.

(See section 2 for definitions of "pausing with skipping", "pausing
without skipping" and "anomalous frame repetition".)

### Frame Rates

For those codecs that only offer automatically set frame rate, this rate
will be decided by the codec. Some codecs will have options to set the
frame rate either automatically or manually. For those codecs that have
options for manually setting the frame rate (and we choose to set it for
the particular case), 5 fps will be considered the minimum frame rate
for VGA and CIF, and 2.5 fps for PDA/Mobile..

Manually set frame rates (constant frame rate) may include:

-   PDA/Mobile: 30, 25, 15, 12.5, 10, 8, 5, 2.5 fps

-   PC1 (CIF): 30, 25, 15, 12.5, 10, 8, 5 fps

-   PC2 (VGA): 30, 25, 15, 12.5, 10,8, 5 fps

Variable frame rates are acceptable for the HRCs

Care must be taken when creating test sequences for display on a PC
monitor. The refresh rate can influence the reproduction quality of the
video and VQEG MM requires that the sampling rate and display output
rate are compatible. For example,

Given a source frame rate of video is 30fps, the sampling rate is 30/X
(e.g. 30/2 = sampling rate of 15fps). This is called frame rate. Then we
upsample and repeat frames from the sampling rate of 15fps to obtain 30
fps for display output.

\[Ed Note: VQEG MM must agree on a scan rate for PC monitors prior to
test (e.g. 50Hz, 60Hz, 75Hz, etc).\]

The intended frame rate of the source and the PVS must be identical.

### Pre-Processing

The HRC processing may include, typically prior to the encoding, one or
more of the following:

-   Filtering

-   Simulation of non-ideal cameras (e.g. mobile)

-   Colour space conversion (e.g. from 4:2:2 to 4:2:0)

This processing will be considered part of the HRC.

### Post-Processing

The following post-processing effects may be used in the preparation of
test material:

-   Colour space conversion

-   De-blocking

-   Decoder jitter

### Coding Schemes

Coding Schemes that will be used may include, but are not limited to:

-   Windows Media Player 9

-   H.263

-   H.264 (MPEG-4 Part 10)

-   Real Video (e.g. RV 10)

-   MPEG 4

### Distribution of Tests over Facilities

### Processing and Editing Sequences

Test sequences will be captured from the decoded video in uncompressed
format. Two capture methods may be employed. The two methods are as
follows:

The captured video file should be in AVI container.

### Randomization

### Presentation Structure of Test Material

# Objective Quality Models

## Model Type

VQEG MM has agreed that Full Reference, Reduced Reference and No
Reference models may be submitted for evaluation. The side-channels
allowable for the RR models are:

-   PDA/Mobile (QCIF): (1k, 10k)

-   PC1 (CIF): (10k, 64k)

-   PC2 (VGA): (10k, 64k, 128k)

Proponents may submit one model of each type for all image size
conditions. Thus, any single proponent may submit up to a total of 13
different models. Note that where multiple models are submitted,
additional model submission fees may apply.

## Model Input and Output Data Format

The progressive video format will be used in the multimedia test

The model will be given a ASCII file listing pairs of video sequence
files to be processed. Each line of this file has the following format:

\<source-file\> \<processed-file\>

where \<source-file\> is the name of a source video sequence file and
\<processed-file\> is the name of a processed video sequence file, whose
format is specified in section 6.1.6 of this document. For no-reference
models only a processed file will be available. File names may include a
path. For example, an input file for the 525 cases might contain the
following:

\[Ed Note: file names below will be corrected for MM\]

**/video/V2src1_525.yuv /video/V2src1_hrc2_525.yuv**

**/video/V2src1_525.yuv /video/V2src1_hrc1_525.yuv**

**/video/V2src2_525.yuv /video/V2src2_hrc1_525.yuv**

**/video/V2src2_525.yuv /video/V2src2_hrc2_525.yuv**

The RR model the first column will be the name of the parameter file.
\[Ed Note: The exact conditions for running the RR models needs to be
worked out. Check the RRNR-TV test plan\] The output file is an ASCII
file created by the model program, listing the name of each processed
sequence and the resulting Video Quality Rating (VQR) of the model. The
contents of the output file should be saved to disc after each sequence
is processed, to allow the testing laboratories the option of halting a
processing run at any time. Each line of the ASCII output file has the
following format:

\<processed-file\> VQR

Where \<processed-file\> is the name of the processed sequence run
through this model, without any path information. VQR is the Video
Quality Ratings produced by the objective model. For the input file
example, this file contains the following:

**V2src1_hrc2_525.yuv 0.150**

**V2src1_hrc1_525.yuv 1.304**

**V2src2_hrc1_525.yuv 0.102**

**V2src2_hrc2_525.yuv 2.989**

Each proponent is also allowed to output a file containing Model Output
Values (MOVs) that the proponents consider to be important. The format
of this file will be

**V2src1_hrc2_525.yuv 0.150 MOV~1~** **MOV~2~,... MOV~N~**

**V2src1_hrc1_525.yuv 1.304 MOV~1~** **MOV~2~,... MOV~N~**

**V2src2_hrc1_525.yuv 0.102 MOV~1~** **MOV~2~,... MOV~N~**

**V2src2_hrc2_525.yuv 2.989 MOV~1~** **MOV~2~,... MOV~N~**

## Submission of Executable Model

For each video format (QCIF, CIF, and VGA), a set of 2 source and
processed video sequence pairs will be used as test vectors. They will
be available for downloading on the VQEG web site
<http://www.vqeg.org/>.

Each proponent will send an executable of the model and the test vector
outputs to the ILG by the date specified in action item "Final
submission of executable model" of Section 5.3. The executable version
of the model must run correctly on one of the two following computing
environments:

-   SUN SPARC workstation running the Solaris 2.3 UNIX operating system
    (SUN OS 5.5). \[Ed. Note: The used of SUN workstation should be
    agreed\]

-   WINDOWS 2000 workstation and Windows XP.

The use of other platforms will have to be agreed upon with the
independent laboratories prior to the submission of the model.

The independent laboratories will verify that the software produces the
same results as the proponent with a maximum error of 0.1% for a
deterministic model. A maximum of 5 randomly selected files will each
model will be used for verification. If greater errors are found, the
independent and proponent laboratories will work together to correct
them. If the errors cannot be corrected, then the ILG will review the
results and recommend further action.

## Registration

\[Ed Note: All the bullet points below in Section 7.4 (between here and
the next Ed Note) have not been decided for the MM testplan and are not
subject to the 2/3 majority rule. NTIA, Opticom, SwissQual, NTT will
investigate the variations for MM to get proper values in the above
section, by June 30, 2005\]

Models must include calibration and registration is required to handle
the following technical criteria:

-   maximum allowable deviation in *Peak Video Level* is +/- 10%

-   maximum allowable deviation in *Black Level* is +/- 10%

-   maximum allowable *Horizontal Shift* is +/- 1 pixels

-   maximum allowable *Vertical Shift* is +/- 1 lines

-   maximum allowable *Horizontal Cropping* is 30 pixels

-   maximum allowable *Vertical Cropping* is 20 pixels

-   no *Vertical or Horizontal Re-scaling* is allowed

-   *Temporal Alignment* between SRC and HRC sequences shall be
    maintained to within +/- 2 video frames.

-   no *Chroma Differential Timing* is allowed

-   no *Picture Jitter* is allowed.

\[Ed Note: All bullet points above in Section 7.4 have not been decided
for the MM testplan and is not subject to the 2/3 majority rule. NTIA,
Opticom, SwissQual, NTT will investigate the variations for MM to get
proper values in the above section, by June 30, 2005\]

No Reference Models should not need calibration

Reduced Reference Models must include temporal registration if the model
needs it. Temporal misalignment of no more than +/-0.25s is allowed.
Please note that in subjective tests, the start frame of both the
reference and its associated HRCs are matched as closely as possible.
Spatial offsets are expected to be very rare. It is expected that no
post-impairments are introduced to the outputs of the encoder before
transmission. Spatial registration will be assumed to be within (1)
pixel. Gain, offset, and spatial registration will be corrected, if
necessary, to satisfy the calibration requirements specified in this
test plan.

\[Ed Note: The exact allowable misalignment value will be examined for
the best value. The text needs to made clearer.\]

The organizations responsible for creating the PVSs shall check that
they fall within the specified calibration and registration limits. The
PVSs will be double-checked by one other organization. After testing has
been completed any PVS found to be outside the calibration limits shall
be removed from the data analyzes. ILG will decide if a suspect PVS is
outside the limits.

# Objective Quality Model Evaluation Criteria

This paragraph describes the evaluation metrics and procedure used to
assess the performances of an objective video quality model as an
estimator of video picture quality in a variety of applications.

## Evaluation Procedure

The performance of an objective quality model is characterized by three
prediction attributes: accuracy, monotonicity and consistency.

The statistical metrics root mean square (rms) error, Pearson
correlation, and outlier ratio together characterize the accuracy,
monotonicity and consistency of a model's performance. The calculation
of each statistical metric is performed along with its 95% confidence
intervals. To test for statistically significant differences among the
performance of various models, the F-test will be used.

The statistical metrics are calculated using the objective model outputs
and the results from viewer subjective rating of the test video clips.
The objective model provides a single number (figure of merit) for every
tested video clip. The same tested video clips get also a single
subjective figure of merit. The subjective figure of merit for a video
clip represents the average value of the scores provided by all subjects
viewing the video clip.

Objective models cannot be expected to account for (potential)
differences in the subjective scores for different viewers or labs. Such
differences, if any, will be measured, but will not be used to evaluate
a model's performance. "Perfect" performance of a model will be defined
so as to exclude the residual variance due to within-viewer,
between-viewer, and between-lab effects

The evaluation analysis is based on DMOS scores for the FR and RR
models, and on MOS scores for the NR model. Discussion below regarding
the DMOS scores should be applied identically to MOS scores. For
simplicity, only DMOS scores are mentioned for the rest of the chapter.

The objective quality model evaluation will be performed in three steps.
The first step is a monotonic rescaling of the objective data to better
match the subjective data. The second calculates the performance metrics
for the model and their confidence intervals. The third tests for
differences between the performances of different models using the
F-test.

## Data Processing

### Mapping to the Subjective Scale

Subjective rating data often are compressed at the ends of the rating
scales. It is not reasonable for objective models of video quality to
mimic this weakness of subjective data. Therefore, in previous video
quality projects VQEG has applied a non-linear mapping step before
computing any of the performance metrics. A non-linear mapping function
that has been found to perform well empirically is (1)

$\text{DMOSp} = \frac{b1}{1 + e^{- b2 \ast (\text{VQR} - b3)}}$ (1)

where DMOSp is the predicted DMOS, and VQR is the model's computed value
for a clip-HRC combination. The parameters b1, b2, b3 are found from
fitting the function to the data \[DMOS, VCR\]. This non-linear mapping
procedure will be applied to each model's outputs before the evaluation
metrics are computed.

If the logistic rescaling does not properly converge for all the models,
then a cubic polynomial monotonic rescaling will be applied to all the
models.

### Averaging Process

Primary analysis of model performance will be calculated per processed
video sequence. Secondary analysis of model performance may be
calculated and reported on (1) averaged data, by averaging all SRC
associated with each HRC (DMOS~H~), and on (2) averaged data, by
averaging all HRC associated with each SRC (DMOS~S~).

[]{#__RefHeading___Toc101866095 .anchor}Aggregation Procedure

The evaluation of the objective metrics is performed in two steps. In
the first step, the objective metrics are evaluated per experiment. In
this case, the evaluation/statistical metrics are calculated for all
tested objective metrics. A comparison analysis is then performed based
on significance tests. In the second step, an aggregation of the
performance results is considered. The aggregation will be performed by
taking the average values for all three evaluation metrics for all
experiments (see section 8.3).

Ed Note: return to discuss normalization tomorrow (Wed)

## Evaluation Metrics

Once the mapping has been applied to objective data, the three
evaluation metrics: root mean square error, Pearson correlation
coefficient and outlier ratio are determined. The calculation of each
evaluation metric is performed along with its 95% confidence interval.

### Pearson Correlation Coefficient

The Pearson correlation coefficient R (see Equation 2) measures the
linear relationship between a model's performance and the subjective
data. Its great virtue is that it is on a standard, comprehensible scale
of -1 to 1 and it has been used frequently in similar testing.

X

$R = \frac{\sum_{i = 1}^{N}{(\text{Xi} - \overline{X}) \ast (\text{Yi} - \overline{Y})}}{\sqrt{\sum_{}^{}{(\text{Xi} - \overline{X)}}^{2}} \ast \sqrt{\sum_{}^{}{(\text{Yi} - \overline{Y)}}^{2}}}$
(2)

Ed Note: Equation needs to be corrected.

Xi denotes the subjective score DMOS and Yi the objective DMOSp one. N
represents the total number of video samples considered in the analysis.

It is known \[1\] that the statistic z (3) is approximately normally
distributed and its standard deviation is defined by (4). Equation (3)
is called Fisher-z transformation.

$z = 1\text{.}\text{1513} \cdot \text{log}_{\text{10}}\left( \frac{1 + R}{1 - R} \right)$
(3)

$\sigma_{z} = \sqrt{\frac{1}{N - 3}}$ (4)

The 95% confidence interval for the correlation coefficient is
determined using one tailed t-Student distribution with t=1.64 and it is
given by (5)

$z \pm 1\text{.}\text{64} \ast \sigma$ (5)

**NOTE.** If more than N\>30 samples are used, then the Gausssian
distribution can be used instead of the t-Student distribution and
therefore t=1.64 is replaced by the normal distribution score zc=2
\[1\].

Ed Note: Text to be switched such that CI described for N\>30 and the
note at the end is for N\<30.

### Root Mean Square Error

The accuracy of the objective metric is evaluated using the root mean
square error (rmse) evaluation metric.

The difference between measured and predicted DMOS is defined as the
absolute prediction error Perror (6)

$\text{Perror}(i) = \text{DMOS}(i) - \text{DMOS}_{p}(i)$ (6)

where the index *i* denotes the video sample.

The root-mean-square error of the absolute prediction error Perror is
calculated with the formula (7)

$\text{rmse} = \sqrt{\left( \frac{1}{N - d}\sum_{N}^{}{\text{Perror}\lbrack i\rbrack ²} \right)}$
(7)

Ed. Note: Check this equation is correct.

Where N denotes the number of samples and d the number of degrees of
freedom of the mapping function (1).

The root mean square error is approximately characterized by a χ\^2 (n)
\[1\], where n represents the degrees of freedom and it is defined by
(8)

$n = N - 1$ (8)

where N represents the total number of samples.

Using the χ\^2 (n) distribution, the 95% confidence interval for the
rmse is given by (9) \[1\]

$\frac{\text{rmse} \ast \sqrt{N}}{\chi_{0\text{.}\text{95}}^{2}(N - 1)} < \text{rmse} < \frac{\text{rmse} \ast \sqrt{N}}{\chi_{0\text{.}\text{05}}^{2}(N - 1)}$
(9)

Ed. Note: Correct equation.

Outlier Ratio

The consistency attribute of the objective metric is evaluated by the
outlier ratio OR which represents number of "outlier-points" to total
points N.

$\text{OR} = \frac{\text{TotaNoOutliers}}{N}$ (10)

where an outlier is a point for which

> $\left| \text{Perror}(i) \right| > 2 \ast \sigma(\text{DMOS}(i)\frac{)}{N}$
> (11)

Ed. Note: Correct equation to st error. Remove indent from Eq 11

where σ(DMOS(i)) represents the standard deviation of the individual
scores associated with the video clip i. The individual scores are
approximately normally distributed and therefore twice the σ value
represents the 95% confidence interval. Thus, 2 \* σ(DMOS(i))value
represents a good threshold for defining an outlier point.

The outlier ratio represents the proportion of outliers in N number of
samples. Thus, the binomial distribution could be used to characterize
the outlier ratio. The outlier ratio is represented by a distribution of
proportions \[1\] characterized by the mean (12) and standard deviation
(13)

$p = \frac{\text{TotalNoOutliers}}{N}$ (12)

$\sigma_{p} = \sqrt{\frac{p \ast (1 - p)}{N}}$ (13)

Thus, using the one tailed t-Student distribution, the 95% confidence
interval of the outlier ratio is given by (14)

$\pm 1\text{.}\text{64} \ast \sigma_{p}$ (14)

**NOTE.** If more than N\>30 samples are used, then the Gausssian
distribution can be used instead of the t-Student distribution and
therefore t=1.64 is replaced by normal distribution score zc=2 \[1\].

**Ed. Note: switch the CI** **to detail Gaussian and note t.**

## Statistical Significance of the Results

### Significance of the Difference between the Correlation Coefficients

The test is based on the assumption that the normal distribution is a
good fit for the video quality scores' populations. The statistical
significance test for the difference between the correlation
coefficients uses the H~0~ hypothesis that assumes that there is no
significant difference between correlation coefficients. The H~1~
hypothesis considers that the difference is significant, although not
specifying better or worse.

The test uses the Fisher-z transformation (3) \[1\]. The normally
distributed statistic (15) \[1\] is determined for each comparison and
evaluated against the 95% t-Student value for the two--tail test, which
is the tabulated value t(0.05) =1.96.

  ------------------------------------------------------------------ --------
  $$Z_{N} = \frac{z1 - z2 - \mu_{(z1 - z2)}}{\sigma_{(z1 - z2)}}$$   \(15\)

  where $\mu_{(z1 - z2)} = 0$                                        \(16\)

  and                                                                \(17\)
  $\sigma_{(z1 - z2)} = \sqrt{\sigma_{z1}^{2} + \sigma_{z2}^{2}}$    
  ------------------------------------------------------------------ --------

σ~z1~ and σ~z2~ represent the standard deviation of the Fisher-z
statistic for each of the compared correlation coefficients. The mean
(16) is set to zero due to the H~0~ hypothesis and the standard
deviation of the difference metric z1-z2 is defined by (17). The
standard deviation of the Fisher-z statistic is given by (18):

  ----------------------------------------------------------------- --------
  $$\sigma_{z} = \sqrt{1(N - 3)}$$                                  \(18\)

  ----------------------------------------------------------------- --------

where N represents the total number of samples used for the calculation
of each of the two correlation coefficients.

### Significance of the Difference between the Root Mean Square Errors

Considering the same assumption that the two populations are normally
distributed, the comparison procedure is similarly to the one used for
the correlation coefficients. The H~0~ hypothesis considers that there
is no difference between rmse values. The alternative H~1~ hypothesis is
assuming that the lower prediction error value is statistically
significantly lower. The statistics defined by (19) has a F-distribution
with n1 and n2 degrees of freedom \[1\].

  ----------------------------------------------------------------------- --------
  $$\zeta = \frac{\text{rmse}_{\text{max}}}{\text{rmse}_{\text{min}}}$$   \(19\)

  ----------------------------------------------------------------------- --------

Ed. Note: Need to change Peavg to rmse

PEavg,max is the highest rmse and PEavg,min is the lowest rmse involved
in the comparison. The ζ statistic is evaluated against the tabulated
value F(0.05, n1, n2) that ensures 95% significance level. The n1 and n2
degrees of freedom are given by N1-1, respectively and N2-1, with N1 and
N2 representing the total number of samples for the compared average
prediction errors.

### Significance of the Difference between the Outlier Ratios

The significance test in this case is identical with the one for the
correlation coefficients, with the modification that the standard
deviation of the z statististic (18) becomes (20)

$\sigma_{p1 - p2} = \sqrt{p \ast (1 - p) \ast (\frac{1}{N1} + \frac{1}{N2}})$
(20)

where N1 and N2 represent the total number of samples of the compared
outlier ratios p1 versus p2. The variable p is defined by (21)

$p = \frac{N1 \ast p1 + N2 \ast p2}{N1 + N2}$ (21)

Ed. Note: check df attached to these equations.

References

\[1\] M. Spiegel, "Theory and problems of statistics", McGraw Hill,
1998.

# Recommendation

The VQEG will recommend methods of objective video quality assessment
based on the primary evaluation metrics defined in Section 8. The Study
Groups involved (ITU-T SG 12, ITU-T SG 9, and ITU-R SG 6) will make the
final decision(s) on ITU Recommendations.

[]{#__RefHeading___Toc101866115 .anchor}Bibliography

-   VQEG Phase I final report.

-   VQEG Phase I Objective Test Plan.

-   VQEG Phase I Subjective Test Plan.

-   VQEG FR-TV Phase II Test Plan.

-   Vector quantization and signal compression, by A. Gersho and R. M.
    Gray. Kluwer Academic Publisher, SECS159, 0-7923-9181-0.

-   Recommendation ITU-R BT.500-10.

-   document 10-11Q/TEMP/28-R1.

-   RR/NR-TV Test Plan

Annex I\
Instructions to the Subjects

Notes: The items in parentheses are generic sections for a Subject
Instructions Template. They would be removed from the final text. Also,
the instructions are written so they would be read by the experimenter
to the participant(s).

(*greeting*) Thanks for coming in today to participate in our study. The
study's about the quality of video images; it's being sponsored and
conducted by companies that are building the next generation of video
transmission and display systems. These companies are interested in what
looks good to you, the potential user of next-generation devices.

(*vision tests*) Before we get started, we'd like to check your vision
in two tests, one for acuity and one for color vision. (*These tests
will probably differ for the different labs, so one common set of
instructions is not possible.*)

(*overview of task: watch, then rate*) What we're going to ask you to do
is to watch a number of short video sequences to judge each of them for
"quality" \-- we'll say more in a minute about what we mean by
"quality." These videos have been processed by different systems, so
they may or may not look different to you. We'll ask you to rate the
quality of each one after you've seen it.

(*physical setup*) When we get started with the study, we'd like you to
sit here (point) and the videos will be displayed on the screen there.
You can move around some to stay comfortable, but we'd like you to keep
your head reasonably close to this position indicated by this mark
(point to mark on table, floor, wall, etc.). This is because the videos
might look a little different from different positions, and we'd like
everyone to judge the videos from about the same position. I (the
experimenter) will be over there (point).

(*room & lighting explanation, if necessary*) The room we show the
videos in, and the lighting, may seem unusual. They're built to satisfy
international standards for testing video systems.

(*presentation timing and order; number of trials, blocks*) Each video
will be (*insert number*) seconds (minutes) long. You will then have a
short time to make your judgment of the video's quality and indicate
your rating. At first, the time for making your rating may seem too
short, but soon you will get used to the pace and it will seem more
comfortable. (*insert number*) video sequences will be presented for
your rating, then we'll have a break. Then there will be another similar
session. All our judges make it through these sessions just fine.

(*what you do: judging \-- what to look for*) Your task is to judge the
quality of each image \-- not the [content]{.underline} of the image,
but how well the system displays that content for you. The images come
in three different sizes; how you judge image quality for the different
sizes is up to you. There is no right answer in this task; just rely on
your own taste and judgment.

(*what you do: rating scale; how to respond, assuming presentation on a
PC*) After [judgin]{.underline}*g* the quality of an image, please
[rate]{.underline} the quality of the image. Here is the rating scale
we'd like you to use (*also have a printed version, either hardcopy or
electronic*):

5 Excellent

4 Good

3 Fair

2 Poor

1 Bad

Please indicate your rating by pushing the appropriate numeric key on
the keyboard (button on the screen). If you missed the scene and have to
see it again, press the XXX key. If you push the wrong key and need to
change your answer, press the YYY key to erase the rating; then enter
your new rating. \[Note, this assumes that a program exists to put a
graphical user interface (GUI) on the computer screen between video
presentations. It should feed back the most recent rating that the
subject had input, should have a "next video" button and an "erase
rating" button. It should also show how far along in the sequence of
videos the session is at present. The program that randomly chooses
videos for presentation, records the data, and contains the GUI, should
be written in a language that is compatible with the most commonly used
computers.\]

(*practice trials: these should include the different size formats and
should cover the range of likely quality*) Now we will present a few
practice videos so you can get a feel for the setup and how to make your
ratings. Also, you'll get a sense of what the videos are going to be
like, and what the pace of the experiment is like; it may seem a little
fast at first, but you get used to it.

(*questions*) Do you have any questions before we begin?

(*subject consent form, if applicable; following is an example*)

The Multimedia Quality Experiment is being conducted at the (*name of
your lab*) lab. The purpose, procedure, and risks of participating in
the Multimedia Quality Experiment have been explained to me. I
voluntarily agree to participate in this experiment. I understand that I
may ask questions, and that I have the right to withdraw from the
experiment at any time. I also understand that (*name of lab*) lab may
exclude me from the experiment at any time. I understand that any data I
contribute to this experiment will not be identified with me personally,
but will only be reported as a statistical average.

Signature of participant Signature of experimenter

Name of participant Date Name of experimenter

Annex II\
Example EXCEL Spreadsheet

  --------- ---------- ------------- ----------- ----------- --------- ---------- ------------- ---------------- ---------- --------- ------------ ----------- ----------- ----------- ---------
  **lab**   **test**   **type**      **subject   **month**   **day**   **Year**   **session**   **resolution**   **rate**   **age**   **gender**   **order**   **scene**   **hrc**     **acr
                                     \#**                                                                                                                                              score**

  ntia      mm1        compression   1000        10          3         2005       1             vga              30         47        m            1           susie       hrc1        4

  ntia      mm1        compression   1000        10          3         2005       1             vga              30         47        m            1           susie       hrc2        2

  ntia      mm1        compression   1000        10          3         2005       1             vga              30         47        m            1           susie       hrc3        1

  ntia      mm1        compression   1000        10          3         2005       1             vga              30         47        m            1           susie       reference   5

  ntt       mm2        robust        2003        10          18        2005       2             cif              25         38        f            2           calmob      pktloss1    1

  ntt       mm2        robust        2003        10          18        2005       2             cif              25         38        f            2           calmob      pktloss2    2

  ntt       mm2        robust        2003        10          18        2005       2             cif              25         38        f            2           calmob      biterror1   1

  ntt       mm2        robust        2003        10          18        2005       2             cif              25         38        f            2           calmob      biterror2   3

  ntt       mm2        robust        2003        10          18        2005       2             cif              25         38        f            2           calmob      reference   4

  yonsei    mm3        livenetwork   3018        10          21        2005       1             qcif             30         27        m            1           football    ip1         4

  yonsei    mm3        livenetwork   3018        10          21        2005       1             qcif             30         27        m            1           football    ip2         3

  yonsei    mm3        livenetwork   3018        10          21        2005       1             qcif             30         27        m            1           football    reference   5
  --------- ---------- ------------- ----------- ----------- --------- ---------- ------------- ---------------- ---------- --------- ------------ ----------- ----------- ----------- ---------

Annex III\
Background and Guidelines on Transmission Errors

Ed. Note: include Jorgens emails here as Annex III:All of the contents
of the email is not agreed to and should not be subject to the 2/3 rule
for editing.

[^1]: Payment will be made directly from each proponent to the selected
    testing facility, according to a table agreed on by ILG and
    distributed to the proponents.

[^2]: Note that the term 'block' does not refer to a visual degradation
    such as blocking errors (or blockiness) but refers to errors in the
    transport stream (transport blocks).