HDTV Group

TEST PLAN

Draft Version 0.1

November 09, 2004

Contact: Vivaik Balasubrawmanian Tel: +1 (480) 554 1905

E-Mail: vivaik.balasubrawmanian@intel.com

Editorial History

+---------+-------------+----------------------------------------------+
| **Ve    | **Date**    | **Nature of the modification**               |
| rsion** |             |                                              |
+---------+-------------+----------------------------------------------+
| 0.0     | November 1, | Initial Draft, edited by Vivaik              |
|         | 2004        | Balasubrawmanian                             |
+---------+-------------+----------------------------------------------+
| 0.1     | November 9, | Incorporated the following changes from NTIA |
|         | 2004        | (Margaret Pinson):                           |
|         |             |                                              |
|         |             | -   Added an editor's note to highlight the  |
|         |             |     unapproved status.                       |
|         |             |                                              |
|         |             | -   Removed references to future test plans  |
|         |             |     (AV & Interactive)                       |
|         |             |                                              |
|         |             | -   Replaced ACR-HRR with DSIS subjective    |
|         |             |     testing methodology                      |
|         |             |                                              |
|         |             | -   Removed redundant sections               |
|         |             |                                              |
|         |             | -   Minimum bit rate for HRCs is now 2       |
|         |             |     Mbits/s.                                 |
|         |             |                                              |
|         |             | -   Replaced inconsistent section on         |
|         |             |     Calibration/Registration with the latest |
|         |             |     text from RRNR test plan.                |
|         |             |                                              |
|         |             | -   Removed evaluation metrics in line with  |
|         |             |     the agreements reached in the Seoul MM   |
|         |             |     meeting.                                 |
+---------+-------------+----------------------------------------------+
|         |             |                                              |
+---------+-------------+----------------------------------------------+
|         |             |                                              |
+---------+-------------+----------------------------------------------+
|         |             |                                              |
+---------+-------------+----------------------------------------------+

**Summary**

List of Acronyms

1\. Introduction 6

2\. Subjective Evaluation Procedure 7

2.1.The \<\<XXX\>\> method 7

2.1.1.General description 7

2.1.2.Viewing conditions 7

2.1.3.Instructions to viewers for quality tests 7

2.1.4.Viewers 7

2.2.Data format 7

2.2.1.Results data format 7

2.2.2.Subject data format 7

2.2.3.Subjective Data analysis 7

3.Test Laboratories 7

3.1.ILG 7

3.2.Proponent Laboratories 7

4.Sequence processing and data formats 8

4.1.Sequence processing overview 8

4.2.Test materials 8

4.2.1.Selection of test material (SRC) 8

4.2.2.Hypothetical reference circuits (HRC) 8

4.2.3.Distribution of tests over facilities 8

4.2.4.Processing and editing sequences 8

4.2.5.Randomization 8

4.2.6.Presentation structure of test material 8

5.Objective Quality Models 8

5.1.Model type 8

5.2.Model input and output data format 8

5.3.Submission of executable model 8

5.4.Registration 9

5.5.Results analysis 9

6\. Objective quality model evaluation criteria 10

6.1.Introduction to evaluation metrics 10

6.2.Evaluation Metrics 11

6.2.1.Metrics relating to Prediction Accuracy of a model 12

6.2.2.Metrics relating to Prediction Monotonicity of a model 12

6.2.3.Metrics relating to Prediction Consistency of a model 12

6.2.4.Metrics relating to agreement 13

6.3.Generalizability 13

6.4.Complexity 14

7.Calendar and actions 14

8\. Recommendation 14

9.Bibliography 15

10.Annex: Optimum mean square quantization method. 15

10.1.The Lloyd Iteration for codebook improvement 15

10.2.The Lloyd algorithm 16

List of Acronyms

ACR-HRR Absolute Category Rating with hidden reference removal

ANOVA ANalysis Of VAriance

ASCII ANSI Standard Code for Information Interchange

CCIR Comite Consultatif International des Radiocommunications

CODEC Coder-Decoder

CRC Communications Research Center (Canada)

DVB-C Digital Video Broadcasting-Cable

FR Full Reference

GOP Group of Pictures

HRC Hypothetical Reference Circuit

IRT Institut Rundfunk Technische (Germany)

ITU International Telecommunications Union

MM multimedia

MOS Mean Opinion Score

MOSp Mean Opinion Score, predicted

MPEG Motion Pictures Expert Group

NR No (or Zero) Reference)

NTSC Nat'l Television Standard Code (60 Hz TV)

PAL (50 Hz TV)

PS Program Segment

QAM Quadrature Amplitude Modulation

QPSK Quadrature Phase Shift Keying

RR Reduced Reference

SMPTE Society of Motion Picture and Television Engineers

SRC Source Reference Channel or Circuit

SSCQE Single Stimulus Continuous Quality Evaluation

VQEG Video Quality Experts Group

VTR Video Tape Recorder

#  Introduction

This document defines the procedure for evaluating the performance of
objective perceptual quality models submitted to the Video Quality
Expert Group (VQEG) formed from experts of ITU-T Study Groups 9 and 12
and ITU-R Study Group 6. It is based on discussions from various
meetings of the VQEG HDTV working group (HDTV).

The goal of the HDTV group is to recommend a quality model suitable for
application to digital video quality measurement in HDTV applications.
HDTV in this context is defined as being of or relating to an
application that creates / consumes High Definition television broadcast
that is digitally transmitted over a communication channel. Common
applications of HDTV that are appropriate to this study include
Television broadcasting, video on demand and satellite / cable
transmissions. The measurement tools recommended by the HDTV group will
be used to measure quality both in laboratory conditions using a FR
method and in operational conditions using RR/NR methods.

In the first stage of testing, it is proposed that video only test
conditions will be employed. It should be noted that presently there is
a lack of HDTV test material for use in testing. The VQEG does desire to
have copyright free (or at least free for research purposes) material
for testing. The capability of the group to perform adequate audio-video
and multimedia testing is dependent on access to a bank of potential
test sequences.

The performance of objective models will be based on the comparison of
the DMOS obtained from controlled subjective tests and the DMOSp
predicted by the submitted models. This testplan defines the test method
or methods, selection of test material and conditions, evaluation
metrics to examine the predictive performance of competing objective
multimedia quality models.

The goal of the testing is to examine the performance of proposed video
quality metrics across representative transmission and display
conditions. To this end, the tests will enable assessment of models for
cable/satellite/terrestrial transmissions and broadband communications
services. The Recommendation(s) resulting from the VQEG MM testing will
be deemed appropriate for services delivered on computer desktop
monitors and high definition display television technologies.

It is expected that subjective tests will be performed separately for
different display conditions (e.g. one specific test for LCD TVs;
another test for desktop computer monitor). The performance of submitted
models will be evaluated for each type of display condition. Therefore
it may be possible for one model to be recommended for one display type
(say, computer monitor) and another model for another display format
(say, plasma television).

The objective models will be tested using a set of digital video
sequences selected by the VQEG HDTV group. The test sequences will be
processed through a number of hypothetical reference circuits (HRC\'s).
The quality predictions of the submitted models will be compared with
subjective ratings from human viewers of the test sequences as defined
by this Test Plan.

A final report will be produced after the analysis of test results.

#  Subjective Evaluation Procedure

## The DSIS method

This section describes the test method according to which the VQEG HDTV
subjective tests will be performed. We will use the DSIS \[ITU-R Rec.
BT.500-11\].

### General description

The selected test methodology is the Double-Stimulus Impairment Scale
(DSIS) method. This choice has been selected due to the fact that DSIS
provides a reliable and standardized method (ITU-R Rec. 500-11, ITU-T
P.910) that allows a large number of test conditions to be assessed in
any single test session.

In the DSIS test method, each test condition is presented once only for
subjective assessment. The test presentation order is randomized
according to standard procedures (e.g. Latin or Graeco-Latin square).
The test format is cyclic, in that the subject is first presented with
the reference video sequence, then with the same video sequence
impaired. Following this, the subject is asked to vote on the second,
keeping in mind the first. Subjective ratings are reported on the
five-grade impairment scale:

5.  imperceptible

```{=html}
<!-- -->
```
4.  perceptible, but not annoying

```{=html}
<!-- -->
```
3.  slightly annoying

```{=html}
<!-- -->
```
2.  annoying

> 1 very annoying

### Application Across Different Video Formats and Displays

The proposed HDTV test will examine the performance of objective
perceptual quality models for different video formats (720p, 1080i and
1080p). Section 2.1.3 defines format and display types in detail. Video
applications targeted in this test include internet video on demand,
HDTV broadcasts, etc.

The instructions given to subjects will request subjects to maintain a
specified viewing distance from the display device. The viewing distance
has been agreed as:

-   720p: 4H.

-   1080i: 4H.

-   1080p: 4H

H=Picture Heights (picture is defined as the size of the video window)

### Display Specification and Set-up

Given that the subjective tests will use different HD display
technologies, it is necessary to ensure that each test laboratory
selects appropriate display specification and common set-up techniques
are employed. The following display technologies should be covered by
the tests :

1.  Professional grade Computer Monitor (eg : Viewsonic Professional
    series P225f)

2.  Consumer grade LCD HD TV.

3.  Consumer grade Plasma HD Television.

4.  Consumer grade Rear Projection HD Television

5.  Consumer grade DLP HD Television.

6.  Consumer grade HD overhead projectors.

> (XXX Note : Should we included EDTVs ?)

### Subjects

Each test will require at least 24 subjects. It is recommended that as
many subjects as possible participate in each test in order to improve
the statistical power of the resulting data. It is preferred that each
subject be given a different ordering of video sequences where possible.
Otherwise, the viewers will be assigned to sub-groups, which will see
the test sessions in different orders. At least two different orderings
of test sequences are required per subjective test.

Only non-expert viewers \[Ed. Note: Definition of "non-expert viewer" is
needed. P. Corriveau agreed to provide this prior to the next VQEG
meeting\] will participate. The term non-expert is used in the sense
that the viewers' work does not involve video picture quality and they
are not experienced assessors. They must not have participated in a
subjective quality test over a period of six months. All viewers will be
screened prior to participation for the following:

-   normal (20/20) visual acuity with or without corrective glasses (per
    Snellen test or equivalent).

-   normal colour vision (per Ishihara test or equivalent).

-   familiarity with the language sufficient to comprehend instruction
    and to provide valid responses using semantic judgement terms
    expressed in that language.

Note; for any test involving audio, appropriate screening for normal
hearing should be applied (following relevant audio test
recommendations)\[e.g., P.800, BS-1116\].

### Viewing Conditions

Each test session will involve only one subject assessing the test
material. Subjects will be seated directly in line with the centre of
the video display at the appropriate viewing distance. The test cabinet
will conform to ITU-T Rec. P.910 requirements.

### Test data collection

The collection and organization of the data files containing the votes
will be under the direct responsibility of the ILG Chair.

## Data Format

### Results Data Format

### Subject Data Format

### Subjective Data Analysis

Each subject\'s results will be checked for completeness. An observer is
discarded if the number of failed votes exceeds one in one of the
sessions. Additionally, the observers will be screened after the test as
specified in sec. 2.3.1 of Annex 2 "Screening for DSIS, DSCQS and
alternative methods except SSCQE method" of recommendation ITU-R
BT.500-10. The post-test screening will [not]{.underline} be applied to
these sub-groups but to the groups which participate in one test as a
whole.

# Test Laboratories and Schedule

Given the scope of the HDTV testing, it is proposed that both
independent test laboratories and proponent laboratories will be given
subjective test responsibilities. The ILG will oversee the testing to
ensure that all laboratories conform to the requirements set out in this
testplan.

## ILG

The independent test group is composed of FUB (Italy), CRC (Canada) and
Verizon (USA). A proposal from France Telecom has been received where FT
would become an ILG. However, FT will only act as a member of the ILG if
the MSCQS method is included in the subjective testing process.
Currently, it has been provisionally agreed for FT to participate using
the MSCQS method, but that the results from this method would only be
valid if they mirror results from laboratories using the DSCQS approach.
FT would receive a reduced fee for acting as an independent test
laboratory.

## Proponent Laboratories

A number of proponents also have significant expertise in and facilities
for subjective quality testing. Proponents indicating a willingness to
participate as test laboratories are BT, Genista, NTIA, NTT, and Opticom
and Yonsei. Precise details of how proponent laboratories would create
test material and distribute results from their tests have yet to be
specified. It is clearly important to ensure all test data is derived in
accordance with this testplan. Critically, proponent testing must be
free from charges of advantage to one their model or disadvantage to
competing models.

## Test schedule

The ILG will verify that the submitted models (1) run on the ILG's
computers and (2) yield the correct output values when run on the test
video sequences. Due to their limited resources, the ILG may encounter
difficulties verifying executables submitted too close to the model
submission deadline. Therefore, proponents are strongly encouraged to
submit a prototype model to the ILG prior to the verification deadline,
to work out platform compatibility problems well ahead of the final
verification date. Proponents are also strongly encouraged to submit
their final model executable 14 days prior to the verification deadline
date, giving the ILG two weeks to resolve problems arising from the
verification procedure.

The ILG requests that proponents kindly estimate the run-speed of their
executables on a test video sequence and to provide this information to
the ILG.

# Sequence Processing and Data Formats

Three subjective tests will be performed. The first test will assess the
subjective quality of video material presented on a simulated viewing
environment. This test will deploy a variety of display technologies and
present video in 720p (1280 X 720p resolution). The second test will
present 1080i (1920 X 1080i resolution) video. The third test will use
1080p (1920 X 1080p resolution) video.

Note that in all subjective tests 1 pixel of video will be displayed as
1 pixel native display. No upsampling or downsampling of the video is
allowed.

Presently, VQEG has access to a set of video test sequences. For
audio-video tests this database needs to be extended to include new
source material containing both audio and video.

## Sequence processing overview

The test material will be selected from a common pool of video
sequences.

## Test materials

The test material will be representative of a range of content and
applications. The list below identifies the type of test material that
form the basis for selection of sequences.

1\) movies, movie trailers

2\) sports,

3\) music video,

4\) advertisement,

5\) animation

6\) broadcasting news (business and current events)

7\) home video

### Selection of test material (SRC)

Selection of test material will be done by the ILG. Proponents will be
asked to provide source material as well as SRC/HRC combinations for
consideration by the ILG when selecting test conditions for the
subjective tests. The test should include some agreed percentage (e.g.
20%) of new SRC/HRC combinations that are unknown to proponents. The ILG
will be responsible for selection of this unknown test material.

Further guidelines on selection of and access to test material are
required if proponents are to be test laboratories.

## Hypothetical reference circuits (HRC)

The subjective tests will be performed to investigate a range of HRC
error conditions. These error conditions may include, but will not be
limited to, the following:

-   Compression errors (such as those introduced by varying bit-rate,
    codec type, frame rate and so on)

-   Transmission errors

-   Post-processing effects

The overall selection of the HRCs will be done such that most, but not
necessarily all, of the following conditions are represented:

### Video bit-rates:

-   720p: 2 -- 16 Mbps

-   1080i: 2 -- 22 Mbps

-   1080p: 2 -- 25 Mbps

### Transmission Errors

Error conditions produced using packet loss rates and bit errors:

-   Level 1: None

-   Level 2: Low

-   Level 3 Medium

-   Level 4: High

Proponents are asked to provide examples of error conditions that are
relevant to the industry. These examples will be viewed at the next
meeting and/or examined after electronic distribution (only open source
video is allowed for this). Error conditions can be introduced using
packet-loss and/or bit error conditions.

When producing test material, care must be taken to ensure that the
codec has stabilized before the actual test sequence begins and after it
has ended.

### Frame Freezing and Frame Skipping 

A frame freeze is defined as any event where the video pauses for some
period of time then restarts without losing any video information. The
temporal delay through the system increases. Frame freezes will not be
included in the current testing.

Frame skipping is defined as events where the video pauses then restarts
with some loss of video information. In frame skipping, the temporal
delay through the system is approximately unchanged. Anomalous frame
skipping (\*definition required\*) is not allowed during the first 1s or
the final 1s of a video sequence. Note that where skipping is included
in a test then source material containing still sections should form
part of the testing.

### Frame rates

For those codecs that only offer automatically set frame rate, this rate
will be decided by the codec. Some codecs will have options to set the
frame rate either automatically or manually. For those codecs that have
options for manually setting the frame rate (and we choose to set it for
the particular case), 25 fps will be considered the minimum frame rate.

Manually set frame rates (new-frame refresh rate) may include:

-   720p: 50, 59.94, 60 fps

-   1080i: 24, 29.97, 30 fps

-   1080p: 50, 59.94, 60 fps

### Pre-Processing

The HRC processing may include, typically prior to the encoding, one or
more of the following:

-   Filtering

-   Colour space conversion (e.g. from 4:2:2 to 4:2:0)

-   3:2 Pull down.

This processing will be considered part of the HRC.

### Post-Processing

The following post-processing effects may be used in the preparation of
test material:

-   Colour space conversion

-   De-blocking

-   Decoder jitter

### Coding Schemes

Coding Schemes that will be used may include, but are not limited to:

-   Windows Media

-   MPEG-2

-   H.264 (MPEG-4 Part 10)

-   MPEG 4

-   DV

### Distribution of tests over facilities

### Processing and editing sequences

Test sequences will be captured from the decoded video in uncompressed
format. Two capture methods may be employed. The two methods are as
follows:

### Randomization

### Presentation structure of test material

# Objective Quality Models

## Model type

VQEG HDTV has agreed that Full Reference, Reduced Reference and No
reference models may be submitted for evaluation. The side channel
allowable for the RR models are:

-   720p: (XXX)

-   1080i: (XXX)

-   1080p: (XXX)

Proponents may submit one model of each type for all image size
conditions. Thus, any single proponent may submit up to a total of 13
different models. Note that where multiple models are submitted,
additional model submission fees may apply.

## Model input and output data format

## Submission of executable model

## Registration

**The choice of HRCs and Processing by the ILG will verify that the
following limits are not exceeded between Original Source and Processed
sequences**:

-   maximum allowable deviation in *Peak Video Level* is +/- 10%

-   maximum allowable deviation in *Black Level* is +/- 10%

-   maximum allowable *Horizontal Shift* is +/- 1 pixels

-   maximum allowable *Vertical Shift* is +/- 1 lines

-   maximum allowable *Horizontal Cropping* is 30 pixels

-   maximum allowable *Vertical Cropping* is 20 lines

-   no *Vertical or Horizontal Re-scaling* is allowed

-   *Temporal Alignment* between SRC and HRC sequences shall be
    maintained to within +/- 1 video frames

-   *Dropped or Repeated Frames* are excluded from above temporal
    alignment limit

-   no visible *Chroma Differential Timing* is allowed

-   no visible *Picture Jitter* is allowed

ILG will verify adherence of all HRCs to these limits by using at least
one, but preferably two softwares (NTIA software suggested) in addition
to human checking. The ILG can use proponent software to fix calibration
errors in selected video sequences.  Preferably, such software should be
written in a language that can be easily understood (e.g., Matlab, C++
source code) and posted to the reflector.

VQEG acknowledges that the ILG can not guarantee perfect adherence to
the calibration limitations in section 3.2.5, particularly for very
degraded HRCs. To prevent inclusion of too many HRC that are
nonconforming, proponents will be allowed after objective data & models
submitted but prior to running subjective tests, to analyze video
sequences for calibration errors & suggest fixes. The proponents will be
given three weeks to perform such verification. If the problem cannot be
addressed satisfactorily before the subjective test has been performed,
the offending sequence will be replaced. If a sequence is found to not
adhere to the calibration limitations after the subjective test has been
performed, the offending sequence will not be discarded.

It is suggested that a follow-on study may be performed at a later time
to test sensitivity of models against purposely inserted
mis-calibrations (spatial shift, temporal shift, gain, offset).

## Results analysis

#  Objective quality model evaluation criteria

Data Analysis

The same techniques that were used in the VQEG PhaseII test and reported
in the final report will be used.

If DSCQS, need to include F-tests and consider aggregation issues.

## Introduction to evaluation metrics

A number of attributes characterize the performance of an objective
video quality model as an estimator of video picture quality in a
variety of applications. These attributes are:

-   Prediction Accuracy

-   Prediction Monotonicity

-   Prediction Consistency

This section lists a set of metrics to measure these attributes. The
metrics are derived from the objective model outputs and the results
from viewer subjective rating of the test sequences. Both objective and
subjective tests will provide a single number (figure of merit) for each
half second of the processed sequence that correlates with the video
quality DMOS of the processed sequence. It is presumed that the
subjective results include mean ratings and error estimates that take
into account differences within the viewer population and differences
between multiple subjective testing labs.

The objective quality model evaluation should be done in 5 steps, as
described below. Two main evaluations are proposed, one common
evaluation, and a post processed evaluation. The post processed
evaluation allows an increase in the number of evaluation criteria in a
meaningful way. The quantized evaluation is often more relevant than
other comparisons.

  -------- --------------------------------------------------------------
  1        Data set verified by ILG.

  2        Evaluation Metrics for the data set

  3        Post processing : quantization

  4        Evaluation Metrics for the quantized data set

  5        Diagnostic
  -------- --------------------------------------------------------------

1.  Objective quality model evaluation.

Evaluation metrics are described below and several metrics are computed
to develop a set of comparison criteria. Furthermore, the data set
should not be shared to keep information secure. Thus, if a proponent
wanted to share the data set to distinguish several reduced reference
bitrate categories, or other specific aspects, it will have to be
discussed before the data analysis starts.

## Evaluation Metrics

This section lists the evaluation metrics to be calculated on the
subjective and objective data. Once the nonlinear transformation (see
Section X) section has been applied to subjective and objective data,
the objective model prediction performance is then evaluated by
computing various metrics on the actual sets of data.

The set of differences between measured and predicted DMOS is defined as
the quality-error set Qerror\[\]:

Qerror\[i\] = DMOS\[i\] -- DMOS~p~\[i\]

Where the index *i* refers to processed video sequence.

The following evaluation metrics along with their 95% confidence
intervals and statistical significance tests (i.e.. F-Test) where
applicable will to be used for models' comparison and evaluation:

**Metric 1**: The simple **root-mean-square error** of the error set
Qerror\[\].

$\sqrt{\left( \frac{1}{N}\sum_{N}^{}{\text{Qerror}\lbrack i\rbrack ²} \right)}$

**Metric 2**:

**Pearson linear correlation** coefficient between DMOS~p~ and DMOS.

**Metric 3**: **Outlier Ratio** of "outlier-points" to total points N.

Outlier Ratio = (total number of outliers)/N

where an outlier is a point for which: ABS\[ Qerror\[i\] \] \>
2\*DMOSStandardError\[i\].

Twice the DMOS Standard Error is used as the threshold for defining an
outlier point.

## Generalizability

Generalizability is the ability of a model to perform reliably over a
very broad range of video content. This is a critical selection factor
given the very wide variety of content found in real applications. There
is no specific metric that is specific to generalizability, so this
objective testing procedure requires the selection of as broad a set of
representative test sequences as is possible. The test sequences and
specific HRC's will be selected by the members of VQEG and should ensure
broad coverage of typical content (spatial detail, motion complexity,
color, etc.) and typical video processing conditions. The breadth of the
test set will determine how well the generalizability of the models is
tested. At least 20 different scenes are recommended as a minimum set of
test sequences. It is suggested that some quantitative measures (e.g.,
criticality, spatial and temporal energy) should be used in the
selection of the test sequences to verify the diversity of the test set.

## Complexity

The performance of a model as measured by the above metrics will be used
as the primary basis for model recommendation. If several models are
similar in performance, then the VQEG may choose to take model reference
data bit rate into account in formulating their recommendations. For
similar performance, the smaller reference data bit rate will be
recommended. Thus, if reference data bitrates are not discriminating
enough, a model comparison should be done within each module defined in
ITU document 10-11Q/TEMP/28-R1.

# Calendar and actions 

+----------------+--------------------------------------+-------------+
| **Due date**   | **Action**                           | **Who**     |
+----------------+--------------------------------------+-------------+
|                | Test plan final version              |             |
+----------------+--------------------------------------+-------------+
|                | Sequence processing                  |             |
+----------------+--------------------------------------+-------------+
|                | Subjective tests                     |             |
+----------------+--------------------------------------+-------------+
|                | Subjective data analysis             |             |
+----------------+--------------------------------------+-------------+
|                | Submission of executable models      |             |
+----------------+--------------------------------------+-------------+
|                | Objective data analysis              |             |
|                |                                      |             |
|                | Final report edition                 |             |
+----------------+--------------------------------------+-------------+
|                | Final report.                        |             |
+----------------+--------------------------------------+-------------+

#  Recommendation

The VQEG will recommend methods of objective video quality assessment
based on the primary evaluation metrics defined in Section 6. The Study
Groups involved (ITU-T SG 12, ITU-T SG 9, and ITU-R SG 6) will make the
final decision(s) on ITU Recommendations.

# Bibliography

-   VQEG Phase I final report.

-   VQEG Phase I Objective Test Plan.

-   VQEG Phase I Subjective Test Plan.

-   VQEG FR-TV Phase II Test Plan.

-   Recommendation ITU-R BT.500-11.

-   document 10-11Q/TEMP/28-R1.

-   RR/NR-TV Test Plan

-   VQEG MM Test Plan

-   "Overall quality assessment when targeting wide-XGA flat panel
    displays" by SVT Corporate Development Technology, Sweden.

Annex I\
Instructions to the Subjects

"In this test, we ask you to evaluate the [overall]{.underline} quality
of the video material you see. We are interested in your opinion of the
video quality of each scene. Please do not base your opinion on the
content of the scene or the quality of the acting. Take into account the
different aspects of the video quality and form your opinion based upon
your total impression of the video quality.

Possible problems in quality include:

-   poor, or inconsistent, reproduction of detail;

-   poor reproduction of colours, brightness, or depth;

-   poor reproduction of motion;

-   imperfections, such as false patterns, or "snow".

    The test consists of a series of judgement trials