VQEG HDTV Group

Test Plan for Evaluation of Video Quality Models for Use with High
Definition TV Content

Draft Version 2.0

February, 2008

Contact: Greg Cermak Tel: +1 781-466-4132 Email:
<greg.cermak@verizon.com>

Leigh Thorpe Tel: +1 613 763-4382 Email: <thorpe@nortel.com>

Editorial History

+---------+-------------+----------------------------------------------+
| **Ve    | **Date**    | **Nature of the modification**               |
| rsion** |             |                                              |
+---------+-------------+----------------------------------------------+
| 0.0     | November 1, | Initial Draft, edited by Vivaik              |
|         | 2004        | Balasubrawmanian                             |
+---------+-------------+----------------------------------------------+
| 0.1     | November 9, | Incorporated the following changes from NTIA |
|         | 2004        | (Margaret Pinson):                           |
|         |             |                                              |
|         |             | -   Added an editor's note to highlight the  |
|         |             |     unapproved status.                       |
|         |             |                                              |
|         |             | -   Removed references to future test plans  |
|         |             |     (AV & Interactive)                       |
|         |             |                                              |
|         |             | -   Replaced ACR-HRR with DSIS subjective    |
|         |             |     testing methodology                      |
|         |             |                                              |
|         |             | -   Removed redundant sections               |
|         |             |                                              |
|         |             | -   Minimum bit rate for HRCs is now 2       |
|         |             |     Mbits/s.                                 |
|         |             |                                              |
|         |             | -   Replaced inconsistent section on         |
|         |             |     Calibration/Registration with the latest |
|         |             |     text from RRNR test plan.                |
|         |             |                                              |
|         |             | -   Removed evaluation metrics in line with  |
|         |             |     the agreements reached in the Seoul MM   |
|         |             |     meeting.                                 |
+---------+-------------+----------------------------------------------+
| 0.5     | September   | Incorporated agreements in the April '05     |
|         | 28, 2005    | VQEG meeting in Scottsdale, AZ.              |
+---------+-------------+----------------------------------------------+
|         |             |                                              |
+---------+-------------+----------------------------------------------+
| 1.0     | September   | Incorporated agreements in the September '05 |
|         | 30, 2005    | VQEG meeting in Stockholm, Sweden.           |
+---------+-------------+----------------------------------------------+
| 1.1     | September   | Incorporate changes from audio conferences   |
|         | 21, 2006    | to date; and accept all previous change      |
|         |             | marks.                                       |
+---------+-------------+----------------------------------------------+
| 1.2,    | September   | Changes agreed to at Tokyo VQEG Meeting      |
| 1.3     | 28, 2006    |                                              |
+---------+-------------+----------------------------------------------+
| 1.4     | September 6 | Changes agreed to at Paris VQEG meeting.     |
|         | 2007        | Re-ordering of sections to be more or less   |
|         |             | chronological; re-group subsections into     |
|         |             | relevant sections.                           |
+---------+-------------+----------------------------------------------+
| 2.0     | Febrary,    | Changes agreed to at Ottawa VQEG meeting.    |
|         | 2008        | Proposals inserted for empty sections and    |
|         |             | marked as not having been approved.          |
+---------+-------------+----------------------------------------------+
|         |             |                                              |
+---------+-------------+----------------------------------------------+

**Table of Contents**

1\. Introduction [9](#introduction)

2\. Division of Labor and Schedule [10](#division-of-labor-and-schedule)

2.1. ILG [10](#ilg)

2.2. Proponent Laboratories [10](#proponent-laboratories)

2.3. Test Schedule [10](#test-schedule)

3\. Objective Quality Models [12](#objective-quality-models)

3.1. Model Type [12](#model-type)

3.2. Full Reference Model Input & Output Data Format
[12](#full-reference-model-input-output-data-format)

3.3. Reduced Reference Model Input & Output Data Format
[12](#reduced-reference-model-input-output-data-format)

3.4. No Reference Model Input & Output Data Format
[13](#no-reference-model-input-output-data-format)

3.3 Submission of Executable Model [13](#submission-of-executable-model)

4\. Subjective Rating Tests [14](#subjective-rating-tests)

4.1. Subjective Dataset Submission [14](#subjective-dataset-submission)

4.2. Number of Datasets to Validate Models
[14](#number-of-datasets-to-validate-models)

4.3. Test Design [14](#test-design)

Subjective [15](#subjective-test-conditions)

4.4. Test Conditions [15](#subjective-test-conditions)

4.4.1. Application Across Different Video Formats and Displays
[15](#application-across-different-video-formats-and-displays)

4.4.2. Viewing Conditions [15](#viewing-conditions)

4.4.3. Display Specification and Set-up
[15](#display-specification-and-set-up)

4.5. Subjective Test Method: ACR-HRR
[15](#subjective-test-method-acr-hrr)

4.6. Length of Sessions [16](#length-of-sessions)

4.7. Subjects [16](#subjects)

4.8. Instructions for Subjects and Failure to Follow Instructions
[16](#instructions-for-subjects-and-failure-to-follow-instructions)

4.9. Randomization [17](#randomization)

4.10. Subjective Data File Format [18](#subjective-data-file-format)

5\. Source Video Sequences [19](#source-video-sequences)

5.1. Selection of Source Sequences (SRC)
[19](#selection-of-source-sequences-src)

5.2. Purchased Source Sequences [19](#purchased-source-sequences)

5.3. Requirements for Camera, Lens and SRC Coding
[19](#requirements-for-camera-lens-and-src-coding)

5.4. Content [19](#content)

5.5. Scene Cuts [19](#scene-cuts)

5.6. Scene Durration [19](#scene-durration)

5.7. Source Scene Selection Criteria
[19](#source-scene-selection-criteria)

6\. Video Format and Naming Conventions
[21](#video-format-and-naming-conventions)

6.1. Storage of Video Material [21](#storage-of-video-material)

6.2. Video File Format [21](#video-file-format)

6.3. Naming Conventions [21](#naming-conventions)

7\. HRC Constraints and Sequence Processing
[22](#hrc-constraints-and-sequence-processing)

7.1. Sequence Processing Overview [22](#sequence-processing-overview)

7.1.1. Format Conversions [22](#format-conversions)

7.1.2. PVS Duration [22](#pvs-duration)

7.2. Constraints on Hypothetical Reference Circuits (HRCs)
[22](#constraints-on-hypothetical-reference-circuits-hrcs)

7.2.1. Coding Schemes [22](#coding-schemes)

7.2.2. Video Bit-Rates: [23](#video-bit-rates)

7.2.3. Video Encoding Modes [23](#video-encoding-modes)

7.2.4. Frame Freezing and Frame Skipping
[23](#frame-freezing-and-frame-skipping)

7.2.5. Rewinding [23](#rewinding)

7.2.6. Frame rates [23](#frame-rates)

7.2.7. Transmission Errors [23](#transmission-errors)

7.3. Processing and Editing of Sequences
[23](#processing-and-editing-of-sequences)

7.3.1. Pre-Processing [23](#pre-processing)

7.3.2. Post-Processing [24](#post-processing)

8\. Calibration \[XXX Review this section\]
[25](#calibration-xxx-review-this-section)

8.1. HRC Calibration Constraints [25](#hrc-calibration-constraints)

HRC Calibration Problems [25](#hrc-calibration-problems)

8.2. [25](#hrc-calibration-problems)

9\. Objective Quality Model Evaluation Criteria
[26](#objective-quality-model-evaluation-criteria)

9.1. Evaluation Criteria [26](#evaluation-criteria)

9.2. Evaluation Procedure [26](#evaluation-procedure)

10\. Recommendation [27](#recommendation)

11\. References [28](#references)

List of Acronyms

ACR-HRR Absolute Category Rating with Hidden Reference Removal

ANOVA ANalysis Of VAriance

ASCII ANSI Standard Code for Information Interchange

CCIR Comite Consultatif International des Radiocommunications

CODEC Coder-Decoder

CRC Communications Research Center (Canada)

DMOS Difference Mean Opinion Score (as defined by ITU-R)

DVB-C Digital Video Broadcasting-Cable

FR Full Reference

GOP Group of Pictures

HD High Definition (television)

HRC Hypothetical Reference Circuit

ILG Independent Lab Group

IRT Institut Rundfunk Technische (Germany)

ITU International Telecommunications Union

ITU-R ITU Radiocommunications Standardization Sector

ITU-T ITU Telecommunications Standardization Sector

MM Multimedia

MOS Mean Opinion Score

MOSp Mean Opinion Score, predicted

MPEG Motion Pictures Expert Group

NR No (or Zero) Reference

NTSC National Television Standard Committee (60-Hz TV, used mainly in US
and Canada)

PAL Phase Alternating Line (50-Hz TV, used in Europe and elsewhere)

PS Program Segment

PVS Processed Video Sequence

RR Reduced Reference

SMPTE Society of Motion Picture and Television Engineers

SRC Source Reference Channel or Circuit

SSCQE Single Stimulus Continuous Quality Evaluation

VQEG Video Quality Experts Group

List of Definitions

[Intended frame rate]{.underline} is defined as the number of video
frames per second physically stored for some representation of a video
sequence. The [intended frame rate]{.underline} may be constant or may
change with time. Two examples of *constant* [intended frame
rates]{.underline} are a BetacamSP tape containing 25 fps and a VQEG
FR-TV Phase I compliant 625-line YUV file containing 25 fps; these both
have an [absolute frame rate]{.underline} of 25 fps. One example of a
*variable* [absolute frame rate]{.underline} is a computer file
containing only new frames; in this case the [intended frame
rate]{.underline} exactly matches the [effective frame
rate]{.underline}. The content of video frames is not considered when
determining [intended frame rate]{.underline}.

[Anomalous frame repetition]{.underline} is defined as an event where
the HRC outputs a single frame repeatedly in response to an unusual or
out of the ordinary event. [Anomalous frame repetition]{.underline}
includes but is not limited to the following types of events: an error
in the transmission channel, a change in the delay through the
transmission channel, limited computer resources impacting the decoder's
performance, and limited computer resources impacting the display of the
video signal.

[Constant frame skipping]{.underline} is defined as an event where the
HRC outputs frames with updated content at an [effective frame
rate]{.underline} that is fixed and less than the [source frame
rate]{.underline}.

[Effective frame rate]{.underline} is defined as the number of unique
frames (i.e., total frames -- repeated frames) per second.

[Frame rate]{.underline} is the number of (progressive) frames displayed
per second (fps).

[Live Network Conditions]{.underline} are defined as errors imposed upon
the digital video bit stream as a result of live network conditions.
Examples of error sources include packet loss due to heavy network
traffic, increased delay due to transmission route changes, multi-path
on a broadcast signal, and fingerprints on a DVD. [Live network
conditions]{.underline} tend to be unpredictable and unrepeatable.

[Pausing with skipping (formerly frame skipping)]{.underline} is defined
as events where the video pauses for some period of time and then
restarts with some loss of video information. In [pausing with
skipping]{.underline}, the temporal delay through the system will vary
about an average system delay, sometimes increasing and sometimes
decreasing. One example of [pausing with skipping]{.underline} is a pair
of IP Videophones, where heavy network traffic causes the IP Videophone
display to freeze briefly; when the IP Videophone display continues,
some content has been lost. Another example is a videoconferencing
system that performs [constant frame skipping]{.underline} or [variable
frame skipping]{.underline}. Constant frame skipping and variable frame
skipping are subset of pausing with skipping. A processed video sequence
containing [pausing with skipping]{.underline} will be approximately the
same duration as the associated original video sequence.

P[ausing without skipping (formerly frame freeze)]{.underline} is
defined as any event where the video pauses for some period of time and
then restarts without losing any video information. Hence, the temporal
delay through the system must increase. One example of [pausing without
skipping]{.underline} is a computer simultaneously downloading and
playing an AVI file, where heavy network traffic causes the player to
pause briefly and then continue playing. A processed video sequence
containing [pausing without skipping]{.underline} events will always be
longer in duration than the associated original video sequence.

[Refresh rate]{.underline} is defined as the rate at which the computer
monitor is updated.

[Rewinding]{.underline} is defined as an event where the HRC playback
jumps backwards in time. Rewinding can occur immediately after a pause.
Given the reference sequence (A B C D E F G H I), two example processed
sequence containing rewinding are (A B C D B C D E F) and (A B C C C C A
B C). Rewinding can occur as a response to transmission error; for
example, a video player encounters a transmission error, pauses while it
conceals the error internally, and then resumes by playing video prior
to the frame displayed when the transmission distortion was encountered.
Rewinding is different from variable frame skipping because the subjects
see the same content again and the motion is much more jumpy.

[Simulated transmission errors]{.underline} are defined as errors
imposed upon the digital video bit stream in a highly controlled
environment. Examples include simulated packet loss rates and simulated
bit errors. Parameters used to control [simulated transmission
errors]{.underline} are well defined.

[Source frame rate]{.underline} (SFR) is the [intended frame
rate]{.underline} of the original source video sequences. The source
frame rate is constant.

[Transmission errors]{.underline} are defined as any error resulting
from sending the video data over a transmission channel. Examples of
transmission errors are corrupted data (bit errors) and lost packets /
lost frames. Such errors may be generated in live network conditions or
through simulation.

[Variable frame skipping]{.underline} is defined as an event where the
HRC outputs frames with updated content at an [effective frame
rate]{.underline} that changes with time. The temporal delay through the
system will increase and decrease with time, varying about an average
system delay. A processed video sequence containing [variable frame
skipping]{.underline} will be approximately the same duration as the
associated original video sequence.

#  Introduction

This document defines evaluation tests of the performance of objective
perceptual quality models conducted by the Video Quality Experts Group
(VQEG). It describes the roles and responsibilities of the model
proponents participating in this evaluation, as well as the benefits
associated with participation. The role of the Independent Lab Group
(ILG) is also defined. The text is based on discussions and decisions
from meetings of the VQEG HDTV working group (HDTV) at the periodic
face-to-face meetings as well as on conference calls and in email
discussion.

The goal of the HDTV project is to recommend a quality model suitable
for application to digital video quality measurement in HDTV
applications. The performance of objective models with HD signals will
be determined from a comparison of viewer ratings of a range of video
sample quality obtained in controlled subjective tests and the quality
predictions from the submitted models. In accordance with decisions made
at the Ottawa meeting, the test plan has been simplified to reduce the
work load for the ILG. The authors of the models ("proponents") will do
most of the work laid out in this Test Plan: selecting and preparing
video source sequences (SRCs), preparing video test sequences (PVSs),
gathering subjective quality ratings for the test sequences, ,carrying
out the objective measurement of those same sequences with their
particular model(s), and for much of the analysis comparing the
subjective and objective results. An ILG within the HDTV group will
coordinate tests and help assure their compliance with the conditions of
this Test Plan.

For the purposes of this document, HDTV is defined as being of or
relating to an application that creates or consumes High Definition
television video format that is digitally transmitted over a
communication channel. Common applications of HDTV that are appropriate
to this study include television broadcasting, video-on-demand and
satellite and cable transmissions. The measurement tools recommended by
the HDTV group will be used to measure quality both in laboratory
conditions using a full reference (FR) method and in operational
conditions using reduced reference (RR) or no-reference (NR) methods.

To fully characterize the performance of the models, it is important to
examine a full range of representative transmission and display
conditions. To this end, the test cases (hypothetical reference circuits
or HRCs) should simulate the range of potential behavior of cable,
satellite, and terrestrial transmission networks and broadband
communications services. The recommendation(s) resulting from this work
will be deemed appropriate for services delivered on high definition
displays using 1080i, 720p, computer desktop monitors, and high
definition display television technologies.

In Phase I of the HDTV testing, video-only test conditions will be
employed. Currently, HDTV source material appropriate for creating test
samples is in short supply. VQEG would like to obtain material
copyright-free or with a royalty-free license for research purposes for
these and future tests. Our ability to perform adequate audio-video and
multimedia testing will depend on access to a bank of appropriate source
material.

Display formats that will be addressed in these tests are: 1080i at 50
and 60 Hz, 720p at 50 and 60 fps, and 1080p at 50 and 60 fps. Currently,
the following are of particular interest:

-   1080i 60 Hz (30 fps) Japan, US

-   720p (50 fps) Europe

-   1080i 50 Hz (25 fps) Europe.

Ratings of hypothetical reference circuits (HRCs) for each display
format used will be gathered in separate subjective tests. The
performance of submitted models will be evaluated separately by display
format. The method selected for the subjective testing is Absolute
Category Rating with Hidden Reference Removal. The quality predictions
of the submitted models will be compared with subjective ratings from
human viewers from other proponents' submitted subjective tests.

It is also proposed that a test of currently standardized standard
definition models be tested for their extensibility to High Definition
TV.

The final report will summarize the results and conclusions of the
analysis along with recommendations for the use of objective perceptual
quality models for each HDTV format.

# Division of Labor and Schedule

The HD group wishes to proceed with the HD project before the MM and
RRNR-TV projects are completed. This test plan has been defined taking
into account the limited ILG resources available, since few ILG
resources are available. A number of pragmatic compromises were made to
enable implementation of a test plan using minimal ILG resources while
continuing to have acceptable checks on the fairness of the process.
Otherwise, the project would be required to waiting an undesirable
period of time, in order to proceed with a plan that reflects ideal
fairness checks. These decisions were:

-   Assign ILG only those tasks that are necessary to ensure independent
    validation.

-   Have proponents design and implement subjective tests.

-   Have proponents submit subjective test results simultaneously with
    models.

## ILG

The independent test group will be taking the role of independent
arbitrator for the HDTV test. The ILG role will be primarily to helping
proponents decide whether their testing abides by the HDTV test plan
restrictions, between the date when the HDTV test plan is finalized, and
the date when models and subjective tests are submitted. Other
proponents cannot participate in these clarification decisions, since
all proponent tests are supposed to be secret from other proponents.

The independent test group for this work consists of the following labs:
XXX

In addition, the ILG can optionally provide HDTV subjective testing free
of charge, and submit those datasets at the model/test submission date.
If too few proponents participate in the HDTV test, then one or more ILG
labs will be hired to perform subjective testing, so that the
restrictions concerning the minimal number of subjective datasets for
the evaluation (Section 4.1) are met.

## Proponent Laboratories

Each proponent will provide one (and only one) subjective test dataset.
The subjective datasets must meet all of the test plan\'s constraints
(e.g., identical number of video sequences and number of test subjects).
If the proponent does not have the facilities to perform subjective
testing, then the proponent may hire an ILG facility to perform the
testing.

VQEG recognizes that a proponent's model may have been trained on the
subjective data submitted.

## Test Schedule

  ---- ------------------------------------------ -------------------------
  1    Approval of test plan.                     

  2    Date to declare intent to participate and  
       the number of models that will be          
       submitted.                                 

  3    Fee payment (if applicable) if additional  
       ILG subjective test are required.          

  4    Donated source video sequences are         
       collected and redistributed among labs.    

  5    Proponents wanting to use purchased SRC    
       obtain agreement from ILG and other        
       Proponents (see section XXX).              

  6    Proponents submit source video sequences   
       to ILG for quality approval.               

  7    Proponents submit their models to ILG and  
       (optionally) to other proponents.          

  8    Proponents using purchased SRC submit      
       final purchase information to other        
       proponents.                                

  9    Proponents submit their SRC, PVSs,         
       subjective data, subjective test design to 
       ILG and all other proponents.              

  10   Calibration checked on all video           
       sequences. PVSs needing optional           
       calibration settings identified, and       
       values agreed upon.                        

  11   Objective model data run on all subjective 
       datasets.                                  

  13   Statistical analysis by proponents and     
       possibly ILG.                              

  14   Draft final report.                        

  15   Approval of final report.                  
  ---- ------------------------------------------ -------------------------

#  Objective Quality Models

## Model Type

VQEG HDTV has agreed that Full Reference (FR), Reduced Reference (RR)
and No-reference (NR) models may be submitted for evaluation. The side
channel allowable for the RR models are:

-   720p: (56 kbs, 128 kbs, 256 kbs)

-   1080i: (56 kbs, 128 kbs, 256 kbs)

Proponents may submit one model of each type (FR, RR, NR) to apply to
both image size conditions. Thus, any single proponent may submit up to
a total of three different models. Note that where multiple models are
submitted, additional model submission fees may apply.

## Full Reference Model Input & Output Data Format

The FR model will be a single program. The model must take as input an
ASCII file listing pairs of video sequence files to be processed. Each
line of this file has the following format:

\<source-file\> \<processed-file\>

where \<source-file\> is the name of a source video sequence file and
\<processed-file\> is the name of a processed video sequence file. File
names may include a path. Each line may also optionally contain
calibration values. Preferably, calibration values should appear the
following order (appearing after \<processed-file\>):

\<Y_gain\> \<Y_offset\> \<cb_gain\> \<cb_offset\> \<cr_gain\>
\<cr_offset\> \<h_shift\> \<v_shift\> \<delay\>

The output file is an ASCII file created by the model program, listing
the name of each processed sequence and the resulting Video Quality
Rating (VQR) of the model.

\<processed-file\> VQR

Where \< source-file \> is the name of the source file run through this
model, without any path information; and \<processed-file\> is the name
of the processed sequence run through this model, without any path
information. VQR is the Video Quality Ratings produced by the objective
model.

Each proponent is also allowed to output one or more files containing
Model Output Values (MOVs) that the proponents consider to be important.

## Reduced Reference Model Input & Output Data Format

RR models must be submitted as two programs:

-   A "source side" program that takes the original video sequence, and

-   A "processed side" program that takes the processed video sequence.

Data communicated must be stored to files, which will be used to check
data transmission rate. The source side program must be able to run when
the processed video is absent. The processed side program must be able
to run when the source video is absent. Any type of model that meets
these criterion may be submitted.

The input control list and output data files will be as listed for the
FR model.

## No Reference Model Input & Output Data Format

The NR model will be given an ASCII file listing only processed video
sequence files. Each line of this file has the following format:

\<processed-file\>

where \<processed-file\> is the name of a processed video sequence file.
File names may include a path. Each line may also optionally contain
calibration values, if the proponent desires.

Output data files will be as listed for the FR model.

NR models will be required to predict the perceptual quality of both the
source and processed video files used in subjective quality tests.

## 3.3 Submission of Executable Model  {#submission-of-executable-model .unnumbered}

Proponents may submit up to five models: one full reference, one no
reference, and one for each of the reduced reference information bit
rates given in the test plan (i.e., 56 kbit/sec, 128 kbit/sec, 256
kbit/sec). Each proponent will submit an executable of the model(s) to
the Independent Labs Group (ILG) for validation. Encrypted source code
also may optionally be submitted. If necessary, a proponent may supply a
specific computer or machine that implements the model. The ILG will
verify that the software produces the same results as the proponent. If
discrepancies are found, the independent and proponent laboratories will
work together to correct them. If the errors cannot be corrected, then
the ILG will review the results and recommend further action.

Proponents may receive other proponents' models and perform validation,
if the model's owner finds this acceptable. An ILG lab will be available
to validate models for proponents who cannot let out their models to
other proponents.

# Subjective Rating Tests

Subjective tests will be performed on two display resolutions: 720p
(1280 X 720p resolution) and 1080i (1920 X 1080i resolution). In both
cases, the tests will assess the subjective quality of video material
presented in a simulated viewing environment, and will deploy a variety
of display technologies.

## Subjective Dataset Submission

Each proponent must submit one subjective dataset. This dataset must
comply with all restrictions in this test plan. All of the video
sequences (source and processed) and all of the subjective data must be
distributed to all other proponents and also to ILG performing model
validation.

Submitted subjective datasets may use source video that must be
purchased (i.e., source video sequences that other proponents must
purchase prior to receiving that subjective dataset). Because the
appropriateness of purchased source may depend upon the price of those
sequences, the total cost must be openly discussed before a proponent
chooses to use purchased source sequences (e.g., VQEG reflector, audio
conference); and the seller must be identified. (Reminder: the scenes to
be purchased must be kept secret until model & subjective dataset
submission). A majority of proponents and at least one ILG lab must be
able to purchase these source video sequence (i.e., for model
validation). In the event that purchases source sequences are used, that
laboratory must provide (along with the subjective dataset submission)
the remaining details needed to purchase these source sequences. If a
proponent cannot afford to purchase the source sequences, then another
proponent or ILG lab will run their model against the purchased video
sequences.

All subjective datasets must be held "secret" prior to model &
subjective dataset submission. That is, no proponents may have any
knowledge of the scenes or HRCs chosen by another proponent. That is, no
other proponent can be told which scenes or HRCs will appear in other
proponents' subjective datasets.

Along with the subjective test, all laboratories will provide a file
that defines the HRCs used in this subjective tests. The file shall
explicitly show the parameter values/settings used for every HRC in the
test. Manufacturer names should be omitted.

## Number of Datasets to Validate Models

A minimum of four datasets will be used to validate the objective
models. These datasets may come from no fewer than three independent
sources. If less than four subjective datasets are available, then the
proponents must pay for ILG laboratories to create the required
subjective datasets.

There will be a minimum of three independent sources of subjective
datasets (e.g., three proponents, or two proponents + one paid ILG
tests); and a minimum of four independent datasets (e.g., at least four
tests where each test has its own set of 160-200 PVSs (as determine
above) and 24 subjects who did not participate in any of the other three
tests). Therefore, each model will be evaluated based on at least three
datasets that were not used to train that model.

## Test Design

The HD Test Plan is designed as a distributed and decentralized effort
of the HDTV Group. Test designs are not expected to be the same across
labs, and are subject only to the following constraints:

-   Each lab will test the same number of PVSs. The number is
    tentatively identified as between 160 and 200, including both
    processed and reference video clips.

-   The minimum number of SRCs in each test is 10 \[XXX subject to
    vote\].

-   The minimum number of HRCs in each test is 10 \[XXX subject to
    vote\].

-   The test design matrix need not be rectangular ("full factorial")
    and will not necessarily be the same across tests.

## Subjective Test Conditions

### Application Across Different Video Formats and Displays

The proposed HDTV test will examine the performance of objective
perceptual quality models for different video formats (720p and 1080i).
Section 5.2.3 defines format and display types in detail. Video
applications targeted in this test include internet video on demand,
HDTV broadcasts, etc.

The instructions given to subjects will request subjects to maintain a
specified viewing distance from the display device. The viewing distance
has been agreed as 1 minute of arc for each resolution:

-   720p: 4H.

-   1080i: 3H.

where H = Picture Height (picture is defined as the size of the video
window, not the physical display.)

### Viewing Conditions

Each test subject will have his/her own video display. Subjects will be
seated directly in line with the centre of the video display at the
specified viewing distance. The test room will conform to ITU-T Rec.
P.910 requirements.

### Display Specification and Set-up

Given that the subjective tests will use different HD display
technologies, it is necessary to ensure that each test laboratory
selects appropriate display specification and common set-up techniques
are employed. Due to the fact that most consumer grade displays employ
some kind of display processing that will be difficult to account for in
the models, all subjective facilities doing testing for HD TV shall use
a transparent \[XXX tbd\] display.

## Subjective Test Method: ACR-HRR

The VQEG HDTV subjective tests will be performed using the ACR-HRR
method.

The selected test methodology is the Absolute Category Rating method
with Hidden Reference Removal (ACR-HRR). The ACR method has been used
successfully for many years \[ITU-T Recommendation P.910, 1999.\] Its
advantages are simplicity, that it can be applied to a relatively large
number of PVSs in a short time, and that it is relatively easy to
implement in computer-controlled experiments.

Hidden Reference Removal has been added to the method more recently to
address a disadvantage of ACR for use in studies in which objective
models must predict the subjective data: If the original video material
(SRC) is of poor quality, or if the content is simply unappealing to
viewers, such a PVS could be rated low by humans and yet not appear to
be degraded to an objective video quality model, especially a
full-reference model. In the HRR addition to ACR, the original version
of each SRC is presented for rating somewhere in the test, without
identifying it as the original. Viewers rate the original as they rate
any other PVS. The rating score for any PVS is computed as the
difference in rating between the processed version and the original of
the given SRC. Effects due to esthetic quality of the scene or to
original filming quality are "differenced" out of the final PVS
subjective ratings.

In the ACR-HRR test method, each test condition is presented once for
subjective assessment. The test presentation order is randomized
according to standard procedures (e.g., Latin or Graeco-Latin square or
via computer). Subjective ratings are reported on the five-point scale:

> 5 Excellent
>
> 4 Good
>
> 3 Fair
>
> 2 Poor
>
> 1 Bad.

Figure borrowed from the ITU-T P.910 (1999):

![](media/image1.wmf){width="5.238194444444445in"
height="1.6354166666666667in"}

## Length of Sessions

The time of actively viewing videos and voting will be limited to 50
minutes per session. Total session time, including instructions,
warm-up, and payment, will be limited to 1.5 hours.

## Subjects

Each test will require exactly 24 subjects.

The HDTV subjective testing will be conducted using viewing tapes or the
equivalent. Video sequences may be presented from a hard disk through a
computer instead of video tapes, provided that (1) playback mechanism is
guaranteed to play at frame rate without dropping frames, (2) playback
mechanism does not impose more distortion than the proposed video tapes
(e.g., compression artifacts), and (3) monitor criteria are respected.

It is preferred that each subject be given a different randomized order
of video sequences where possible. Otherwise, the viewers will be
assigned to sub-groups, which will see the test sessions in different
randomized orders. At least two different randomized presentations of
clips (A & B) will be created for each subjective test. If multiple
sessions are conducted (e.g., A1 and A2), then subjects will view the
sessions in different orders (e.g., A1-A2, A2-A1). Each lab should have
approximately equal numbers of subjects at each randomized presentation
and each ordering.

Only non-expert viewers will participate. The term non-expert is used in
the sense that the viewers' work does not involve video picture quality
and they are not experienced assessors. They must not have participated
in a subjective quality test over a period of six months. All viewers
will be screened prior to participation for the following:

-   normal (20/30) visual acuity with or without corrective glasses (per
    Snellen test or equivalent).

-   normal colour vision (per Ishihara test or equivalent).

-   familiarity with the language sufficient to comprehend instruction
    and to provide valid responses using the semantic judgment terms
    expressed in that language.

## Instructions for Subjects and Failure to Follow Instructions

For many labs, obtaining a reasonably representative sample of subjects
is difficult. Therefore, obtaining and retaining a valid data set from
each subject is important. The following procedures are highly
recommended to ensure valid subjective data:

-   Write out a set of instructions that the experimenter will read to
    each test subject. The instructions should clearly explain why the
    test is being run, what the subject will see, and what the subject
    should do. Pre-test the instructions with non-experts to make sure
    they are clear; revise as necessary.

-   Explain that it is important for subjects to pay attention to the
    video on each trial.

-   There are no "correct" ratings. The instructions should not suggest
    that there is a correct rating or provide any feedback as to the
    "correctness" of any response. The instructions should emphasize
    that the test is being conducted to learn viewers' judgments of the
    quality of the samples, and that it is the subject's opinion that
    determines the appropriate rating.

-   Monitor each subject in the experiment. The experimenter should be
    close enough to each subject to see if they are alert, to see if
    they have lost their place, to see if they look confused, and to see
    if they are responding reasonably (e.g., not giving the same rating
    to every PVS). Monitoring is especially important during the first
    few presentations of video clips and the corresponding responses.

-   Paying subjects helps keep them motivated.

If it is suspected that a subject is not responding to the video stimuli
or is responding in a manner contrary to the instructions, their data
may be discarded and a replacement subject can be tested. The
experimenter will report the number of subjects' datasets discarded and
the criteria for doing so. Example criteria for discarding subjective
data sets are:

-   The same rating is used for all or most of the PVSs.

-   The subject's ratings correlate poorly with the average ratings from
    the other subjects (see Annex II).

-   Different subjective experiments will be conducted by several test
    laboratories. Exactly 24 valid viewers per experiment will be used
    for data analysis. A valid viewer means a viewer whose ratings are
    accepted after post-experiment results screening. Post-experiment
    results screening is necessary to discard viewers who are suspected
    to have voted randomly. The rejection criteria verify the level of
    consistency of the scores of one viewer according to the mean score
    of all observers over the entire experiment. The method for
    post-experiment results screening is described in Annex VI. Only
    scores from valid viewers will be reported .

The following procedure is suggested to obtain ratings for 24 valid
observers:

> 1\. Conduct the experiment with 24 viewers
>
> 2\. Apply post-experiment screening to eventually discard viewers who
> are suspected to have voted randomly
>
> 3\. If n viewers are rejected, run n additional subjects.
>
> 4\. Go back to step 2 and step 3 until valid results for 24 viewers
> are obtained.

## Randomization

For each subjective test, a randomization process will be used to
generate orders of presentation (playlists) of video sequences.
Playlists can be pre-generated offline (e.g. using separate piece of
code or software) or generated by the subjective test software itself at
runtime.

Randomization refers to a random permutation of the set of PVSs used in
that test.

> Note: The purpose of randomization is to average out order effects,
> ie, contrast effects and other influences of one specific sample being
> played following another specific samples. Thus, shifting does not
> produce a new random order , e.g.:
>
> Subject1 = \[PVS4 PVS2 PVS1 PVS3\]
>
> Subject2 = \[PVS2 PVS1 PVS3 PVS4\]
>
> Subject3 = \[PVS1 PVS3 PVS4 PVS2\]

If a random number generator is used (as stated in section 4.1.1), it is
necessary to use a different starting seed for different tests.

An example script in Matlab that creates playlists (i.e., randomized
orders of presentation) is given below:

> rand(\'state\',sum(100\*clock)); % generates a random starting seed
>
> Npvs=200; % number of PVSs in the test
>
> Nsubj=24; % number of subjects in the test
>
> playlists=zeros(Npvs,Nsubj);
>
> for i=1:Nsubj
>
> playlists(:,i)=randperm(Npvs);
>
> end

## Subjective Data File Format

Subjective data should NOT be submitted in archival form (i.e., every
piece of data possible in one file). The working file should be a
Miscrosoft Excel spreadsheet listing only the following necessary
information:

-   Experiment ID

-   Source ID Number

-   HRC ID Number

-   Video File

-   Each Viewer's Rating in a separate column (Viewer ID identified in
    header row)

All other information should be in a separate file that can later be
merged for archiving (if desired). This second file should have all the
other \"nice to know\" information indexed to the subjectIDs: date,
demographics of subject, eye exam results, etc. A third file, possibly
also indexed to lab or subject, should have ACCURATE information about
the design of the HRCs and possible something about the SRCs.

An example table is shown below (where HRC "0" is the original video
sequence).

  ------------ ----- ----- ------------------- -------- -------- -------- -------- ----- --------
                                               Viewer   Viewer   Viewer   Viewer   ...   Viewer
                                               ID       ID       ID       ID             ID

  Experiment   SRC   HRC   File                1        2        3        4        ...   24
               Num   Num                                                                 

  XYZ          1     1     xyz_src1_hrc1.avi   5        4        5        5        ...   4

  XYZ          2     1     xyz_src2_hrc1.avi   3        2        4        3        ...   3

  XYZ          1     7     xyz_src1_hrc7.avi   1        1        2        1        ...   2

  XYZ          3     0     xyz_src3_hrc0.avi   5        4        5        5        ...   5
  ------------ ----- ----- ------------------- -------- -------- -------- -------- ----- --------

# Source Video Sequences

## Selection of Source Sequences (SRC)

Selection of source sequences will be made by the proponents.
Coordination among proponents may be provided by the ILG. Proponents can
not have any knowledge of the source sequences selected for any
subjective test other than their own.

## Purchased Source Sequences

See section 4.1 for constraints on the use of purchased source
sequences.

## Requirements for Camera, Lens and SRC Coding

The source video can only be used in the testing if an expert in the
field considers the quality to be good or excellent on an ACR-scale. The
source video should have no visible coding artifacts.

The ILG will view the scene pools from all proponents and confirm that
all source video sequence have sufficient quality. The ILG will also
ensure that there is a sufficient range of source material and that
individual SRCs are not over-used. After the approval of the ILG, all
scenes will be considered final. No scene may be discarded or replaced
after this point for any technical reason.

## Content

The source sequences will be representative of a range of content and
applications. The list below identifies the types of test material that
form the basis for selection of sequences.

1\) movies, movie trailers

2\) sports

3\) music video

4\) advertisement

5\) animation

6\) broadcasting news (business and current events)

7\) home video

8\) general TV material (e.g., documentary, sitcom, serial television
shows)

## Scene Cuts

Scene cuts shall occur at a frequency that is typical for each content
category.

## Scene Durration

Final source sequences will 10 seconds. Source scenes used for HRC
creation will typically use extra content at the beginning and end.

## Source Scene Selection Criteria

Source video sequences selected for each test should adhere to the
following criteria:

1.  All source may have different frame rates (25fps and 30fps).

2.  All source may have different image resolutions (720x1280 and
    1080x1920).

3.  At least one scene must be very difficult to code.

4.  At least one scene must be very easy to code.

5.  At least one scene must contain high spatial detail.

6.  At least one scene must contain high motion and/or rapid scene cuts
    (e.g., an object or the background moves 50+ pixels).

7.  If possible, one scene should have multiple objects moving in a
    random, unpredictable manner.

8.  At least one scene must be very colorful.

9.  If possible, one scene should contain some animation or animation
    overlay (e.g., cartoon, scrolling text).

10. If possible, at least one scene should contain low contrast (e.g.,
    soft or blurred edges).

11. If possible, at least one scene should contain high contrast (e.g.,
    hard or clearly focused edges, such as the SMPTE birches scene).

12. If possible, at least one scene should contain low brightness (e.g.,
    dim lighting, mostly dark).

13. If possible, at least one scene should contain high brightness
    (e.g., predominantly white or nearly white).

# Video Format and Naming Conventions

## Storage of Video Material

Video material will be stored, rather than being presented from a live
broadcast. The most practical storage medium at the time of this Test
Plan is a computer hard disk. Hard disk drives will be used as the main
storage medium for distribution of video sequences among labs. As well,
having material stored as files on a hard disk allows for randomization
of the PVSs for playback to each subject (or simultaneously-viewing
group).

## Video File Format 

All SRC and PVSs will be stored in uncompressed AVI files in UYVY color
space.

## Naming Conventions

All Source video sequences should be numbered (e.g., SRC 1, SRC 2). All
HRCs should be numbered, and the original video sequence must be number
"0" (e.g., SRC 1 / HRC 0 is the original video sequence #1). All files
must be named: \<experiment\>\_src\<src_id\>\_hrc\<hrc_id\>.avi, where
\<experiment\> is a string identifying the experiment; \<src_id\> is
that source sequence's number, and \<hrc_id\> is that HRC's number. For
example:

> xyz_src1_hrc0.avi
>
> xyz_src1_hrc1.avi
>
> xyz_src1_hrc2.avi
>
> xyz_src2_hrc0.avi
>
> xyz_src2_hrc1.avi
>
> xyz_src2_hrc2.avi

# HRC Constraints and Sequence Processing 

## Sequence Processing Overview

The HRCs will be selected separately by the individual proponent or ILG
running that test. While audio will not be used in the present tests,
the audio tracks on source sequences should be retained wherever
possible in both source and processed video clips (SRCs and PVSs) for
use in future tests. In cases where IP is involved in the HRC, transport
streams should be saved and Ethereal dumps should be captured and stored
whenever possible.

### Format Conversions 

A PVS must be the same scale, resolution, and format as the original. An
HRC can include transformations such as 720p to NTSC to 720p as long as
one pixel of video is displayed as one pixel native display. No
up-sampling or down-sampling of the video image is allowed in the final
PVS.

Where a progressive display is used and the test sample requires
de-interlacing, then this de-interlacing will be performed offline, and
the model will be given the same de-interlaced sample as is shown to the
viewer.

### PVS Duration

PVSs to be used in testing will be 10 seconds long. SRC may be longer
and trimmed to length before testing.

## Constraints on Hypothetical Reference Circuits (HRCs)

The subjective tests will be performed to investigate a range of HRC
error conditions including both mild and severe errors. These error
conditions may include, but are not be limited to, the following:

-   Compression artifacts (such as those introduced by varying bit-rate,
    codec type, frame rate and so on)

-   Pre- and post-processing effects

-   Transmission errors

HRCs in one experiment may be the same or different from HRCs in other
experiments. The HDTV group will determine an equitable way to aggregate
models' performances across different kinds of HRCs.

The overall selection of the HRCs should be done such that most, but not
necessarily all, of the codecs, bit rates, encoding modes and
impairments set out in the following sections are represented.

### Coding Schemes

Coding schemes that may be used in the current tests are:

-   VC1

-   MPEG-2

-   H.264 (AVC high profile and main profile).

Coding schemes not to be included in the current test are:

> • H.264 (SVC)
>
> • DivX
>
> • MJPEG-2000.

### Video Bit-Rates:

Bit rates were chosen to accommodate the coding schemes above and to
span a wide range of video quality: 

-   720p: 1--30 Mbps

-   1080i: 1--30 Mbps

### Video Encoding Modes

The encoding modes that will be used may include, but are not limited
to:

-   Constant-bit-rate encoding (CBR)

-   Variable-bit-rate encoding (VBR)

### Frame Freezing and Frame Skipping 

A frame freeze is defined as any event where the video pauses for some
period of time then restarts without losing any video information. The
temporal delay through the system increases. Frame freezes will not be
\[may be\] included in the current testing.

Frame skipping is defined as events where the video pauses then restarts
with some loss of video information. In frame skipping, the temporal
delay through the system is approximately unchanged. Frame skipping may
be included in the more severe error conditions. Note that where
skipping is included in a test then source material containing still
sections should form part of the testing.

### Rewinding

XXX \[to be discussed & decided\] Rewinding is an allowed impairment for
the HD tests, provided that the time alignment of each frame is within
the test plan limitations.

### Frame rates

For those codecs that only offer automatically-set frame rate, this rate
will be decided by the codec. Some codecs will have options to set the
frame rate either automatically or manually. For those codecs that have
options for manually setting the frame rate, and should an HRC require a
manually set frame rate, the minimum frame rate used will be 24 fps.

Manually set frame rates (new-frame refresh rate) may include:

-   720p: 24, 25, 50, 59.94, 60 fps \[Ed. Note: Check if 24, 25 fps is
    reasonable for 720p and 1080p\]

-   1080i: 24, 25, 29.97, 30 fps

### Transmission Errors

Transmission error conditions will be included in first phase of the
project. The types of errors that may be used include packet errors
(both IP and Transport Stream) such as packet loss, packet delay
variation, jitter, overflow and underflow, bit errors, and over the air
transmission errors. Error concealment and forward error correction
should be included in at least some of the HRCs.

## Processing and Editing of Sequences

### Pre-Processing

The HRC processing may include, typically prior to the encoding, one or
more of the following:

-   Filtering

-   Colour space conversion (e.g. from 4:2:2 to 4:2:0)

-   3:2 Pull down.

This processing will be considered part of the HRC.

### Post-Processing

No post-processing effects may be used explicitly in the preparation of
test material (e.g., de-blurring, de-blocking, noise filtering).
Post-processing that is part of a codec is allowed if it cannot be
avoided.

# Calibration \[XXX Review this section\]

## HRC Calibration Constraints

**The choice of HRCs and Processing by the ILG will verify that the
following limits are not exceeded between Original Source and Processed
sequences**:

-   maximum allowable deviation in *luminance gain* is +/- 10%

-   maximum allowable deviation in *luminance offset* is +/- 20

-   maximum allowable deviation in *Cb and Cr gain* is +/- 20%

-   maximum allowable deviation in *Cb and Cr offset* is +/- 40

-   maximum allowable *Horizontal Shift* is +/- 1 pixels

-   maximum allowable *Vertical Shift* is +/- 1 lines

-   maximum allowable *Horizontal Cropping* is 30 pixels

-   maximum allowable *Vertical Cropping* is 20 lines

-   no *Vertical or Horizontal Re-scaling* is allowed

-   *Temporal Alignment* between SRC and HRC sequences shall be
    maintained to within +/- 1 video frames

-   *Dropped or Repeated Frames* are excluded from above temporal
    alignment limit

-   no visible *Chroma Differential Timing* is allowed

-   no visible *Picture Jitter* is allowed

Laboratories will verify adherence of all HRCs to these limits by using
at least one, but preferably two software packages (NTIA software
suggested) in addition to human checking.

## HRC Calibration Problems

Since subjective data sets will be finalized prior to submission and
remain secret until then, calibration cannot be double checked (i.e., by
other proponents) until after model submission.

If a proponent identifies a calibration problem at this time, then those
calibration values will be addressed by optional allowing models to
inputs calibration values. In this case, all models must use identical
calibration values -- or the default "no calibration".

# Objective Quality Model Evaluation Criteria

This section describes the evaluation metrics and procedure used to
assess the performances of an objective video quality model as an
estimator of video picture quality in a variety of applications.

## Evaluation Criteria

Each model will be evaluated against all datasets. Primary analysis will
consist of each model evaluated on datasets unknown to that proponent
(i.e., computed by other proponents or ILG). The dataset produced by the
model's proponent will be reported but must be clearly marked as such
(e.g., "training data").

## Evaluation Procedure

The performance of an objective quality model to each subjective dataset
will be characterized by (1) calculating DMOS values, (2) mapping to the
subjective scale, (3) computing the following three evaluation metrics:

-   Pearson Correlation Coefficient

-   Root Mean Square Error

-   Outlier Ratio

along with the 95% confidence intervals of each, and finally (4) testing
for statistically significant differences among the performance of
various models with the F-test.

These formulae are given in the MultiMedia Test Plan, version 1.19. Note
that equation (11) must be fixed by, substituting "stderr" for "sigma".

#  Recommendation

The VQEG will recommend methods of objective video quality assessment
based on the primary evaluation metrics defined in Section 6. The Study
Groups involved (ITU-T SG 12, ITU-T SG 9, and ITU-R SG 6) will make the
final decision(s) on ITU Recommendations.

# References

-   VQEG Phase I final report.

-   VQEG Phase I Objective Test Plan.

-   VQEG Phase I Subjective Test Plan.

-   VQEG FR-TV Phase II Test Plan.

-   Recommendation ITU-R BT.500-11.

-   document 10-11Q/TEMP/28-R1.

-   RR/NR-TV Test Plan

-   VQEG MM Test Plan

"Overall quality assessment when targeting wide-XGA flat panel displays"
by SVT Corporate Development Technology, Sweden.

\[1\] M. Spiegel, "Theory and problems of statistics", McGraw Hill,
1998.

Annex I\
Example Instructions to Viewers

\[XXX Subject to Revision Depending on Data-Collection Method Chosen\]

"In this test, we ask you to evaluate the [overall]{.underline} quality
of the video material you see. We are interested in your opinion of the
video quality of each scene. Please do not base your opinion on the
content of the scene or the quality of the acting. Take into account the
different aspects of the video quality and form your opinion based upon
your total impression of the video quality.

Possible problems in quality include:

-   poor, or inconsistent, reproduction of detail;

-   poor reproduction of colours, brightness, or depth;

-   poor reproduction of motion;

-   imperfections, such as false patterns, or "snow".

The test consists of a series of judgement trials. During each trial,
two versions of a single video sequence, which may or may not differ in
picture quality, will be shown in the following way:

![](media/image2.wmf)

"A" is the first version, "B" is the second version. The first
presentation of a trial will be announced with the written caption "A",
and the second with "B". This pair of presentations will then be
repeated, thereby completing a single trial.

In judging the overall quality of the presentations, we ask you to use
judgement scales like the samples shown below.

![](media/image3.wmf)

SAMPLE QUALITY SCALE

As you can see, there are two scales for each trial, one for the "A"
presentation and one for the "B" presentation, since both the "A" and
"B" presentations are to be judged.

The judgement scales are continuous vertical lines that are divided into
five segments. As a guide, the adjectives "excellent", "good", "fair",
"poor", and "bad" have been aligned with the five segments of the
scales. You are asked to place a [single horizontal]{.underline} line at
the point on the scale that best corresponds to your judgement of the
overall quality of the presentation (as shown in the example).

![](media/image4.wmf)

You may make your mark at any point on the scale, which most precisely
represents your judgement.

In making your judgements, we ask you to use the first pair of
presentations in the trial to form an impression of the quality of each
presentation, but to refrain from recording your judgements. You may
then use the second pair of presentations to confirm your first
impressions and to record your judgements in your Response Booklet.

We will now show you four demonstration trials.

DEMONSTRATION TRIALS PRESENTED AT THIS POINT

Annex II\
Method for Post-Experiment Screening of Subjects

A statistical criterion for rejecting a subject's data is that it
correlates with the average of the other subjects' data no better than
chance. The linear Pearson correlation coefficient per PVS for one
viewer vs. all viewers is defined as:

$$r1(x,y) = \frac{\sum_{i = 1}^{n}{x_{i}y_{i} - \frac{\left( \sum_{i = 1}^{n}x_{i} \right)\left( \sum_{i = 1}^{n}y_{i} \right)}{n}}}{\sqrt{\left( \sum_{i = 1}^{n}{x_{i}^{2} - \frac{\left( \sum_{i = 1}^{n}x_{i} \right)^{2}}{n}} \right)\left( \sum_{i = 1}^{n}{y_{i}^{2} - \frac{\left( \sum_{i = 1}^{n}y_{i} \right)^{2}}{n}} \right)}}$$

Where

xi = MOS of all viewers per PVS

yi = individual score of one viewer for the corresponding PVS

n = number of PVSs

i = PVS index.

**Rejection criterion**

A subject's data are declared to be no better than chance if they
correlate less than

1.96 \*( sigma sub Z) from paragraph 6.4.1 above, where
$\sigma_{z} = \sqrt{1(N - 3)}$. For N = 180, sigma sub Z = 0.075, and
1.96 \* sigma sub Z = 0.147. The Fisher Z to R transformation gives the
corresponding R = 0.148. Therefore, to reject a subject's data on the
grounds of randomness,

1\. Calculate R.

2\. Exclude a viewer if R\<0.15.