# Minutes of the Seoul, Korea VQEG Meeting

**Summary**

The meeting was hosted by CITY (Center for Information Technology of
Yonsei University) and held on the campus of Yonsei University.

The meeting was chaired by Arthur Webster (NTIA, USA) and began on
Monday 18 October with brief reports from the various Ad-Hoc and sub
groups in VQEG:

Independent Lab Group -- Phil Corriveau (INTEL, USA) by phone.

RRNR-TV- Alex Bourret (BT, U.K.)

Multimedia Ad Hoc Group Kjell Brunnstrom (Acreo, Sweden)

HDTV- Margaret Pinson (NTIA, USA)

Model Combination and Tools -- Margaret Pinson (NTIA, USA)

MMPretest Coordinator -- Arthur Webster (NTIA, USA)

Subjective Labs Setup Group -- (No report available)

POC for Source and HRC Sequence collection -- Chulhee Lee (Yonsei,
Korea)

RRNR-TV session was chaired by Alex Bourret (BT, UK) and followed the Ad
Hoc Group reports.

Although the test plan is complete, a problem has arisen with the lack
of secret source material in 625 for the RRNR-TV test. This is delaying
the progress of the test.

HDTV discussions followed the RRNR-TV session

Steve Wolf and Vittorio Baroncini are co-chairs of this Ad-Hoc Group but
were not present. Vivaik Balasubrawmanian (INTEL, USA) volunteered to be
editor of the test plan. This effort is in the initial stages.

The VQEG Ad-Hoc Group on Multimedia & Joint Rapporteurs' Group on
Multimedia Quality Assessment between ITU-T SG 9 and SG 12 (JRG-MMQA)
Meetings began on Tuesday and continued through Friday afternoon. These
sessions were co-chaired by David Hands (BT, UK) and Kjell Brunnstrom
(Acreo, Sweden).

The bulk of this meeting was spent on the presentation of preliminary
test results and on editing the Multimedia test plan. Representatives
from ITU-T SG9 and SG12 and ITU-R WP6Q were present.

Results of the preliminary test were presented by NTIA, NTT, BT, Yonsei,
and SwissQual. The preliminary tests uncovered a number of problems
associated with test sequence processing, AVI file players, monitor
selection, and other subjective test set up issues. Further testing will
be done and discussions will take place on the MMTEST reflector and the
MM Forum.

The MM test plan was significantly advanced.

See below for detailed minutes of Monday's sessions.

See file
<ftp://ftp.its.bldrdoc.gov/ituvidq/Seoul_VQEG_Oct04/Minutes_VQEG_MM_Seoul_Oct04.doc>
for detailed minutes of the MM/JRG-MMQA sessions Tuesday, Wednesday,
Thursday, and Friday.

See file:
<ftp://ftp.its.bldrdoc.gov/ituvidq/Seoul_VQEG_Oct04/Participants_Seoul_Oct04.doc>

# Detailed Minutes for Monday's Meetings

# Reports from working groups

1\. ILG (Philip C)

-   for RRNR sufficient 525 material (including secret sequences)

-   but issue with 625 material (VB may provide secret new material, but
    none offered yet)

-   Require 3-5 minutes of new \[secret\] 625 material

Discussions with owners of FR Phase II material ongoing to ensure all
interested parties can have access to the video (existing agreement ends
31 Dec 2004).

Still require software for checking spatial rescaling of sequences

-   software available from NTIA

-   but sw has some limitations

Test labs

-   CRC

-   FT

-   FUB

-   Verizon (does not have SSCQE apparatus, but CRC/FT will loan
    necessary apparatus)

Schedule -- must be revised (at least 2 month delay is needed, could be
more if problem with material continues).

PB -- TDF have prepared and sent new HRC to FUB. PC -- no record of
this. Need to confirm.

Similarly, CL prepared new 625 material and await advice from ILG on who
to send this material to.

Nortel will provide new 525 HRC material that extends error
characteristics to include ???

2\. RRNR-TV (Alex B)

-   testplan has been approved

-   main issue remains 625 test material

-   main task is to review schedule

3\. HDTV

MP provided S Wolf's report on HDTV test

-   test material is currently main issue

-   KB has provided HDTV material from Swedish TV and this is on NTIA's
    file store

-   Logitech provided some advertisement material

-   NTIA have performed HDTV subjective test

-   Results expected to be made available at next meeting

-   PC may be able to provide new HDTV material on D5 format (MP and PC
    to discuss offline)

-   AW mentioned an Australian source may provide HDTV material (CL will
    discuss this with the Australian contact at the next ITU-R 6Q
    meeting)

4\. Collaboration and Tools Group

MP provided S Wolf's report

-   tools currently being evaluated for use by VQEG

-   need to discuss placing tools on a VQEG website (sw available from
    NTIA, FT, Swissqual, NTT)

-   registration measurement, spatial scaling tools, e2e RR measurement
    tool available 2005

-   Opticom offered MM sw (Linux) that could be placed on the VQEG
    server

5\. MM Pre-test

AW reported on pre-test work

-   a number of docs have been placed on forum

-   several tests have been performed and results to be presented at
    this meeting

6\. Subjective labs set-up group

No report from this group.

7\. POC for source and HRC sequence collection

CL contacted 2 Korean broadcasters and has received about 3 hours
digibeta material (also available as VQEG PII yuv file format). Material
only available for VQEG tests and not open source. Legal document needs
to be signed to obtain access to this material.

# RRNR-TV

-   number of network error HRCs

-   ILG problems (especially secret test material)

-   Review testplan

-   Test schedule

1\. Network Loss

-   Nortel keen to see more HRCs with network loss

-   Current testplan states for 525 and for 625 (is there 625 line
    network loss HRCs available? FUB preparing/have some HRCs?) there
    will be two transmission error HRCs

-   Nortel would like to see this increased to 4 transmission error HRCs

-   CL argued that the nature of transmission error is clearly defined
    so that these types of error do not come as a surprise to proponent
    models

-   Nortel would be prepared to generate these new HRCs

-   Fastweb could create the 625 HRCs (if not, Nortel may be able to
    provide 625 HRCs)

-   

-   Nortel asked if they could specify nature of errors

-   Say types of error are those associated with packet loss

-   Decision on adding new HRCs to be made Monday afternoon

-   vote on whether to increase number of transmission errors:

-   no decision

-   MP proposed modified text:

-   'min of 2 transmission error HRCs and max of 25% of processed video
    clips'

-   in favour = 6

-   against = 0

Nortel will provide samples + descriptions of their processed sequences
(provided by 1 Nov 2004); they propose to use the ISO reference decoder
and remove effects of loss concealment techniques.

2\. Review of Testplan

Very bottom p.18 -- new text from MP approved

Top p.23 (Section 5.1.1) - new text from MP approved

Bottom p.23 (Section 5.1.3) - new text from MP approved

Bottom p.23 (Section 5.1.4) - new text from MP approved

3\. Schedule

Following changes agreed:

a)  submission of new HRCs by proponents -- changed from July 14 to Nov
    18

b)  submission of exe model -- changed to Jan 18

> \['b' means that all associated items on the schedule changed by 2
> months\]
>
> \[see revised testplan for full details of changes to schedule\]
>
> CL may provide registration measurement sw.
>
> PB asked for clarification testplan:
>
> \- on p.18 tolerances: is 1 line related to field or frame
>
> MP said it was per frame
>
> PB asked how material will be delivered -- MP said files will be
> delivered as 1 minute segments rather than as a single 60 minute file
>
> AW -- when and how should we return to the test schedule?

-   need advice from ILG

-   AB will arrange an audiocall to finalise the schedule

# HDTV Test

PC will provide HDTV material that NTIA will place on ftp site.

AW asked if anyone has HDTV material

OS will discuss with colleagues possibility of providing material.

CL will investigate obtaining HDTV material.

Vivaik agreed to become editor of HDTV testplan.

MP to contact PC to discuss getting HDTV material.