Minutes from VQEG Multimedia (Editorial Group) Audio Call, Thursday 10
February 2005.

Chairs: David Hands (BT) and Kjell Brunnstrom (Acreo)

Participants

David Hands (BT) \-\-- co-chair

Kjell Brunnstrom (Acreo) \-\-- co-chair

Arthur Webster (NTIA)

Philip Corriveau (Intel)

Greg Cermak (Verizon)

Margaret Pinson (NTIA)

Steve Wolf (NTIA)

Chulhee Lee (Yonsei)

Filippo Speranza (CRC)

Ron Renaud (CRC)

Irina Cortanis (Ericcson)

**Agenda**

1\. Introduction

2\. Chapter updates

3\. AOB

1.  Introduction

DH opened the audio call, the agenda was agreed and the purpose of the
audio call outlined. The audio call was arranged to obtain updates on
each chapter of the MM testplan, identify any outstanding issues or
problems and to detail work to be done before next full VQEG meeting.

2\. Progress Update

*Chapter 2:* GC provided an update of the subjective test chapter. GC
noted that he had fully reviewed and updated the chapter. This was
distributed to the editorial group for comment. GC commented that he
thought the chapter was almost complete. One contentious issue concerns
allowing subjects to review test sequences (e.g. if they lost
concentration or missed a sequence). No decision made yet. This issue
will be discussed in an audio call to agree upon test platform. GC also
identified the issue of a common GUI that may be used in the test. This
will also be discussed in the test platform audio call.

*Chapter 3:* KB provided an update on test labs and schedule. KB said
that the schedule is dependent upon progress on the testplan as a whole
and no modifications have been made. Agreement on test laboratory issues
require all test labs, including ILG and proponents to discuss and come
to a common opinion. Focused discussions by the proponents group has not
happened. KB will contact Christian Schmidmer to discuss how best to
define test requirements for proponent labs. FS raised two problems with
the testplan. Firstly, the requirement for 20% of unknown test material
is a real issue, especially with many labs and potentially thousands of
test sequences. The problems with this requirement needs to be resolved.
Secondly, the availability of a common (free) video player remains. FS
noted that even Mplayer is only really appropriate for Linux.

KB will arrange an audio call to discuss test platform (issues include
unknown test sequences, video player, data collection, video card, PCU
requirements).

*Chapter 4:* MP referred to an email identifying six issues that need to
be resolved before Chapter 4 can be finalised. These issues were:

-   The exact process to transform from Rec. 601 & HD resolution video
    to VGA, CIF and QCIF resolution video.

-   The acceptable HD camera(s) and/or specifications for collecting HD
    source material. For instance, would the new 3-chip Sony HVR-Z1U
    camera be acceptable? While this camera performs compression, the
    resolution reduction process in item 1 above may make this
    compression transparent.

-   Specification of the attributes for the secret test material in
    section 4.2.1.

-   Specification of transmission errors (section 4.3.2) and live
    network conditions (section 4.3.3). Quan and Christian were
    intending to discuss this issue.Specification of process for
    up-sampling low frame rate sequences (e.g., 5, 8 Hz) to full frame
    rate sequences (e.g., 25 Hz, 30 Hz), and corresponding PC monitor
    scan rates (section 4.3.5).

-   Specification of distribution of tests over facilities (section
    4.3.9).

It was noted that insufficient discussion took place on either the
reflector or the forum. MP will organise an audio call to discuss these
issues. PC agreed to host both audio calls.

*Chapter 5:* CL reviewed each of the 5 sections attached to Chapter 5.
The main issues concerned selection of colour space (yuv or RGB) and
registration. On colour space, it was agreed that this should be
included in the audio call to be organised by MP as it affects both
chapters 4 and 5. The audio call should have as a **requirement** a
decision to be made on colour space. On registration, CL said that
recent tests he had performed indicated time warping. The current
testplan allows a maximum of +/-0.25s of temporal variation in a test
sequence. CL was unsure if this was adequate. CL will send a message to
the reflector outlining the issue of time warping.

*Chapter 6:* IC gave an update on the evaluation metrics to be used. IC
outlined 4 main issues: data processing, selection of metrics, F-test,
aggregation. IC is in the process of preparing an updated chapter on
evaluation metrics that will be posted to the reflector in mid-March.
The chapter will include details of the transformation method to be used
(polynomial or logistic), metrics, a description of the F-test and a
method for aggregation. IC raised the issue of output from different
models. Currently, the testplan states that for FR and RR models a DMOS
will be produced but for NR models a MOS will be obtained. This is
problematic for defining a single evaluation chapter. KB will check
minutes from previous meeting to ascertain if this is correct or if the
meeting agreed that DMOS are to be calculated for all types of models.

AW reported that he had sent an ANSI set of definitions to the reflector
in December. He had received no comment and assumes that these are
acceptable. MP has submitted (9 February 2005) additional definitions.
MP noted that these are fundamental to any Chapter 4 discussions and
will be looking to get these agreed either prior to or at the start of
the audio call on Sequence Processing and Data Formats. MP to send
definitions to full VQEG reflector.

3\. AOB

Chapter owners are to send updated chapters to DH by 4 April. DH will
then update the testplan and send to the full VQEG reflector by 18
April.

Close.