Video Quality Experts Group

September 25 -- 29, 2006

Tokyo, Japan.

Final Participants List

  --------------------------- -------------------------- -----------------
  **Aygen**                   **Arman**                  **Genista**

  **Bourret**                 **Alex**                   **BT**

  **Brunnström**              **Kjell**                  **Acreo**

  **Dhondt**                  **Yves**                   **Ghent Univ.**

  **Ferguson**                **Kevin**                  **Tektronix**

  **Ford**                    **Carolyn**                **NTIA/ITS**

  **Horita**                  **Yuukou**                 **Toyama Univ.**

  **Huynh-Thu**               **Quan**                   **Psytechnics**

  **Juric**                   **Pero**                   **SwissQual**

  **Kurita**                  **Takaaki**                **NTT**

  **Le Callet**               **Patrick**                **IRCCyN**

  **Lee**                     **Chulhee**                **Yonsei Univ.**

  **Miyamoto**                **Yoshihiro**              **NEC**

  **Okamoto**                 **Jun**                    **NTT**

  **Pinson**                  **Margaret**               **NTIA/ITS**

  **Sazzad**                  **Z.M.Parvez**             **Toyama Univ.**

  **Schmidmer**               **Christian**              **Opticom**

  **Sugimoto**                **Osamu**                  **KDDI R&D**

  **Takahashi**               **Akira**                  **NTT**

  **Watanabe**                **Keishiro**               **NTT**

  **Webster**                 **Arthur**                 **NTIA/ITS**

  **Winkler**                 **Stefan**                 **Genista**

  **Yamada**                  **Toru**                   **NEC**

  **Corriveau (phone)**       **Phil**                   **INTEL**
  --------------------------- -------------------------- -----------------

### Agenda

**Video Quality Experts Group**

**September 25-29, 2006 - Meeting**

**Tokyo, Japan**

**www.vqeg.org**

(Note: The ITU-T JRG-MMQA met coincident with the Multimedia Sessions of
the VQEG Meeting)

**Final Agenda**

**Monday, September 25**

8:00 Start / Designate Note Taker (Webster/Speranza)

Introductions

Meeting Logistics

Updates (Maximum 15 minutes - each Group)

Independent Lab Group (ILG)(Brunnstrom)

RRNR-TV (Bourret/ Lee)

Multimedia (Brunnstrom)

HDTV (Pinson)

Tools and Subjective Labs Setup Group (Le Callet)

POC for Source and HRC Sequence collection (Lee)

10:00 Begin MM/JRG-MMQA

11:30-1:00 Lunch Hosted by KDDI

1:00 MM and JRG-MMQA

0\) MM TestPlan

1\) The 20% clause x

2\) De-Interlacing x

3\) The use of actual broadcast hardware for the creation of HRCs x

5\) Distribution of originals. x

6\) Secret Source & HRCs. x

7\) Replicating experiments. x

8\) Hybrid approach x

9\) Schedule Discussions x

5:00 End of Day

**Tuesday, September 26**

8:00 Start / Designate Note Taker

Review Monday's Decisions (Webster/Speranza)

MM Continues

New items:

ILG replicating test x

6a. Fitting functions (tbd after input from statisticians)

Tektronix contribution on data analysis; x

Opticom discussion of polynomial & logistics fitting (tbd after input
from statisticians)

Mini Test Demonstrations x

Source/PVS Material Review (not needed)

5:00 End of Day

**Wednesday, September 27**

8:00 Start / Designate Note Taker

Review Tuesday's Decisions (Webster/ Speranza)

Review MM Decisions

RRNR-TV Discussion

Following the inclusion of H264 and VC1 codecs in HRC list :

\- Decision on model access to codec information x

\- Decision on allowing bitstream models x

\- Impact on the test organization x

\- Discussion on result analysis x

Revision of calendar and actions

Final Review of RRNR-TV Decisions

5:00 End of Day

7:00 Dinner Hosted by KDDI at Dome Hotel

**Thursday, September 28**

8:00 Start / Designate Note Taker

Review Wednesday's Decisions (Webster/Corriveau)

10:00- 11:30 HDTV (TBD)

Source and PVS availability

Industry needs

11:30-1:00 Lunch

1:00 -- 5:00 HDTV -- Continued

Source and PVS availability

Current testing

5:00 End of Day

**Friday September 29**

8:00 Start / Designate Note Taker

Review Thursday's Decisions (Webster/Corriveau)

Final Review of HDTV decisions (if needed)

10:00 Review and Documentation of VQEG Decisions

Review action items

11:00 Other Business

Next Meeting

11:30-1:00 Lunch

1:00-3:00 Write Liaison statements to ITU etc.

3:00 Close VQEG meeting

# VQEG meeting minutes - Monday 25 September 2006 

## Introduction of meeting attendees

Agenda and logistics review (Monday lunch hosted and Wednesday dinner
hosted by KDDI).

Updates from the different ad-hoc groups:

-   ILG: KB said he would wait for more info to come in from FS, GC and
    PC.

    -   ILG labs: IrCCyN, FUB, FT, CRC, INTEL, ACREO and Verizon.

    -   ILG has been working at dividing workload between labs
        concerning SRC and HRC selection

    -   De-interlacing issue to prepare the SRCs remain an issue

    -   ILG worked at defining the exact fees but this is difficult to
        finalize because some proponents have not committed to a number
        of models they will submit.

-   RRNR-TV:

    -   AB reported an audio-call in July.

    -   H.264 and VC-1 have been added in the RRNR-TV test plan. This
        introduced new points to be addressed. Initially the test plan
        targeted only MPEG-2. Because new codecs are now considered,
        some discussion started about the possibility to provide some
        side information (i.e. codec name) to the objective models --
        both RR and NR.

-   MM:

    -   KB reported that major progress since the Boston meeting is the
        completion of all the annexes in the test plan

    -   CRC has done the SRC selection from the original material from
        KDDI

    -   CL has brought all the original material from the different MM
        content providers. This data can be exchanged provided each lab
        has signed the required NDA.

    -   MP said that Vivaik is unable to spend time on the
        de-interlacing because of Intel restructuring.

    -   ACREO has updated the subjective testing software and uploaded
        it on the ftp server. KB encouraged all labs to test the
        software and feedback any issue.

    -   MM test plan timetable must be revisited and discussed.

-   HDTV:

    -   MP reported that 2 audio conferences took place and open issues
        remain to be addressed.

    -   CS said Opticom is likely to become a proponent in the HDTV
        effort

-   Tools and subjective lab set-up:

    -   PLC reported that current progress in MM effort is the updated
        subjective testing software by ACREO

    -   Tools are needed to perform the data analysis in the MM effort

    -   Solutions for HDTV subjective testing are still being
        investigated.

-   SRCs:

    -   CL has received all the video data, except the KDDI data which
        was sent only to CRC to make to a selection. The selected
        portions will be shared to proponents

    -   QH commented that portions not selected from the KDDI original
        material can not be part of the secret SRCs.

    -   NDAs: most MM proponents have signed NDAs. The organizations
        that have not signed yet the NDAs are Lucent and Toyama
        University.

Announcement from NTIA/ITS:

NTIA is withdrawing from the MM effort and remains a proponent in
RRNR-TV and HDTV effort. NTIA wishes to become an ILG lab for the MM
effort. NTIA may require a fee as an ILG lab. NTIA can help in
obtaining/shooting new SRCs, de-interlacing the video material and
producing HRCs.

The fact that an organization can be a proponent in one effort and an
ILG lab in another effort will be discussed during the meeting
\[discussion held Tuesday provided decisions immediately below\].

Decision:

C. Schmidmer states that NTIA should not have access to the Opticom
fitting software so that they will not potentially have an advantage in
other tests besides MM.

Decision:

Any secret content that NTIA sees cannot then be used in other tests as
secret content.

AW reported that ITU rules demand now that all Rapporteurs meetings
documents and Liaison Statement will have to be posted officially to the
ITU. This includes documents from the JRG-MMQA meeting.

## MM discussion

KB reported that the main remaining issue in the current MM test plan is
the step-by-step process and timeline. The main technical obstacle in
the MM effort is the de-interlacing of original interlaced material to
progressive video, especially for VGA resolution.

ILG sent a list of concerns that they think they need to be addressed
before the step-by-step schedule can be finalized:

-   The "20% clause"

-   De-interlacing

-   The use of actual broadcast hardware for the creation of HRCs

-   Distribution of originals

-   Secret SRCs and HRCs

-   Replication of experiments

-   Detail about the schedule

The details of this list can be found in the document named 'ILG
concerns list.doc' on the ftp server. These points will be discussed
later during the meeting.

SW said he would like the MM test plan to be re-discussed concerning the
current video file format (i.e. AVI) and the possibility to change the
chose format in order to provide more information to the objective
models.

### Subjective testing display

KB reported that Section 4.1.3 in the test plan has a sentence 'Annex V
contains a list of preferred LCD monitors for use in the subjective
tests'. Currently Annex V does not specify such list. KB said that
specifying a list might be too restrictive. His proposal is to allow use
of LCD displays that have a 'TCO 06 label'. Currently only 3 display
have this certification:

-   BenQ FP241W, FP241WZ, FP241VW (Q24W5)

-   EIZO FlexScan S2110W ColorEdge CE210W (S2110W)

-   Samsung 215TW (DP21)

### Data analysis

Editorial changes in Section 8:

Equations (5) and (14): CI=...

Subsection title added for outlier ratio.

Equation (17) is missing but is in the version 1.13 of the test plan.

The word 'monotonic' is missing in the text for the cubic polynomial
mapping (Agreement was reached at the Boston meeting -- see Boston
meeting minutes -- to perform the cubic polynomial **monotonic** fit as
the primary fit, with the logistic fit as the backup in case of no
convergence).

Discussion re-opened on which mapping to be used: if one model does not
converge with the monotonic cubic polynomial, then should all models use
the logistic mapping or just the model for which the polynomial mapping
did not converge?

### Review of the different points of concern sent by ILG:

**20% clause:**

Current test plan specifies that "For each proponent subjective test, no
more than 20% of test sequences may be derived from a single proponent."
FS suggests that the restriction only applies to proponents (not to ILG
labs). Another proposal is to increase the limit from 20% to 34% or 50%.

Vote to change the limit from 20% to 34%:

For: Acreo, Psytechnics, KDDI, Yonsei, NTIA, SwissQual, Toyama, NTT, BT,
Genista

Against: none

[Agreement reached:]{.underline} For each proponent subjective test, no
more than 34% of test sequences may be derived from a single proponent.
This does not apply to PVSs created by the ILG.

Vote to exclude the set of common sequences from this limit:

For: Acreo, Psytechnics, KDDI, Yonsei, NTIA, SwissQual, NTT, Opticom,
IrCCyN, Genista, BT

Against: none

[Agreement reached:]{.underline} For each proponent subjective test, no
more than 34% of test sequences may be derived from a single proponent.
This does not apply to PVSs created by the ILG or to common sequences.

Vote to increase the limit to 50%:

For: Acreo, Psytechnics, IrCCyN, NTIA, SwissQual, BT, Genista

Against: Yonsei

[Decision reached]{.underline}: For each proponent subjective test, no
more than 50% of test sequences may be derived from a single proponent.
This does not apply to PVSs created by the ILG or to common sequences.

**De-interlacing:**

Proposal from ILG: Modify section 6.1.4 to allow other de-interlacing
mechanisms, e.g., proprietary algorithms and hardware de-interlacing.

[Agreement reached to modify Section 6.1.4 to]{.underline}:

De-interlacing will be performed when original material is interlaced,
using the de-interlacing function "KernelDeint" in Avisynth. If the
deinterlacing using KernelDeint results in source sequence that has
serious artifacts, the Blendfield or Autodeint may be used as
alternative methods for deinterlacing. Proprietary algorithms and/or
hardware deinterlacing may be used if the above three methods prove
unsatisfactory.

(Agreement reached from vote and support by following organizations:
Acreo, Psytechnics, IrCCyN, NTIA, SwissQual, BT, Genista, NTT, Opticom

**Use of actual broadcast hardware:**

CRC would like to use a hardware encoder taking only SDI input and
outputs only 352x240 (vs. 352x288 in test plan) and 176x120 (vs. 176x144
in test plan). The group agreed to the fact that
interlacing-deinterlacing can be part of the HRC. However, it is unknown
if CRC's system warps the input video or not. Also padding from 352x240
to 352x288 (or 176x120 to 176x144) is disallowed in the test plan.

No agreement reached on this point.

**Detail about schedule:**

Currently, the entire source pool is to be de-interlaced as a whole
before 12-sec portions are selected to produce the reference sequences.

Proposal by ILG: First the ILG selects 12-sec clips for experiments, and
then only those clips are de-interlaced. So, reverse order of these two
steps.

No decision reached on this point since this involves re-visiting the
step-by-step procedure. This point will be discussed later during the
meeting.

**Secret Source & HRCs:**

Proposal from ILG: The secret material in each proponent test will be
part of the common set.

[Agreement reached to add following sentence in 4.1.7]{.underline}:

The common set of PVSs will include the secret PVSs and secret SRCs

**Replicating experiments:**

Proposal from ILG: The ILG would like to have the option open to use
some of the secret tests to replicate experiments (i.e., run viewers
through another lab's experiment).

No decision reached.

**Tuesday notes VQEG Tokyo Sept. 26, 2006**

**Thanks to M.Pinson for taking Tuesday notes**

**Filippo's MM ILG update**

Nortel will be ILG for MM test.

On fee payment, plan must be updated. Answer needed promptly on whether
NTIA will require a fee or not.

Selection of source sequences from KDDI, KBS, Yonsei, CRC and FT have
been done. SVT is almost done. NTIA & Psytechnics are not done but will
be relatively easy because footage already 12-seconds in duration.
SwissQual will require a small effort; Opticom will require a lot of
effort (lots of content); and NTT effort fairly small -- 4 scenes each 1
minute in duration.

**Filippo's RRNR-TV & HDTV ILG update**

Not much yet to do; RRNR-TV issue is source sequences. Addition of H.264
will require extra processing. Should consider starting over with new
sequences, e.g., from MM test. Test plans are still moving targets;
feedback is needed on video material (SRC sequences).

See also Filippo Speranza's report in the meeting file folder.

**MM test plan discussions continue**

On replicating experiments: Concerns raised that some proponents don't
want two replicated experiments combined into one experiment for model
comparison purposes. For model evaluation, each of the replicated
experiments should be treated as two separate tests, and aggregated as
per MM test plan specifications (i.e., don't average the two sets of
viewers.) Concern raised that such replicated tests shouldn't have a
double weight in aggregation. Quan disagrees with the following sentence
from the below agreement: "The first experiment run will provide the
data to be used for model performance evaluation." Note that the ILG may
or may not perform replication.

**Agreement was reached** to insert the following text into the test
plan: "The ILG has the option to open to use some of the secret tests to
replicate experiments (i.e., run viewers through another lab's
experiment). Models' performance evaluation will follow the procedures
laid out above. The data collected will not be considered an additional
experiment for the purposes of model comparison -- i.e., no double
weight for any single experiment. The first experiment run will provide
the data to be used for model performance evaluation. The replication
will be used for other analysis. This data will be shared with
proponents."

On data analysis: proposal to add sentence to section 8.2.2: "The
mapping function will maximize correlation between DMOSp and DMOS while
simultaneously minimizing mean squared error." A decision on this issue
will be delayed until Wednesday, so that Greg Cermak and Irena can be
consulted.

On data analysis: proposal to add sentence to section 8.2.2 (replacing
existing sentence that begins with the same phrase), "If the cubic
monotonic polynomial rescaling does not properly converge for a model,
then a logistic monotonic rescaling will be applied to all the models
ant he best method will be used for each model, but the same method must
be used for a given model across all tests." An email has been sent to
Greg & Irena asking for their opinions on this issue as well.

Questions raised concerning subjective testing software's conversion
from YUV to RGB for image display, which is currently performed by the
display card.

Acreo's subjective test software was demonstrated using NTIA's CIF
resolution min-test.

**Agreement was reached** to add the following text to annex 5:
"Experimenter is required to change the "SubjectID" for each subject in
the MM test."

**Agreement was reached** to change Acreo's software to *not* require a
click after the "okay" & prior to the play of video. After the "okay"
button click, grey should be presented for one second, then the next
video sequence will play, then one-half second of grey, and then the
scoring window shall appear.

A request was made for someone to supply a program that will update the
subject number within the script that Acreo's software uses. Carolyn
Ford volunteered to supply such a program.

Interest was expressed in having an alternative input mechanism, using a
10-key keypad instead of mouse for data input.

Concerns were raised about visual impact of black clipped border against
grey player background having a large impact on subjective scores.
Suggestion made to change player background from mid-level grey to
black. An alternative suggestion is to have a border of black around the
video, e.g., 12 pixels/lines for VGA, 6 pixels/lines for CIF, and 3
pixels/lines for QCIF. This suggestion has a problem in that it would be
a lot of work for Acreo, and Kjell might not have the time and/or
resources. A third suggestion was to use a black background with a grey
cardboard or cloth screen in front to form the mid-level grey
background. The problem with this is that the scoring window would be
larger than the QCIF image size, causing problems.

These suggestions will be posted to the reflector, and a decision to be
made on Wednesday.

MM test plan schedule modified. See post of updated test plan.

JPEG 2000 and/or Motion JPEG 2000 should probably be added to the MM
test plan list of potential codecs.

**Meeting Minutes Wednesday 27. Sep. 2006-09-27**

**Moderated by Kjell & Arthur**

**Note taker: Christian Schmidmer**

New attendee: Mr. Horita from Toyama Univ.

Start 8:39

-   Identification of MM work items not yet completed:

    -   Hybrid approach

    -   Fitting functions (no answer from reflector yet)

    -   Still Video

-   Review of notes from Tuesday

    -   Short repetition of the discussion on experiment replication
        between AW and QHT

## Continuation of MM Testplan Discussion

**Decision was taken** to include JPEG 2000 Part 3 and VC1 into the list
of interesting codecs in the testplan (Testplan V1.14: 6.3.7).

Question from QHT: Do the license statements for the codecs we are going
to use allow us the usage of the codec SDKs for VQEG purpose? No
definite answer can be given by the group, but we don't see the VQEG use
as commercial. Brand names of used codecs will go to the ILGs only.

**Discussion of answeres received via email:**

*Note explaining problem: Spatial scaling is not allowed by the
testplan, but cropping to some extent is explicitly allowed. Prb. is if
the subjects see this as a degradation but the models ignore it.
Ignoring probably reflects the practice of people using commercial
players.*

Antony Rix: Black frames around images are commonly used today in
consumer devices.

Filipo Speranza: Doesn't like option 3 and is in favour of all grey
background. He proposes to avoid black lines in cropped sequences by
e.g. resizing the images or converting the black lines to grey. Black
borders in the SRCs should be avoided.

Phil Coriveau: Option 3 is best (black border in grey background)

**Vote on having a black frame around the images:**

Pro: Acreo, KDDI, Yonsei, NTIA, Swissqual, OPTICOM, Genista, BT, Toyama
Univ.

Against: NTT, CRC

No vote: Psytechnics

-   **Decision was taken** to change the testplan accordingly

Vote on whole screen black: Nobody

Vote on masking screen with cardboard / or cloth: Nobody

Vote on having a black frame around the video (inside the grey
background): KDDI, Yonsei, Swissqual, NTIA, OPTICOM, Genista, INTEL

-   **Decision was taken** to have a black frame on a grey background
    around the images (Testplan V1.14: 4.1.3). Soft wording for the size
    of the frame was appreciated by NTIA since we still want to gain
    experience with the optimal size. An according editors note was
    added to the testplan.

> Agreed upon text: *"Video sequences will be displayed using a black
> border frame (0) on a grey background (128). The black border frame
> will be of the following size:*
>
> *36 lines/pixels VGA*
>
> *18 lines/pixels CIF*
>
> *9 lines/pixels QCIF*
>
> *\[Ed. Note: The size of the black border may change after some
> initial trials of the software change. This will not require 2/3 to
> change size of black border.\]*
>
> *The black border frame will be on all four sides."*

-   Testplan was changed to avoid black lines/borders in SRCs (Testplan
    V1.14: 6.1.5)

> Agreed upon text: *"The source region selection must not include
> overscan---i.e. black borders from the overscan are not allowed.*"

Discussion on black borders in original unprocessed sequences: This is
currently allowed and nobody seems to have a strong opinion on this. It
seems desirable to be more specific in the testplan on this, but no
further action was taken.

**Deinterlacing / Resizing**: Answer from Filipo, that his hardware
system can't be used for the purpose.

**Discussion of hybrid approach:**

Genista asks if side information (e.g. codec, bitrate etc.) should be
supplied to the RR/NR models in the MM test. It was discussed that this
is currently not the target of the MM group and that it is too late for
changing the test plan in this direction. Future projects might however
deal with it.

**Review of the test schedule (continued from Tuesdays discussion):**

**11:30..12:55 Lunch Break**

**Review of the test schedule (continued)**

**Agreed upon version of schedule:**

1.  *Approval of test plan \[14 February 2006\]*

2.  *Declaration of intent to participate and the number of models to
    submit \[6 weeks prior to model submission date\]*

3.  *VQEG compiles a list of HRCs that are of interest to the MM test.
    Proponents will send details of proposed HRCs and indicate which
    ones they can create to the points of contacts and example PVSs (HRC
    point of contacts Quan Huynh Thu and Philip Corriveau). \[24 March
    2006\]*

4.  *Each test lab will produce a test design for each subjective test
    they plan to perforom. This test design should be sent to Filippo S
    and Vivaik B and Margaret P for initial review to ensure no
    duplication. \[Draft test designs produced by Tokyo meeting; test
    design for review by 31 October 2006\] A full and final review of
    test plans will be performed by the ILG and proponent labs. \[30
    November 2006\]*

5.  *Proponents informed by ILG to whom fee payment is made. \[31 May
    2006\] Revision (not necessarily final revision) to list of who pays
    whom to be sent out to all concerned by October 15, 2006.*

6.  *Content license agreements distributed to proponents and ILG. \[31
    May 2006\]*

7.  *Each proponent must have signed (and content provider received) all
    NDAs by October 10, 2006. No guarantees of a three month review of
    the video source will be possible if NDAs are not received by
    content providers by October 10, 2006.*

8.  *The proposed lists of HRCs for each experiment are examined by VQEG
    for problems (e.g., one organization creating too many HRCs, overlap
    between experiments, using NTT guidelines). \[9 June 2006\]*

9.  *ILG send invoice to proponent. \[4 weeks prior to model submission
    date\]*

10. *All source video sent to content point of contact (Chulhee Lee)
    \[29 September 2006\]*

11. *Source video sent by POC to most ILG. \[October 15, 2006\]*

12. *ILG/VQEG will select source video pool files and distribute source
    pool to regional points of contact (Asia: Chulhee Lee, Europe: Kjell
    Brunnstrom, North America: Filippo Speranza) \[ October 15, 2006\]*

13. *Participating organizations obtain copies of source pool, except
    for the secret SRC. The requesting organization has to pay any costs
    involved. \[October 25, 2006\]*

14. *Receiving organization must send acknowledgement to MM reflector
    that they have received the source pool files. When all proponents
    have acknowledged to the MM reflector that they have received all
    source pool, there will be a 3 month period until the submission of
    models. Secret content may still be collected by ILG. Proponents are
    not allowed to provide secret content. \[November 1, 2006\]*

15. *Fee payment if applicable. Payment will be made directly from each
    proponent to the selected testing facility, according to a table
    agreed on by ILG and distributed to the proponents \[same date as
    model submission\]*

16. *Proponents submit their models (executable and, only if desired,
    encrypted source code). Procedures for making changes after
    submission will be outlined in a separate document (see Annex VII on
    storing encrypted version of submitted source code). To be approved
    prior to submission of models. \[9 February 2007\]*

17. *NDAs for secret SRCs distributed and signed by proponents \[2 weeks
    after model submission Feb 23, 2007\].*

18. *ILG selects and distributes all SRC used for each experiment,
    secret SRC, backup SRCs, and common set of PVSs to be included in
    [every]{.underline} experiment, as proposed by NTT (e.g., 5 SRC & 5
    HRC, which would be 25 of 160 video sequences or 15%). This is the
    step where the ILG (or other organization) must have deinterlaced
    and resized the 12 second source video sequences. \[3 weeks after
    model submission March 2, 2007\]*

19. *Proponents and ILG inform VQEG of any problem source content.
    Problem content must be made available on ftp site and reviewed by
    proponents / ILG. Majority decision needed to reject suspect source.
    \[6 weeks after model submission March 23, 2007\]*

20. *Organizations will generate the PVSs and check calibration using
    the scenes that were sent to them and send all the PVSs to a common
    point of contact. \[9 weeks after model submission April 13, 2007
    \]*

21. *All PVSs are sent by each test laboratory to a regional point of
    contact. Point of contact then distributes PVSs to all the
    proponents \[11 weeks after model submission 27 April 2007\]*

22. *Proponents check the calibration and registration of the PVSs in
    their experiment. \[13 weeks after model submission 11 May 2007\]*

23. *If a proponent testlab believes that their experiment is unbalanced
    in terms of qualities or have calibration problems, they may ask the
    ILG and the proponent group to review the selection of test
    material. If 2/3^rd^ majority agrees then selection of PVSs will be
    amended by the ILG. An even distribution of qualities from excellent
    to bad is desirable. \[15 weeks after model submission 25 May
    2007\]*

24. *Proponents check calibration of all PVSs and identify potential
    problems. They may ask the ILG to review the selection of test
    material and replace if necessary. If a proponent or ILG does not
    review the test content, then they lose the right to object to the
    content composition of that test. \[15 weeks after model submission
    25 May 2007\]*

25. *Proponents run their models and the ILG performs validity checks on
    a subset of test sequences. \[21 weeks after model submission 6 July
    2007\]*

26. *Each organization runs their test and submits results to the ILG.
    Any source content used in a subjective test with a MOS of \<4 will
    be evaluated by the ILG. The ILG will determine whether the source
    and its associated processed files are valid. Any invalid test
    content will be removed before data analysis is performed and before
    proponents submit their objective data to the ILG. Subjective test
    finished by \[23 weeks after model submission 20 July 2007\]*

27. *Verification of submitted models \[23 weeks after model submission
    20 July 2007\]*

28. *ILG distribute subjective and objective data to the proponents and
    other ILG \[24 weeks after model submission 27 July 2007\]*

29. *Statistical analysis \[31 weeks after model submission 14 September
    2007\]*

30. *Draft final report \[35 weeks after model submission 12 October
    2007\]*

31. *Approval of final report \[38 weeks after model submission 2
    November 2007\]*

**Discussion on still videos:**

Still videos are currently neither explicitly allowed or disallowed by
the testplan.

**Vote on having (completely) still SRCs in the test:**

**Pro**: nobody

**Against**:KDDI, Yonsei, Swissqual, OPTICOM, NTT, IRCCYN, INTEL

**Neutral**: Psytechnics

**Decision was taken** to change the testplan was changed accordingly.

Agreed upon text: *"There will be no completely still video scenes in
the test" (V1.14: 6.2)*

**The schedule will be postet on the FTP site and an email for
notification will be sent out. Comments (on the schedule only) are
required by Oct. 13^th^.**

**END OF MM MEETING (14:00)**

**Start of RR/NR TV Discussion 14:00**

Change of moderator to Alexandre Bourrett

Changes on the testplan required due to newly added codecs (H.264, VC1):

-\> Models have to be retrained for new codecs and some proponents are
concerned of having several codecs in the same test.

The following concerns were raised:

-   If there is no such information, the models may all fail.

-   If the information is provided, the amount test data used for
    validation are not sufficient

-   General models which don't need such information are appreciated,
    but the comparison with models that use such information is unfair.

It was also proposed to start a project on hybrid models.

Existing solutions like MDI and V-Factor were mentioned

Current standardisation in ITU-T SG12 was mentioned.

Market requirements were briefly discussed (many channels in parallel in
realtime using NR technology)

**Question by AW**: How many parties are interested to supply a model
for which type of model:

  ------------- ----------- ------------- ------------- -----------------
                No Info (A) A + Codec = B B + Bitrate = C + Bitstream = D
                                          C             

  RR            4           3             3             5

  NR            1           3             1             7
  ------------- ----------- ------------- ------------- -----------------

CL points out that the ITU Focus group will come out with a
recommendation on IPTV soon.

**Coffee break**

During the break it was discussed that an additional Model "E" might be
required which is lightweight and takes only transport parameters into
account (like MDI or V-Factor). The combination of E (realtime
information on transport quality) and A (non-realtime information on
video quality) was reported by Swissqual as a market need. ITU-T SG12
Q14 is working on model type "E".

Vote:

**Vote: Do we want to change the RRNR testplan to accommodate models
that require additional sideinfo?**

Pro: BT, Acreo,

Against: KDDI, Yonsei, NTIA, Swissqual, OPTICOM, NEC, Toyama,
Psytechnics

Neutral: IRCCYN

**Decission was taken** NOT to change the testplan

**Vote: Does anybody object to project D being a separate project?**

No objections, testplan remains unchanged on this.

NEC wants to be a proponent for RRNR-TV. The number of models they are
going to submit is not yet decided upon.

NEC has no opinion yet on the test size

Toyama Univ. is fine with the current test size

BT is fine with the current test size.

NTIA is fine with the current test size.

Yonsei is fine with the current test size.

Genista and TDF are not present

Psytechnics and OPTICOM raise their concern that the test is too small.

**Proposal** to amend 3.2.1 bullet point 4 in a way that it reads Video
material from FRTV phase II Tests [and Multimedia Test]{.underline} may
be used

**This was with no objections approved**

**5:15 Meeting adjourned for the day**

VQEG meeting minutes (hours), September 28 2006, Tokyo

Weather: sunny

Arthur's mood: cloudy

**Decision**: Change to MM testplan -- added to 8.2.4 "ILG experiments"
to clarify replication of experiments section. Quan, Pero, Chulhee will
write a proposal regarding choice of algorithms to be used.

**Decision**: Change to MM testplan to add "Mapping algorithms used for
official data analysis must be documented"

Finish RR discussion.

**Decision**: Agreement to use NDAs signed for MM test for RRNR-TV

**Decision**: Deadline for model submission: April 9 '07. No dissenters.

More **revisions**/additions of schedule, see RRNR testplan section 6.

Re-re-re-visit the MM testplan statistical analysis question, again, one
more time.

**Decision** to remove logistic rescaling option. Alter sentence
slightly.

> Vote:
>
> For: KDDI, Acreo, NTIA, Yonsei, Swissqual, Opticom, NTT, Tektronix,
> Psytechnics.
>
> Neutral: ear-sin
>
> Opposed: none.

**Change** the 2^nd^ decision above to replace "documented" with
"referenced."

**Add** line to schedule, before Statistical Analysis item: "Optional
submission of mapping coefficients by proponents due to ILG. 28 weeks
after model submission."

**Add**: Proposed paragraph by Chuhlee, Quan, Pero and Christian for
section 8.2.2, regarding the computations of mapping functions. No
dissenters.

HDTV test plan discussions.

Presentation/proposal from NTT : "Proposal to add the packet-loss
condition in HDTV test plan" Vote:

> For: KDDI, NTIA, Acreo, Yonsei, SwissQual, Opticom, NEC, NTT,
> Tektronix
>
> Against: BT, IRCCyN
>
> **Proposal accepted**

Proposal for definition for transmission errors: add "types of errors
will include packet errors such as packet loss (both IP and Transport
Stream), packet delay variation, jitter, overflow and underflow, bit
errors, and over the air transmission errors. Error concealment and
forward error correction should be included in at least some of the
HRC's." Vote: no dissenters.

Note added to plan that source and HRC's should be captured with audio
when possible in preparation for follow-on phases.

**Action item:** Pero will generate user's guide for using Ethereal by
next meeting.

Review of changes generated by audio conferences.

**Decision**: Add possible frame rates-- 1080i/50 & 60hz/30fps, 720p
50/60fps, 1080p. Accepted, no objections.

Accepted changes discussed in audio conferences. Minor wording
adjustments.

**Action Item**: Patrick (IRCCyN) will investigate requirements for
transparent display.

Scene change reqt, change to what's "typical" for the content.

Switch to discussion about new project for the "hybrid" RRNR concept.

Chulhee will be one chair (no dissention)

Pero will be co-chair (no dissention)

Chris will help maybe

Name: Hybrid -- Perceptual/Network (Opticom, SwissQual)

Hybrid -- Perceptual/Bitstream (6 votes) \<- **Decision**

Return to HDTV discussion, continue accepting changes from conf call

**Decision**: Remove list in 2.1.3. (for: Opticom, BT, Tektronix,
IRCCyn, against: SwissQual)

Camera spec decision delayed until next meeting. See section 4.3 for
verbage.

4.4.3, delete sentence about frame skipping in 1^st^ and last second.

Frame rate conversion as an hrc needs to be checked, esp 720p to 20 or
25fps

Number of viewers is TBD

End of day...

QED: Quit and Eat Dinner

**Friday**

**NO Agreement was reached** (for MM test plan) to add the following
text when computing polynomial coefficients to "maximize correlation
between DMOSp and DMOS (over a, b, and c) and then use a linear function
to minimizing mean squared error (constant multiplier k for a, b, and c;
then determination of d). This will produce a' = k\*a, etc." A long
discussion ensued, in which complicated details were considered for
inclusion in the test plan.

The multiple party NDA covering the MM test plan was discussed. NTT
requires this confidentiality agreement. **Agreement was reached** to
have all MM, RRNR-TV, and HDTV proponents & ILG sign a multiple party
NDA, so that the data sets and other data can be protected by a
confidentiality agreement. All organizations should run this
confidentiality agreement by their lawyer within one month, and report
requested changes. MM test plan schedule revised to mention this
multiple party confidentiality agreement.

**Agreement was reached** on how to phrase "maximize correlation and
minimize RMSE". See MM test plan.

**New MM Content Currently Available**

Video Conferencing -- NTIA, NTT , Yonsei possibly,

Movies & Movie Trailers -- Psytechnics, SVT, Opticom

Sports -- Yonsei, SVT, Opticom, Psytechnics

Music Video -- NTIA

Advertisement -- Psytechnics, Opticom

Animation -- Opticom, NTIA,

Broadcasting News -- KBS, Opticom,

Home Video -- SwissQual, Yonsei, NTIA,

If anyone has a suggestion on how to obtain music video (e.g., like seen
on M-TV), please contact the ILG. This footage would be greatly
appreciated for secret scenes.

**HDTV**

**Agreement was reached** that different tests should be conducted by
each lab (different PVS\'s).

**Agreement was reached** to modify data analysis changes from MM
(Section 8) into HDTV (section 6), and replace MM references with HDTV
reference\\s.

**Action item**: please bring examples to the next meeting, showing the
worst quality of transmission errors VQEG would like to see in the HDTV
test plan.