I N S T R U C T I O N S F O R J U D G I N G V I D E O S U M M A R I E S

B a c k g r o u n d

NIST is evaluating experimental systems that attempt to create video

summaries - compressed versions of longer videos. Such summaries are

very useful in many situations because they allow someone to get a

good idea of what the video is about much faster than watching the

whole video.

The data we are using to test the systems is so-called \"rushes\" video

from several BBC TV series. Rushes are the raw material from which

final programs are made and can be 20-40 times larger than the final

program. This is because rushes contain lots of views of the same

objects and events (e.g., from different distances and angles) for the

program editors to choose from and retakes necessitated by various

problems (e.g., actors getting lines wrong, trouble with props).

Your job is to use some software (the assessment system) to help us

judge how good a job the summarization systems are doing.

A s s e s s m e n t t a s k

Each video to be summarized has been examined and a list of important

segments has been created, in which each such segment is identified

by an object or event in that segment.

Each of you will judge a couple dozen summaries for each of 12 full
videos - grouped by the full video being summarized.

Each time you are ready for a new group of summaries, the system

will remind you to do two things:

1\) study the list of inclusions. This means read the list

carefully at least twice until you have the differences

between the included items you are looking for well in

mind

2\) play (at \~ 5 times normal speed) the full video being\
summarized. This is to allow you to become familiar with

the material, the different shots, angles, etc.

3\) Then for each of the summaries assigned to you, your task will be

to:

a\) watch the summary **just once** (pausing as much as needed) and
check

off which items on a list of important segments from the full

video are included in the summary.

**NOTE**: The time required to check for the included segments will

be recorded by the assessment system so it's very important that

you not start the inclusion checking until you're ready and that

you concentrate only on that procedure until you're done.

b\) answer two questions about the quality of the summary on a

5-point scale from \"Strongly agree\" to \"Strongly disagree\"

describing the degree to which you (dis)agree with the

statement:

1\) \"**It is easy to see and understand what is in this summary.\"**

This is an overall judgment of the summary and how it presents

the information.

2\) **\"This summary contains many nearly identical segments.\"**

> This an overall judgment about how many **nearly identical** segments
> are contained in the summary. Remember -- different views (angles,
>
> distances, etc.) of the same event should not count as nearly
> identical.

P r o c e d u r e

1\) Login

Type in you userid and password. Then press the \"Enter\" key.

2\) Start video summary assessment system at beginning of day\'s session

![](media/image1.png){width="0.3333333333333333in"
height="0.3333333333333333in"}

Click **ONCE** on the icon with the red triangle labeled \"Video

Summary Assessment\" at the bottom of the screen - the summary

assessment system will appear.

If you are starting a set of summaries for a new video, then a

message box will appear reminding you to play the full video and

study the inclusion descriptions before starting to judge. Click

on the \"OK\" to remove the message box.

![](media/image2.jpeg){width="2.6875in" height="2.7708333333333335in"}

If the video window (upper left) is not dark, wait until it is -

sometimes up to a full minute the first time for the day. If it does

not go dark, seek assistance before proceeding

3\) Judge summaries

a\) If you are starting a group of summaries for a new video,

you need to do two things before beginning to judge:

i\) Study the inclusion list on the right. Read it through at

least twice, noting the details of each item. Familiarize

yourself with each of the descriptions, but don\'t attempt to

memorize them.

ii\) Click on the button labeled \"Play full video\" . This will

cause the full video being summarized to play at about 5

times normal speed. This will take up to 7 minutes. Study

the video to see what the shots are about and what different

angles, distances, and camera motion are included. Compare

what you see to the list of inclusions you will be looking

for. Let the whole video play.

![](media/image3.jpeg){width="6.0in" height="4.5in"}

b\) When ready to start the timed judging of inclusions click once

with the mouse on the button labeled \"Start judging inclusions\".

This starts the clock.

c\) **You will only be able to watch the summary once!** But you can

pause as many times as you want. If you click in the inclusion

window anywhere outside the check boxes or associated text while

the summary is playing, it will pause. If you click there while

it is paused, it will start running again.

d\) While watching the summary once, pausing as much as needed,

click on all items in the inclusion list that you see in the

summary. Be careful not to click on an item unless you see

video that meets the full item description.

e\) When the summary has played to the end and you have finished

marking the inclusions, immediately click once with the mouse

on the button labeled \"Done judging inclusions\". This stops

the clock.

![](media/image4.jpeg){width="6.0in" height="4.5in"}

f\) Answer the 2 questions in the lower left quadrant

and then click once with the mouse on the button

labeled: \"Done judging usability\". This records all

your judgments and gets the next summary ready for

judging.

![](media/image5.jpeg){width="6.0in" height="4.5in"}

g\) If you want to take a break, this is the time in the

assessment cycle to do that. Just leave the assessment

system as is until you return. Then begin again at

step 3a) above.

If you are leaving for the day, go to step 4 below

If you are ready to judge the next summary now, then begin

again at step 3a) above.

4\) To exit video summary assessment system at end of day\'s session

Click once with the mouse on the button labeled

\"Exit\"

5\) To log out

Right mouse click and choose \"Logout\"

R e m i n d e r s:

Read the descriptions of the inclusions carefully. Do not mark them

unless what you see meets the description.

Camera motion:

\- \"Pan\" means pivoting the camera left or right.

\- \"Tilt\" means gradually pointing the camera more up or down

-\"Zoom out/in\" causes the picture to get larger/smaller, seem

closer/farther

Pan, tilt, zoom are **actions** that involve a change of view.
\"Closeup\"

**is not an action** - it just means a view from close. If the inclusion

description asks for a zoom then just a closeup will NOT do.

BBC standards may not be the same as yours. If you encounter material

you would rather not see, please squint, get past the problem, and let

us know it if becomes a nuisance.