NTIA Handbook HB-06-434 

In-Service Video Quality Metric (IVQM) 

User’s Manual 

Margaret H. Pinson 

Stephen Wolf 

NTIA Handbook HB-06-434 

In-Service Video Quality Metric (IVQM) 

User’s Manual 

Margaret H. Pinson 

Stephen Wolf 

U.S. DEPARTMENT OF COMMERCE 

Carlos M. Gutierrez, Secretary 

Michael D. Gallagher, Assistant Secretary 

for Communications and Information 

December 2005 
 

 
 

DISCLAIMER 

Certain commercial equipment, software packages, and materials are identified in this report to

specify adequately the technical aspects of the reported results. In no case does such 

identification imply recommendations or endorsement by the National Telecommunications and 

Information Administration, nor does it imply that the material or equipment identified is the 

best available for this purpose. 

The software described within was developed by an agency of the U.S. Government. NTIA/ITS 

has no objection to the use of this software for any purpose since it is not subject to copyright 

protection in the United States. 

No warranty, expressed or implied, is made by NTIA/ITS or the U.S. Government as to the

accuracy, suitability and functioning of the program and related material, nor shall the fact of 

distribution constitute any endorsement by the U.S. Government. 

 iii

 

CONTENTS

Page 

1 INTRODUCTION .....................................................................................................................1 

2 REQUIREMENTS.....................................................................................................................3 

3 INSTALLATION PROCEDURE..............................................................................................5 

4 OPERATIONAL OVERVIEW .................................................................................................6 

4.1 Before Starting IVQM ................................................................................................... 6 

4.2 Starting IVQM ............................................................................................................... 6 

4.3 Setup Screen................................................................................................................... 7 

5 IVQM MONITORING SCREEN............................................................................................16 

5.1 Time History Plot of VQM .......................................................................................... 17 

5.2 Frames from Last Video Capture................................................................................. 17 

5.3 Text Box Depicting Results......................................................................................... 18 

5.4 Status Bar..................................................................................................................... 19 

5.5 Menu Options............................................................................................................... 19 

5.6 Buttons......................................................................................................................... 20 

6 KNOWN ISSUES, BUGS, AND DEFECTS ..........................................................................22 

7 REFERENCES ........................................................................................................................25 

FIGURES

Page 

Figure 1. Snapshot of initialization screen...................................................................................... 6 

Figure 2. Snapshot of setup screen, master computer..................................................................... 8 

Figure 3. Snapshot of setup screen, slave computer. ...................................................................... 9 

Figure 4. Snapshot of IVQM monitoring screen, master computer.............................................. 16 

Figure 5. Snapshot of IVQM monitoring screen, slave computer. ............................................... 17 

 v
 

 

IN-SERVICE VIDEO QUALITY METRIC (IVQM) USER’S MANUAL 

Margaret H. Pinson and Stephen Wolf1

The purpose of this handbook is to provide a user’s manual for the inservice video quality metric (IVQM) tool. IVQM performs automated 

processing of live video signals. This program runs under the Windows 

XP® operating system on two PCs communicating through an IP

connection. IVQM performs image acquisition, temporal registration, 

other video calibration (spatial registration, spatial scaling, valid region, 

and gain/level offset), and video quality estimation. 

IVQM compares the source video sequence to the destination video 

sequence (i.e., as output by the video system under test). Each program 

alternates between video capture and video analysis. Every

source/destination video sequence pair is processed through three main 

steps. First, the sequences are buffered onto a hard drive. Second, the 

sequences are temporally registered. Third, the video quality of the 

destination video sequence is estimated. Quality estimates are reported on 

a scale of zero to one, where zero means that no impairment is visible and 

one means that the video clip has reached the maximum impairment level.

Some video sequences may also be used to estimate other calibration

values (spatial registration, spatial scaling, valid region estimation, and 

gain/level offset). The user has control over how often these other

calibration values are calculated. 

Key words: automatic measurements; digital video; end-to-end; in-service; metrics; 

objective video quality performance; video calibration; video quality 

1 INTRODUCTION 

The Institute for Telecommunication Sciences (ITS) has developed an automated inservice video quality metric (IVQM) software tool that performs automated processing of 

video signals. Most of the video quality measurement algorithms implemented by the 

IVQM software tool are described in detail in [1]-[4]. These algorithms include

calibration of the sampled video streams (e.g., gain/level offset, spatial scaling, spatial 

1

 The authors are with the Institute for Telecommunication Sciences, National Telecommunications and 

Information Administration, U.S. Department of Commerce, 325 Broadway, Boulder, CO 80305.

 
 

registration, valid region estimation, and temporal registration), as well as the calculation 

of video quality parameters and models of overall quality perception. The purpose of this 

document is to provide a user’s manual for the IVQM tool. 

Program IVQM runs under the Windows XP operating system and requires two PCs

connected to an IP network. Each PC must have a video capture device. These devices 

must be compliant with Windows Driver Model (WDM), Video for Windows (VfW) or

another driver interface supported by the MATLAB® image acquisition toolbox (see 

http://www.mathworks.com/access/helpdesk/help/toolbox/imaq/imaq.html). 

The “source” PC will capture and process the source video sequence. The “destination” 

PC will capture and process the destination video sequence (i.e., as output by some video 

transmission system). The pair of computers will communicate results over an IP 

network connection. IVQM alternates between image acquisition (i.e., video capture) 

and video analysis. The video analysis requires significantly more time than does the 

video capture. Video quality results are plotted on a graph and also presented to the user 

in an ASCII text report. The video sequences themselves are discarded. 

 2
 

2 REQUIREMENTS 

IVQM requires two computers, each meeting the following minimum criteria:

• 2 GHz Pentium

• Windows XP, Service Pack 2 

• 1.5 GB RAM 

• 4 GB free disk space 

• FTP server or shared directory 

• WDM, VfW or other MATLAB supported video capture device 

• Time synchronization program (for video delay estimates) 

IVQM requires the use of two computers. Each computer must have in excess of 1 GB of 

free memory that is available for use by the IVQM application. A minimum of 2 GB of 

memory is recommended to meet this requirement. Each computer must have a local 

hard drive with a minimum of 4 GB free disk space. The speed of these disk drives will 

impact performance. 

Warning: Insufficient disk space or computer memory will prevent IVQM from 

running successfully. 

The two computers must be connected to a common IP network. The computers can 

coordinate activities using one of two methods. The first method is to utilize a shared 

directory. The second method is to transmit files using FTP. When communicating by 

FTP, both computers must have a local FTP server installed and running. The remote 

FTP clients must be able to read, write, and delete files using the local FTP server. 

Each computer must have a compatible video capture device. The performance and

specifications of the video capture device will impact the accuracy of the video quality 

estimates produced by IVQM. At a minimum, the video capture system should be able to 

capture video at full frame rates (e.g., 30 fps, 29.97 fps, 25 fps) for the desired spatial 

resolution. The capture device must be WDM or VfW compliant, or be otherwise 

supported by the MATLAB image acquisition toolbox. The video capture device must 

provide video to the PC in either the YCbCr (i.e., YUV) or RGB color space. Use 

matched video capture devices on both computers (i.e., the same brand and model). 

Otherwise, IVQM will be measuring the difference in quality of the frame capture 

devices. These video quality metric (VQM) differences may be substantial (e.g., 0.25 

VQM). Five video capture devices that meet these requirements are: 

• Leadtek’s “Winfast TV USB-2 Interface TV/FM Tuner Box” 

• Leadtek’s “Winfast TV USB II Deluxe” 

• Hauppauge’s “Impact VCB” part# 64405, model# 00558 

• Hauppauge’s “WinTV-USB2” 

• Pixel Magic Systems “SDI Capture Card” 

 3
 

Devices that support uncompressed video capture utilizing the Brooktree (Bt) Bt848, 

Bt849, Bt878 and Bt879 chipsets may work if the WDM video capture driver at the 

following location is installed: http://btwincap.sourceforge.net/ If you identify a 

compatible video capture device other than those listed above, please let us know 

(vqm@its.bldrdoc.gov). 

If the user desires to obtain useful estimates of video delay, each computer must have a 

program for synchronizing the system clocks of the source and destination computers 

with a time standard. One tool that satisfies this requirement is the NIST Time Program

[5]. 

MATLAB does not need to be installed on either computer. As part of the installation 

procedure, the MATLAB Component Runtime library will be installed (see Section 3). 

As a final note, some web cameras are compliant with the video capture device interface. 

Although these cameras can be used to explore the setup screen, they cannot be used for

actual video quality estimation. Unfortunately, the camera signal cannot be 

simultaneously routed to both the IVQM program and the video transmission system

under test. 

 4
 

3 INSTALLATION PROCEDURE 

The IVQM software was developed using MATLAB and its associated toolboxes. It is 

therefore necessary to install the MATLAB Component Runtime (MCR) library before 

running IVQM. 

If a prior version of IVQM was installed, you must first uninstall the old version of the

MCR library before proceeding. To uninstall the old version of the MCR library, select

“Start,” “Settings,” “Control Panel” and then double-click on “Add or Remove

Programs.” Find the MATLAB Component Runtime library in the list of programs and 

select “Remove.” 

Follow this installation procedure for IVQM: 

1. Copy the files on the IVQM installation CD to a directory on your computer. This 

directory will be denoted as c:\ivqm for the rest of the installation instructions given 

below. 

2. Double click MCRInstaller.exe in c:\ivqm and follow the instructions to install the 

MCR library on your computer. 

3. After completing installation, check to make sure that the MCR library installed 

properly. From the “Start” menu, select “Control Panel,” and in the window that appears, 

double-click “Add or Remove Programs” and see if the MATLAB Component Runtime 

library appears in the list of installed programs. If not, repeat step 2. 

 5
 

4 OPERATIONAL OVERVIEW 

4.1 Before Starting IVQM

Before starting IVQM, synchronize the clock of your computer with a time standard 

using a utility such as the NIST Time Program [5]. This action should be performed 

immediately prior to starting IVQM. This synchronization is necessary for IVQM to 

correctly display the video system’s delay to the screen. The time synchronization should 

be performed multiple times for greater accuracy, to be certain that the computer’s 

internal clock is correctly synchronized with a time standard. If the system clocks are not 

synchronized, the only impact to IVQM will be inaccurate reporting of the end-to-end

video delay. 

4.2 Starting IVQM 

Open a command prompt window by selecting “Start,” “All Programs,” “Accessories,” 

“Command Prompt.” Change to the c:\ivqm installation directory in step 1 of section 3 

by typing “cd c:\ivqm” at the command prompt. Alternately, you may set up your 

command window to start in directory c:\ivqm. 

To start the IVQM software, type “ivqm” at the command prompt. The IVQM software 

will start and display the screen shown in Figure 1. The warning message is normal and

will not affect the performance of the IVQM software. See section 4.1 before starting the 

IVQM software. 

Figure 1. Snapshot of initialization screen. 

 6
 

Note: After starting IVQM at a command prompt, please wait patiently for the 

initialization screen to appear. IVQM needs to load several dynamic linked libraries into 

memory.

Program IVQM shares tasks between the two computers. One of the two computers will 

be the master, while the other computer will be the slave. Some user interface controls 

are only available on the master computer (e.g., video quality model selection). The 

slave computer will receive these settings from the master computer. Other user interface 

controls must be set on each computer separately (e.g., how to communicate with the 

other computer). The video quality analysis displayed to the user will be identical on 

both computers. 

Figure 1 shows a snapshot of the initialization screen presented to the user when IVQM 

starts. At this point, the user must select two choices using radio buttons: (1) master or

slave, and (2) source or destination (video). When the options are set for one computer,

the complimentary options must be selected for the other computer (i.e., if one computer 

is the master / source computer, then the other computer must be the slave / destination

computer). Once these choices have been made, press the “start” button on each

computer. 

Note: After pressing the “start” button, please wait patiently for the setup screen to

appear. IVQM needs to examine all video capture options, and this might take several 

minutes. 

4.3 Setup Screen 

The setup screen may be reached by pressing “start” from the initialization screen, or 

selecting “setup” from the Edit menu option on the monitoring screen. The master setup 

screen is divided into four sections (see Figure 2) and the slave setup screen is divided 

into two sections (see Figure 3). All of the items in Figure 2 and Figure 3 are explained 

in Sections 4.3.1, 4.3.2, 4.3.3, 4.3.4 and 4.3.5. 

Most of the options on the setup screen are selected using pull-down menus. Each pulldown menu has a small gray down-arrow on the right hand side. Selecting the pull-down 

menu will present the user with a list of the available options. All other user inputs to the

setup screen are text boxes. The keyboard shortcut “<CTRL>v” may be used to paste 

text into these text boxes. 

Throughout this section, the two computers will be referred to as the “local” and 

“remote” computers. The local computer is the computer where setup is being 

performed, and the remote computer is the other computer. 

 7
 

Figure 2. Snapshot of setup screen, master computer. 

 8
 

Figure 3. Snapshot of setup screen, slave computer. 

4.3.1 End-To-End Communication Medium 

There are two possible choices for end-to-end communication between the two 

computers. When communicating through a shared directory, the user has only to specify 

the shared directory path. The drive holding this shared directory must have several 

gigabytes of free space. 

When communicating through FTP, the user must specify the following five pieces of 

information: 

1. Local FTP Site Path. The directory path to the local FTP site’s directory where 

the remote computer will be writing data. The drive holding this directory should 

have several gigabytes of free space.

2. FTP Host. The remote computer’s FTP Host (e.g., name or address). 

3. FTP Username. The username that should be entered to establish an FTP 

connection. For anonymous FTP, enter the username “anonymous.” 

 9
 

4. FTP Password. The password associated with the above username. This field 

should be left blank if there is no password for the remote computer’s FTP 

account (e.g., anonymous FTP). 

5. FTP Sub-Directory. After connecting to the remote computer’s FTP host, 

change to this sub-directory. If no change of directory is required, then this field

should be left blank. 

When all of the above five options have been specified, press the “Validate FTP 

Communication” button to check the FTP connection. This is an optional check. The

same validation will be performed when the “Accept Setup” button is pressed. 

4.3.2 Video Source 

The choices for video source will depend upon the number of compliant devices available 

on your computer. Any device listed has met some – but perhaps not all – of the 

compliance criteria. To run using a video capture device, select the “Video Capture 

Device” option. If no compatible video capture devices are available on your system, 

then this option will not be available. The alternate choice for video source is 

“Validation File.” 

When capturing video, the user must specify the following five pieces of information. 

Start at the top, with “Video Capture Device” and work down. The first four pieces of 

information are interdependent, so changes may automatically modify other values. 

1. Video Capture Device. The name of the video capture device, as reported to the

computer by the device driver. All compliant video capture devices will be listed 

on the pull-down menu. Warning: Use matched video capture devices on both 

computers (i.e., the same brand and model). Otherwise, IVQM will be measuring 

the difference in quality of the frame capture devices. These differences may be

substantial (e.g., 0.25 VQM). 

2. Image Size & Color Space. Select the correct format information for the 

incoming video signal. Sometimes the driver will report this in terms of image 

size (e.g., 720x480) and color space (e.g., RGB). The strings used here are 

reported directly from the video capture device’s driver. Warning: The image 

size selected must be identical on source and destination computers. Use the 

same color space on both computers, so that any color space conversion errors are 

introduced equally in both source and destination video sequences. YUV is the 

preferred color space for video capture (e.g., “YUY2” or “YUV” or “UYVY”). 

3. Video Source. If your device has multiple input video stream options, select the 

appropriate port. The strings used here are reported directly from the video 

capture device’s driver. 

4. Capture Interlaced or Progressive. The IVQM program initializes this field

with the current best guess based upon the image size specified above. 

 10
 

Unfortunately, the device driver does not report this required information directly. 

After specifying this (and the above three options) the user is strongly encouraged 

to press the “Preview Video Stream” button to check the validity of this setting.

The options available are “interlace, lower field first,” “interlace, upper field 

first,” and “progressive.” The interlace options are for video displays which 

separate each frame into two fields and alternately update each field; the 

progressive option is for systems that update the entire frame simultaneously. 

Warning: This option must be set identically on both source and destination 

computers. 

5. Directory for Temporary Storage. The IVQM program requires up to 2 GB of 

disk space to buffer video sequences. Run speed will be impacted by the speed of 

the hard drive containing this directory. 

When all of the above five fields have been entered, press the “Preview Video Stream”

button to display a short sample of the selected video stream in a separate window. If 

“Capture Interlaced or Progressive” is progressive, the video will be played back one

frame at a time. If “Capture Interlaced or Progressive” is interlaced, the video will be 

played back one field at a time using field freeze (i.e., one field is copied into both fields 

of the displayed frame). IVQM will beep and pause one second after displaying each 

field or frame. The preview window may be manually closed at any time and will be 

automatically closed when the “Accept Setup” button is pressed. 

This “preview video stream” button demonstrates whether or not the video capture device 

will operate as intended (e.g., whether the video capture device is fully compliant and 

capturing the video signal correctly). Pressing the “preview video stream” button can be 

used to validate the “Capture Interlaced or Progressive” flag. If an interlaced / field 

freeze playback sequence appears to jump backwards in time, change from lower field 

first to upper field first (or vice versa). 

The alternate choice for video source is “Validation File.” This run mode is intended for 

validation purposes only. Full functionality of the IVQM program is only available when 

using a compliant video capture device. When running from a validation file, enter the 

full path and name of the validation file. 

4.3.3 Calibration

The accuracy of IVQM is significantly impacted by the accuracy of the calibration

estimates. Therefore, care should be taken when selecting the calibration options. 

The accuracy of calibration estimates can be improved significantly by combining 

calibration estimates from several different sequences. The automated calibration 

algorithms have been optimized for fast run-speed and lower bandwidth. The cost of this

optimization is a slight decrease in accuracy. To compensate, we highly recommend 

choosing to run the optional calibration algorithms and to combine the results from 

multiple video sequences. See [4] for details on the automated calibration algorithms. 

 11
 

Calibration options are only available on the master computer. 

4.3.3.1 Combined Temporal Registration Uncertainty 

This option specifies the maximum delay uncertainty of the video transmission system, 

plus the maximum delay uncertainty of the end-to-end IP communication between the 

two computers. This uncertainty is increased by the random nature of coordinating a 

“record video now” request between two computers with independent clocks via an IP 

communication channel. The “combined temporal registration uncertainty” will be used 

together with the “delay between source & destination capture” (see Section 4.3.3.2) to 

synchronize video capture at the source and destination computers. 

The combined temporal registration uncertainty determines the search window that is 

examined to temporally register the source and destination video sequences. This must 

be done every time a video sequence is captured. If the correct temporal registration lies 

outside of the uncertainty searched, then accurate video quality estimations cannot be 

calculated since the source and destination video streams cannot be temporally registered. 

However, there are drawbacks to choosing a too-large value. As uncertainty increases, 

IVQM will require longer periods of time and more memory to estimate video quality, as 

additional video must be captured to allow for the temporal registration uncertainty. 

4.3.3.2 Delay Between Source & Destination Capture 

This pull-down menu specifies a known delay through the video transmission system. 

The known delay will be associated with the combined time taken by coding, 

transmitting, and decoding the video sequence. This pull-down menu is by default zero 

seconds. Larger values are appropriate for a system where the user knows the delay 

through the system under test is large (e.g., 15 seconds) and thus the combined temporal 

registration uncertainty (section 4.3.3.1) cannot be made large enough to include the 

known delay. When this menu is set to a non-zero value, the destination capture will

begin approximately this number of seconds after the source capture. 

4.3.3.3 Scale, Shift, Gain/Offset & Valid Region 

This pull-down menu specifies how often IVQM will estimate the calibration values (i.e.,

other than temporal registration). These calibration algorithms estimate spatial scaling, 

spatial shift, luminance gain & offset, and valid video region. Spatial scaling detects

transmission systems that spatially stretch or shrink the video picture. Spatial shift 

detects transmission systems that shift the video spatially (e.g., up or down). Luminance 

gain & offset are equivalent to detecting changes in the contrast and brightness of the

destination video with respect to the source video. Valid region estimation eliminates the 

edges of captured video signals that do not contain valid picture information. 

The above calibration values are normally constant for any given video transmission 

system. Any time-varying deviations from the overall, constant value are considered 

 12
 

impairments. For example, if the spatial shift were to change dynamically over time, the 

video picture would appear to bob up and down. Because these calibration values are 

normally constant for a given video system under test, more accurate estimates can be 

obtained by combining calibration results from multiple video sequences. 

Currently, there are six calibration options, listed below. Items to consider when 

selecting the calibration option include prior knowledge of the calibration parameters for 

the video system under test, the amount of IP bandwidth that is available to transmit the 

calibration information (i.e., calibration requires approximately 2 Mbits/sec for each

video sequence that is calibrated), and whether or not the calibration parameters for the 

system under test are changing. The “scale, shift, gain/offset & valid region” calibration 

options are: 

1. Do not calculate. Do not estimate the above calibration parameters. IVQM will 

run faster when this option is selected. However, the accuracy of the video

quality estimations will be negatively impacted if the video system under test has

calibration problems. 

2. Calculate at the beginning & combine all results. The calibration parameters 

will be calculated using the first few video sequences captured. These results will 

be combined to form more accurate estimations, which will be applied to all later 

sequences. The number of results to be combined must also be specified, using

the “Number of Sequences Used” pull-down menu. Greater calibration accuracy

should result when more sequences are used. This mode is preferred when 

watching the performance of a single transmission system over a long period of 

time. 

3. Calculate periodically & combine all results. The calibration parameters will 

be calculated at a set interval (e.g., every 15 minutes). All results will be 

combined to form more accurate estimations. Thus, the longer IVQM runs, the 

more accurate the calibration will become. The interval must be set using the 

“Calculate Every” pull-down menu. 

4. Calculate periodically & combine last 9 results. The calibration parameters 

will be calculated at a set interval (e.g., every 15 minutes). The last 9 results will 

be combined to form more accurate estimations. Thus, if the transmission system

under test is changed, the calibration values will slowly converge to the new 

system’s calibration values. The interval must be set using the “Calculate Every”

pull-down menu. This mode would be preferred when watching the performance

of a transmission system that may change dynamically (e.g., from MPEG-2 to 

MPEG-4). 

5. Calculate periodically & combine last 3 results. The calibration parameters 

will be calculated at a set interval (e.g., every 15 minutes). The last three results

will be combined to form more accurate estimations. Thus, if the transmission 

system under test is changed, the calibration values will quickly converge to the 

new system’s calibration values. The interval must be set using the “Calculate 

 13
 

Every” pull-down menu. Since only three video sequences are used to estimate 

calibration quantities, calibration estimates will be less accurate than the 

“combine last 9 results” option. However, IVQM will adjust faster to system 

changes. 

6. Calculate periodically and discard previous results. The calibration

parameters will be calculated at a set interval (e.g., every 15 minutes). All 

previous calibration results will be discarded. Thus, if the transmission system 

under test is changed, the calibration values will adjust to the new system’s 

calibration values as soon as the interval has passed. The interval must be set 

using the “Calculate Every” pull-down menu. Since only one video sequence is 

used to estimate calibration quantities, calibration values may occasionally be 

erratic (e.g., the accuracy of some calibration algorithms is sensitive to scene 

content).

The selected calibration option criteria will apply from the time “start” is pressed on both 

the master and slave computers, until “stop” is pressed on either the master or slave 

computer. See Section 5.6 for more information on starting and stopping cycles. 

4.3.4 Video Quality 

Video quality model options are only available on the master computer. 

4.3.4.1 Video Quality Model (VQM) 

Four video quality models are currently available.

1. Developer’s Model. The Developer’s Model minimizes CPU usage. This model

requires about 1.2 Mbits/sec of IP bandwidth for 525-line / NTSC video 

sequences. The Developer’s model is fully documented in [1]. 

2. General Model. The NTIA General Model has been standardized by ANSI

(ANSI T1.801.03-2003 [6]) and included in two ITU Recommendations (ITU-R 

BT.1683 [7] and ITU-T J.144 [8]). The General Model takes about 6 times longer

than the Developer’s Model to run and requires about 14 Mbits/sec of IP

bandwidth for 525-line / NTSC video sequences. The NTIA General Model is 

also fully documented in [1] and summarized in [2]. 

3. Low Bandwidth Model. This model minimizes the IP network bandwidth, 

requiring approximately 10 kbits/sec bandwidth for 525-line / NTSC video 

sequences. The run-speed of this model is comparable to that of the General 

Model. An overview of the Low Bandwidth Model is given in [3]. 

4. Fast Low Bandwidth Model. This variant of the Low Bandwidth Model 

minimizes CPU usage. 

 14
 

The bandwidth of the IP network used to support end-to-end communication should be 

considered when choosing a model. The above bandwidth rates must be multiplied by 

the video sequence length (e.g., 10 seconds) to give the total bits to be transmitted for

each captured video sequence. IVQM is not a real-time system. The speed of computing 

the requested VQM (Developer’s, General, Low Bandwidth, and Fast Low Bandwidth) 

for a captured video sequence will depend upon the speed of the computer as well as the

time required to transmit the features over the IP connection. 

4.3.4.2 Video Sequence Length

This pull-down menu specifies the length of the video sequence used to estimate the 

selected VQM. The actual video capture will grab that length plus twice the combined

temporal registration uncertainty selected in section 4.3.3.1. However, the VQM

calculation will only utilize the length of video sequence specified by this pull-down 

menu. 

4.3.5 Accept Setup 

When you are satisfied with the setup options, press the “accept setup” button. IVQM

will then check the validity of the setup information, including the video capture system’s 

operation and the communication medium’s availability. These checks will take several 

seconds. If a problem is found, this problem must be fixed before proceeding. If no 

problems are found, the setup screen will disappear and the user will be presented with 

the main IVQM screen.

Warning: While checking the setup, IVQM will perform a sample video capture to

verify that the video capture device can capture video at full frame rate. If the capture 

frame rate (as measured by the video capture device) corresponds to a standard video 

frame rate, then IVQM will proceed to the monitoring screen. If IVQM detects a nonstandard frame rate (e.g., 27.794 frames per second), IVQM will report this error and 

return to the setup screen. Usually, a non-standard frame rate indicates a slow video 

capture device. However, occasionally a video capture device that meets the full video

frame rate capture benchmark will fail – perhaps due to the presence of other programs 

running on the computer. See Section 6, issue #3. 

 15
 

5 IVQM MONITORING SCREEN

Figure 4 depicts the IVQM monitoring screen on the master computer, and Figure 5 

depicts the IVQM monitoring screen on the slave computer. The IVQM monitoring

screen contains a menu bar, a time history plot of the selected VQM (Developer’s, 

General, Low Bandwidth, and Fast Low Bandwidth), three sample frames from the last 

video capture (first, middle, and last), a text box listing detailed results, two buttons

(“Start VQM” and “Stop VQM”), and a status bar at the bottom of the screen. 

Figure 4. Snapshot of IVQM monitoring screen, master computer. 

 16
 

Figure 5. Snapshot of IVQM monitoring screen, slave computer. 

5.1 Time History Plot of VQM 

The upper left-hand corner of the monitoring screen contains a time-history plot of VQM

scores. Quality estimates are reported on a scale from zero to one, where zero means that 

no impairment is visible and one means that the video clip has reached the maximum 

impairment level. The capture time of the sequences is displayed on the X-axis. Missing

line segments between VQM values (red dots) indicate that the user stopped and then restarted the video quality estimation process. Calibration values and setup parameters 

may also have changed at those points. 

5.2 Frames from Last Video Capture 

Video frames from the last capture are displayed in the upper right-hand corner of the

monitoring screen. IVQM displays the first, middle, and last frame captured. Please 

remember that the video segment captured includes twice the combined temporal 

registration uncertainty. Thus, only a portion of the captured sequence will be used to

 17
 

estimate quality. The purpose of this display is to provide the user feedback about the 

video capture device’s performance and the actual video segment used to estimate 

quality. 

5.3 Text Box Depicting Results 

The bottom third of the monitoring screen contains a scrolling text box listing results. 

When IVQM starts monitoring, the setup of both master and slave computers will be 

displayed in this box. During operation, the box will be updated to list current calibration

values and VQM estimates. 

The calibration values will be listed only when they are updated. The calibration values 

will be preceded by the time at which the video capture occurred, according to the local 

computer’s internal clock. These are the combined calibration results, calculated using

the formula selected during setup (see Section 4.3.3.3). This formula is applied to video 

captured between the last press of the “Start” button on the master computer and the 

present time. The name of each calibration value is abbreviated to save space, as follows: 

• Scale: H= #, V= # – The number following “H” indicates the horizontal spatial 

scaling factor with respect to 1000. The number following “V” indicates the 

vertical spatial scaling factor with respect to 1000. Numbers less than 1000 

indicate that the destination video is stretched; numbers greater than 1000 indicate 

that the destination video is shrunken. A spatial scaling factor of 1000 indicates 

no spatial scaling. The fractional amount of stretching or shrinking is the reported 

value divided by 1000. See [4]. The calculation of spatial scaling is by default 

disabled and can be enabled using Menu Options (see section 5.5). 

• Shift: H=#, V=# – The number following “H” indicates the horizontal spatial 

shift, in pixels. Positive numbers indicate that the destination video is shifted to

the right; and negative numbers indicate that the destination video is shifted to the

left. The number following “V” indicates the vertical spatial shift, in frame lines.

Positive numbers indicate that the destination video is shifted down; and negative 

numbers indicate that the destination is shifted up. For interlaced video signals, 

an odd number indicates re-framing. See [1]. 

• Valid: (#U,#L)(#B,#R) – These four coordinates identify the estimated valid region 

of the video signal. The coordinates (#U,#L) indicate the upper left coordinate, 

and (#B,#R) indicate the bottom-right coordinate. Numbering starts at (1,1) for the 

upper left-most pixel, and all coordinates are included in the valid region. These 

coordinates are reported with respect to the source video sequence’s coordinates 

(i.e., after correcting for spatial scaling and shifting). 

• Gain: # Offset: # – Luminance gain and offset estimates. Gain is like contrast; 

and offset is like brightness. Gain is a multiplicative factor centered on one that

will usually fall between 0.9 and 1.1. Gain values greater than one indicate that 

the destination video has a wider range of luminance values than the source video; 

and gain values less than one indicate that the destination video has a more

narrow range of luminance values. Offset is an additive factor that will usually 

fall between -15 and +15. Offset values greater than zero indicate that the 

 18
 

destination video is brighter than the source video; and offset values less than zero 

indicate that the destination video is dimmer. 

The VQM estimate will be listed for every video capture. The following three values will 

always be displayed: time, delay, and VQM. At the beginning of the line will be 

reported the time at which the video capture occurred, according to the local computer’s

internal clock. The second value reported will be the delay through the video 

transmission system. The third value reported is the VQM that was calculated using the 

model selected in Section 4.3.4.1. If the user optionally requested parameter values, then 

the individual quality parameters comprising the current VQM will also be listed. 

Special care should be taken when examining the delay through the video transmission 

system. The accuracy of this number depends entirely upon the accuracy of the 

synchronization of each computer’s clock to a time standard prior to starting IVQM. 

Small transmission delays do not necessarily justify reducing the combined temporal

registration uncertainty in Section 4.3.3.1 since this number also includes the maximum 

IP communications delay. 

Occasionally, the automated temporal registration algorithm cannot estimate delay. This 

may, for example, occur when the video sequence being examined contains no motion or 

very little motion. In this event, the transmission delay will be reported as NaN (Not A 

Number) and IVQM will default to the last valid video transmission delay estimate. 

Occasionally, IVQM will detect and report unusual events. Possible events include: 

dropped frames detected during frame capture (i.e., computer failed to capture video at

full frame rate), excessive white level clipping, excessive black level clipping, scaling 

search limit exceeded, spatial shift search limit exceeded, and luminance gain & offset 

search limit exceeded. Any potential influence of the event on VQM accuracy will be 

included in the on-screen report. 

5.4 Status Bar 

The bottom of the monitoring screen is devoted to a gray status bar. When IVQM is

monitoring video quality, the text in this gray bar will be updated with the current status

and operations that are being performed. 

5.5 Menu Options 

Due to the demanding and complex algorithms being performed, the menu options may 

be difficult to select while monitoring a system. 

On the “File” menu, the user has the option of exiting IVQM. The user also has the 

option to “Save Results As...” The latter choice brings up a window where the user may 

specify a text file name. The contents of the results text box will be saved to the 

specified file. 

 19
 

From the “File” menu of the master computer, the user can request that the video

sequences be saved to disk. Video sequences will be saved on both source and 

destination computer. Video files will be written to the directory for temporary storage 

(see 4.3.2, item 5), using automatically generated file names based upon the master 

computer’s video capture time. Select “Save Current Video to AVI” to save the video 

sequence currently being processed in uncompressed AVI file format. Select “Save 

Current Video to UYVY” to save the video sequence currently being processed in

uncompressed big-YUV file format (see [1] for a definition of the big-YUV file format). 

Select “Save Last Video to AVI” to save the previous video sequence (i.e., associated 

with the last VQM score) in AVI file format. Select “Save Last Video to UYVY” to save 

the previous video sequence in big-YUV file format. 

On the “Edit” menu, the user has the option of returning to the setup screen (“Setup...”). 

This option will only be available when IVQM is not monitoring a video system. The 

user also has the option to “Clear Results Screen.” This will erase the contents of the 

results text box (see Section 5.3). 

On the “Options” menu, the user has the option to “View Parameters.” Selecting this 

option will cause the individual parameters comprising the current model to be reported 

in the results text box, along with the VQM. See the models’ technical descriptions for a 

listing of the parameters that comprise the selected model [1]-[3]. On the “Options” 

menu, the user also has the option “Estimate Spatial Scaling.” This option is available 

only on the Master computer. When “Estimate Spatial Scaling” is checked, IVQM will 

estimate and remove spatial scaling (e.g., horizontal stretch) in the destination video. 

On the “Help” menu, the “About...” option will display the version of IVQM being 

utilized. 

5.6 Buttons 

The operation of the two buttons differs slightly between the master and slave computers.

The names on the buttons differ to remind the user of this difference. 

On the master computer, the two buttons are “Start VQM” and “Stop VQM”. These 

buttons start and stop monitoring, respectively. 

On the slave computer, the two buttons are “Automatically Calculate VQM” and “Halt.” 

The “Automatically Calculate VQM” button causes the slave to start and stop monitoring 

whenever the master computer’s buttons are pressed. The “Halt” button causes the slave 

to stop any current monitoring cycle (i.e., stop both the master and slave). Also, the slave 

computer will not re-start monitoring until the “Automatically Calculate VQM” button is

pressed. 

The program may take a while to stop the monitoring cycle after the “Stop VQM” or 

“Halt” button is pressed. The status of the buttons will indicate what is happening. For 

example, when “Stop VQM” has been pressed, the button text will be grayed out and that 

button will become non-responsive. 

 20
 

Thus, as long as the video capture and end-to-end communications options remain 

constant, monitoring can be controlled from the master computer with only limited input 

to the slave computer. 

 21
 

6 KNOWN ISSUES, BUGS, AND DEFECTS 

Following is a list of the known issues in the IVQM software. 

1. Out of Memory. Insufficient memory (e.g., less than 1 GB free system memory) 

will cause IVQM to fail in an awkward manner, without any proper failure notice. 

This will usually occur after IVQM starts monitoring a system and prior to the

first VQM estimation. Modifying your computer’s memory usage performance 

options may increase the amount of contiguous memory available to IVQM. To 

perform this modification, do the following. Right-click on “My Computer”, 

select “Properties”, and on the “System Properties” window that appears, select 

the “Advanced” tab. On that window, press the “Settings” button in the 

“Performance” section. Then in the “Performance Options” window that appears, 

select the “Advanced” tab. In the “Memory usage” section, select “System 

cache” and press “OK” and reboot your computer. 

2. Random IVQM Failure. For no reason that we have been able to determine, the

IVQM software occasionally reports a fatal error and aborts. This error is most

likely to occur after IVQM has been running for several hours. The fatal error can 

occur as a result of a segmentation violation and appears to be due to some defect 

in the hardware driver or the MATLAB Image Acquisition Toolbox software that 

is used by IVQM. If this occurs, please exit and start IVQM up again. We are 

working to resolve this issue. 

3. Video Capture Device Cannot Operate at Full Video Frame Rates. Occasionally, 

a video capture device that has worked perfectly will not be able to capture video

at the full frame rate. Thus, when examining 525-line / NTSC video at 29.97 fps, 

the video capture device might only be able to capture 19.38 fps. IVQM will 

detect this problem when checking the setup validity (see Section 4.3.5) and after 

capturing video sequences for IVQM analysis. The IVQM software performs a

check on each captured sequence to ensure that there were no dropped frames. If

dropped frames are detected, a re-capture is initiated by the IVQM software. 

Occasional dropped frames are normal behavior for many video capture devices.

However, if dropped frames are detected on every captured video sequence, there 

may be a bug or defect somewhere in the video capture device, device driver, or 

operating system. Please exit IVQM and reboot the computer to see if this 

resolves the issue. If the problem persists, replace the video capture device or

check for an updated device driver. 

4. Different Video Capture Devices on Source and Destination Computers. If at all

possible, use matched video capture devices on both computers (i.e., the same

brand and model). Otherwise, IVQM may be measuring the difference in quality 

between the two video capture devices. The impact on reported VQM values may 

be substantial (e.g., 0.25 VQM.) If you suspect this may be an issue with your 

test setup, do the following test. Route identical video to both source and 

destination computers, and monitor quality at the largest possible image

 22
 

resolution with calibration selected. This setup will analyze the differences 

between the two video capture devices. If the resulting IVQM is higher than 0.1 

on any of the time history points after calibration is completed (depending upon 

the calibration options selected, this might require several IVQM sequences to be 

analyzed), one of your video capture devices should be replaced. 

5. Video Capture Device Compatibility. IVQM was developed to be compatible 

with multiple video capture devices. However, we have had limited success 

locating compatible video capture devices that can capture at full uncompressed

video frame rates. If you identify a compatible video capture device other than 

those listed in this documentation, please let us know (vqm@its.bldrdoc.gov). 

6. System Time. Due to a defect in the underlying software used to develop the 

IVQM tool, the delay estimation ignores changes to system time after IVQM 

starts. Thus, the delay estimates will steadily become less accurate as IVQM 

operates. 

7. Master / Slave Coordination. When beginning monitoring, the master & slave 

computers perform limited cross checks on the setup configurations. Thus, for

example, the user must be careful to select identical video capture sizes on both

systems. 

8. IVQM Cannot Operate on Stored Video Files. IVQM is not designed for

processing stored video files that contain the original and processed video 

sequences. The video source YUV file option that appears on the setup screen is 

intended solely as a diagnostic and validation tool. 

9. QCIF & QSIF Not Supported. IVQM does not support QCIF and QSIF image 

sizes. Video stream resolution must be at least 320 pixels x 240 lines. The VQM 

models are not intended to be used on these smaller image resolutions. 

10. Calibration Duration Erratic. The automated calibration will sometimes run faster 

and other times run slower. This is expected behavior. 

11. Delay Experienced After Pressing Stop. When the user presses the “stop” or 

“halt” button, IVQM will often take quite a while to respond. The remote system 

may take significantly longer to halt than the local system. Please be patient. 

Multiple button presses will not speed up this process and may cause IVQM to 

malfunction. 

12. Manual Crash Recovery. If IVQM aborts due to a software or hardware bug, then 

the user should manually erase all files in the transmission communications 

directories (i.e., the shared or FTP directories on source and destination 

computers). Failure to erase the contents of these directories prior to restarting 

IVQM may result in unpredictable behavior. 

 23
 

13. “Save As” File Incompatible with Notepad. The text file produced by the “Save 

As” option on the “File” menu is not formatted properly by the Windows program 

Notepad. The text file will be read and formatted properly by the Windows 

program Wordpad. 

14. Scaling Algorithm Ineffectual with Some Content. The spatial scaling algorithm

may yield poor results for some video scenes. Still scenes in particular may cause

the spatial scaling algorithm to yield incorrect scaling values. Incorrect scaling 

factors (like other calibration errors) will cause the VQM to be erroneous and 

typically too large (i.e., the VQM estimates will indicate worse quality than what 

would result if proper spatial scaling had been performed on the destination 

video). 

 24
 

7 REFERENCES 

[1] S. Wolf and M. Pinson, “Video quality measurement techniques,” NTIA Report 02-

392, Jun. 2002. Available at www.its.bldrdoc.gov/n3/video/documents.htm. 

[2] M. Pinson and S. Wolf, “A new standardized method for objectively measuring video 

quality,” IEEE Transactions on Broadcasting, v. 50, n. 3, pp. 312-322, September 

2004. Available at www.its.bldrdoc.gov/n3/video/documents.htm. 

[3] S. Wolf and M. Pinson, “Low bandwidth reduced reference video quality monitoring 

system,” in Proc. First International Workshop on Video Processing and Quality 

Metrics for Consumer Electronics, Scottsdale, Arizona, Jan. 23-25, 2005. Available 

at www.its.bldrdoc.gov/n3/video/documents.htm. 

[4] M. Pinson and S. Wolf, “Reduced reference video calibration algorithms,” NTIA 

Report TR-06-433, Oct 2005. Available at 

www.its.bldrdoc.gov/n3/video/documents.htm. 

[5] NIST Time Program. Available at 

http://www.boulder.nist.gov/timefreq/service/pdf/nts.pdf. 

[6] ANSI T1.801.03 – 2003, “American National Standard for Telecommunications – 

Digital transport of one-way video signals – Parameters for objective performance 

assessment,” American National Standards Institute. Available at www.ansi.org. 

[7] ITU-R Recommendation BT.1683, “Objective perceptual video quality measurement 

techniques for standard definition digital broadcast television in the presence of a full 

reference,” Recommendations of the ITU, Radiocommunication Sector. 

[8] ITU-T Recommendation J.144Revised, “Objective perceptual video quality 

measurement techniques for digital cable television in the presence of a full

reference,” Recommendations of the ITU, Telecommunication Standardization 

Sector. 

[9] ITU-R Recommendation BT.601, “Encoding parameters of digital television for 

studios,” Recommendations of the ITU, Radiocommunication Sector. 

 25