1



### ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMIN IN G

POWER CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROLENABLED TELEVISIONS:

ENERGY EFFICIENCY PROGRAM FOR CONSUMER PRODUCTS:

Television Sets

Revised February 2013

U.S. Department of Energy

Assistant Secretary

Office of Energy Efficiency and Renewable Energy

Building Technologies Program

Appliances and Commercial Equipment Standards

Washington, DC 205852 This Document was prepared for the Department of Energy

by staff members of

Navigant Consulting Inc.3

ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMIN IN G POWER

CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROL-ENABLED

TELEVISIONS

TABLE OF CONTENTS

1.1 INTRODUCTION ...............................................................................................................5

1.2 TEST PROCEDURE ...........................................................................................................6

1.2.1 ADJACENT TEST SET- UP ................................................................................................6

1.2.2 DISTANCE TEST SET- UP .................................................................................................7

1.3 TEST LABS AND PRODUCTS TESTED .........................................................................8

1.4 STATISTICAL ANALYSIS METHODS ...........................................................................8

1.4.1 REGRESSION ANALYSIS ................................................................................................9

1.4.2 COEFFICIENT OF VARIATION ANALYSIS ..................................................................9

1.5 RESULTS ..........................................................................................................................10

1.5.1 SATURATION POINT .....................................................................................................10

1.5.2 LAB EFFECTS ..................................................................................................................14

1.5.3 TESTER EFFECTS ...........................................................................................................15

1.5.4 COEFFICIENT OF VARIATION .....................................................................................16

APP ENDIX A: AVERAGE POWER CONSUMPTION BY TEST SET-UP, TV, AND

IL LUMINANCE VALUE .................................................................................................17

APPENDIX B: BOX AND WHISKER PLOTS ...........................................................................18

APPENDIX C: COEFFICIENT OF VARIATION BY TEST SET- UP .......................................23

APPENDIX D: REGRESSION ANALYSIS ................................................................................24

APPENDIX E: OUTLIER ANALYSIS ........................................................................................27

APP ENDIX F: COEFFICIENTS AND HYPOTHESIS TESTING OF STATISTICAL

SIGNIFICANT OF LAB AND TESTER EFFECTS IN REGRESSIONS .......................284 LIST OF TABLES

Table 1: Characteristics of TVs used in DOE Round Robin Study .................................................8

Table 2: Predicted Range of Lab Effects under Each Model .........................................................15

Ta ble 3: Average Predicted Range of Tester Effects under Each Model ......................................15

Table 4: Average Power Consumption by Test Set-up and Illuminance Value ............................17

Ta ble 5: Coefficient of Variation for Adjacent Test Set-up ................................ ..........................23

Ta ble 6: Coefficient of Variation for Distance Test Set-up ................................ ...........................23

Ta ble 7: Coefficient of Lab Effects by Test Set-up under WLS....................................................28

Table 8: Coefficient of Lab Effects by Test Set-up under OLS with Outliers Removed ..............29

Table 9: Coefficient of Lab Effects by Test Set-up under OLS with Outliers ..............................29

Ta ble 10: Joint Statistical Significance of Lab Effects by Test Set-up (WLS Regression) ...........29

Table 11: Coefficient of Tester Effects by Test Set-up and Lab under WLS ................................29

Ta ble 12: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers

Removed ................................................................................................................30

Table 13: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers ...........30

Table 14: Joint Statistical Significant of Tester Effects by Test Set-up and Lab (WLS

Regression) ............................................................................................................30

LIST OF FIGURES

Figure 1: Adjacent Test Set-up ................................ ........................................................................7

Figure 2: Distance Test Set-up ................................ .........................................................................7

Figure 3: Average Saturation Point by Test Set-up and TV ..........................................................10

F igure 4: Average Power Consumption for TV 1 by Illuminance.................................................11

Figure 5: Average Power Consumption for TV 2 by Illuminance.................................................11

Figure 6: Average Power Consumption for TV 3 by Illuminance.................................................12

Figure 7: Average Power Consumption for TV 4 by Illuminance.................................................12

Figure 8: Average Power Consumption for TV 5 by Illuminance.................................................13

Figure 9: Average Power Consumption for TV 9 by Illuminance.................................................13

Figure 10: Average Power Consumption for TV 7 by Illuminance...............................................14

Figure 11: Average Power Consumption for TV 8 by Illuminance...............................................14

Figure 12: Average CV by Illuminance Value and Test Set-up ....................................................16

Figure 13: Average CV by TV and Test Set-up ............................................................................16

Figure 14: Observations for TV 1 under the Distance Test Set-up at 12 Lux ...............................275

ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMIN IN G POWER

CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROL-ENABLED

TELEVISIONS

1.1 INTRODUCTION

The U.S. Department of Energy (DOE) initiated a television round robin test study to

assess the precision of distance and adjacent test set -ups for measuring television (TV) power

c onsumption. Both test set-ups measure power consumption of TVs with Automatic Brightness

Control (ABC) a technology at various illuminance levels. The distance test set-up was derived

from the ENERGY STAR V. 6.0 Displays test method investigation and uses a specified lamp to

il luminate the TV sensor from a distance of 5 feet. The adjacent test set-up places the same lamp

in a closed apparatus directly in front of the TV’s sensor to illuminate the TV sensor.

DOE evaluated the round robin data in two ways:

 Performed a regression analysis b using three regression models to predict lab and tester

effects c for each test set-up, considering all observations recorded during round robin

testing. This analysis provides a robust estimate of lab and tester effects across all labs

and all TVs, while using the largest available sample size. The report considers both the

size of the systematic effect (in watts) caused by a given variable (e.g. a lab or tester) on

predicted power consumption as well as the statistical significance d of this effect size.

Lab and tester effects were determined for the distance and adjacent test set-ups by

separately performing regression analysis on the observations from each test set-up.

a ABC is a technology which allows TVs to adjust screen brightness based on a room’s ambient light. It both reduces power consumption and improves picture quality in low-light situations.

b Regression analysis refers to the overall analysis performed using different regression models (WLS, OLS with heteroskedasticity consistent standard errors, etc.)

c Lab effects are defined as the predicted effect a particular lab testing a TV will have on that TV’s measured power consumption when compared to other labs; tester effects are defined as the predicted effect a particular tester testing a TV will have on that TV’s measured power consumption when compared to other testers in the same lab. If lab effects were zero then all labs would on average measure the same power consumption, and if tester effects were zero then all testers in a lab would on average measure the same power consumption. One of the main purposes of the regression analysis was to determine whether there is enough statistical evidence to say whether different labs or different testers within a lab have differences in measured power

c onsumption which cannot be attributed to random chance.

d The phrase ‘statistically significant’ is used to indicate that 95% of the time, the absolute difference between two test results will not exceed a calculated limit of statistical significance,

e xcept when another percentage is mentioned. This is a standard method of assessing whether a given pattern is unlikely to have occurred by chance.6

 Performed a precision analysis by calculating the Coefficients of Variation (CVs) e for

each set (defined throughout the paper as the set of observations under a given test set-up

on a given TV at a given lux point f ).

Lab and tester effects are considered in the regression analysis, and the precision analysis

focuses on lab and tester variability for a given TV, test set-up and illuminance value. g

1.2 TEST PROCEDURE

Four labs conducted round robin testing using the two test set-ups described in this

section. Each test set-up measured the power consumption (in watts) of different TV models as a

func tion of the ambient illuminance of a room (in lux). For each test set-up, each lab tested each

TV three times. h Tests used the primary HDMI input to play an International Electrotechnical

Commission (IEC) test clip with no surfaces other than the table the TV was on within two feet

of the sensor. For both test set-ups, testers used a standard spectrum halogen incandescent

reflector lamp with a conditioned power supply as the source of illuminance. To minimize

autocorrelation, i testers powered down each TV and all testing equipment; TVs were physically

re moved from the test station, and then set up again for the next testing round. Testers also

stabilized each TV for one hour prior to testing. Labs were instructed to have a different tester

perform each individual test, if possible. Requiring multiple testers also helps minimize

a utocorrelation and allows for the analysis of tester effects within labs where ever possible.

1.2.1 ADJACENT TEST SET- UP

Prior to testing using the adjacent test set-up, each lab constructed an apparatus to

il luminate the sensor in a closed system. The apparatus had a diameter at one end which fit

completely around the light source and a diameter at the other end of two inches. The tester

secured the light source so no light escaped between the light source and the apparatus. The

tester placed the two inch diameter end of the apparatus flush against the TV bezel, completely

covering the ABC sensor. The tester aligned the center of the lamp at the same height as the

center of the ABC sensor with respect to the floor. Figure 1 illustrates the adjacent test set-up.

e CV is the ratio of the standard deviation to the mean.

f Lux is the SI unit for illuminance.

g Set refers to data obtained by different testers and different labs for a given TV, tested with a given test set-up (distance or adjacent), and tested at a given illuminance value.

h TV 6 was not tested by Lab D.

i The correlation which commonly occurs between values measured in sequence.7 (Not to scale)

Figure 1: Adjacent Test Set- up

1.2.2 DISTANCE TEST SET- UP

For the distance test set-up, testers placed a light source at a distance of five feet from the

surface of the ABC sensor. To ensure the light entered the sensor at a zero degree vertical angle,

the distance from the floor to the center of the lamp was set to be equal to the distance from the

floor to the center of the sensor. Each TV was placed with all four corners equidistant from a

reference plane (e.g., wall). Testers aligned the center of the lamp to be perpendicular with

respect to the sensor. Figure 2 shows the set-up for the distance test set-up.

(No t to scale)

Figure 2: Distance Test Set- up8

1.3 TEST LABS AND PRODUCTS TESTED

Four laboratories, Lab A, Lab B, Lab C, and Lab D, participated in the television round

robin test study and DOE selected eight different TV models for testing. Each participating lab

tested both the adjacent and distance test set-ups three times on each of the eight TVs, except for

Lab D which did not perform the adjacent test set-up on TV 6 due to technical issues with that

TV. All four labs took measurements at 0, 10, 12, 35, 50, 100, and 300 lux on each TV, using

both test set-ups; Lab A and Lab C also took measurements at 75 lux. Table 1 lists the

characteristics of the eight televisions tested. All TVs meet the ENERGY STAR specification

Version 5.3. 1

Table 1 : Charac teristics of TVs used in DOE Round Robin Study

Test Unit Screen Type Screen Size (inches) Manufacture Date

TV 1 LED Backlit 55 March 2010

TV 2 LCD 46 September 2010

TV 3 LED Backlit 42 May 2010

TV 4 LED 46 February 2012

TV 5 LED 47 March 2012

TV 6 LED 32 August 2010 j

TV 7 LED 46 March 2011 k

TV 8 CCFL 53 May 2009 l

1.4 STATISTICAL ANALYSIS METHODS

To analyze the round robin test results, two primary methods were used:

 A regression analysis estimated the size and joint statistical significance m of lab and tester

effects for both the entire sample of round robin data and a variety of subsamples. Three

regression models were used:

o Weighted Least Squares regression 2 (WLS regression)

o Ordinary Least Squares regression with heteroskedasticity-consistent standard

errors (OLS regression) n

j Estimated based on release date of TV.

k Id.

l Id.

m Joint Statistical Significance is a measure of whether the labs in the round robin are statistically different from each other, or whether the testers in a lab are statistically different from each other. This is different from individual statistical significance because individual statistical significance relies on a reference value to compare against. This is one of the primary ways of analyzing lab and tester effects.

n Heteroskedasticity occurs when the errors in a regression model are correlated with the explanatory variables of that model. Heteroskedasticity-consistent standard errors are a method of correcting for this in hypothesis testing. Note that most hypothesis testing was performed on

the WLS model.9 o Ordinary Least Squares regression with heteroskedasticity-consistent standard

errors and outliers removed (OLS regression with outliers removed).

 A coefficient of variation was calculated for each set to evaluate in-set precision.

1.4.1 REGRESSION ANALYSIS

The regression analysis o examines the round robin data to predict power consumption of

a given TV as a function of the TV, illuminance value, test set-up, lab, and tester. Each

regression model predicts the average effect size in watts for each of these variables on power

consumption and the standard error p of that prediction. The regression analysis helped DOE

understand differences between the distance and adjacent test set-ups, such as whether the

observations at different labs or the observations of different testers at the same lab are

statistically different from each other. For more information on the methodology of the

regression analysis see Appendix D: Regression Analysis.

The different regression models were used to account for the different properties of the models.

 The WLS regression gives weights to each observation which are inversely proportional

to the variance of that observation, which means that most of the weight is given to the

most typical observation. This helps the model avoid trying to incorporate outliers and is

more appropriate for hypothesis testing but may underestimate the unidirectional effects

of values with higher variance.

 The OLS regression gives equal weights to all observations, which provides a broader

sample of the observation range but is more susceptible to extreme values and is less

appropriate for hypothesis testing.

The OLS regression with outliers removed was used to better account for the

observations with moderate variance from the predicted value without being affected by the most

extreme values. See Appendix E: Outlier Analysis for details on the methodology by which

outliers were removed.

1.4.2 COEFFICIENT OF VARIATION ANALYSIS

The coefficient of variation is calculated as the ratio of the sample standard deviation of a

given set to that set’s mean. In other words, a set’s mean and sample standard deviation were

calculated, and then the standard deviation was divided by the mean. Outliers were not removed

in this analysis due to sample size concerns and because outliers affected the analysis less than in

the OLS regression.

o For more detail on the nature of these regressions, see the appropriate regression subsections in

1.5 Results and in Appendix D: Regression Analysis.

p Standard Error is a measure of the error of the sample mean (the average of the observations for a given test set-up, TV, and lux value) as an estimator of the population mean (the actual average power consumption of a TV). It is calculated by dividing the sample standard deviation of a set by the square root of the sample size of that set. Note this is not to be confused with standard

deviation.10

1.5 RESULTS

1.5.1 SATURATION POINT

Sensor saturation occurs when an increase in illuminance does not lead to an increase in

display brightness (or alternatively power consumption). It is defined in this report as th e illuminance value at which a TV reaches average power consumption greater than 95% of its

power consumption at 300 lux. Figure 3 shows the average saturation value for each TV by test

set-up and Figures 4 through 11 shows the average measured power consumption for each TV as

a function of illuminance value. For more information see Appendix A: Average Power

Consumption by Test Set-up, TV and Illuminance Values

Figure 3: Average Saturation Point by Test Set-up and TV 0 50 100 150 200 250 300 TV 8 TV 7 TV 6 TV 5 TV 4 TV 3 TV 2 TV 1

Saturation Value (Lux) 

## Average Saturation Value vs. TV

Adjacent Distance11 (Error bars reflect the standard deviation of each set)

Figure 4: Average Power Consumption for TV 1 by Illuminance

(Error bars reflect the standard deviation of each set)

Figure 5: Average Power Consumption for TV 2 by Illuminance 0 50 100 150 200

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 1

Adjacent Distance

0 20 40 60 80 100 120 140

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 2

Adjacent Distance12 (Error bars reflect the standard deviation of each set)

Figure 6: Average Power Consumption for TV 3 by Illuminance

(E rror bars reflect the standard deviation of each set)

Figure 7: Average Power Consumption for TV 4 by Illuminance 0 20 40 60 80 100 120

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 3

Adjacent Distance

0 20 40 60 80 100

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 4

Adjacent Distance13 (Error bars reflect the standard deviation of each set)

Figure 8: Average Power Consumption for TV 5 by Illuminance

(E rror bars reflect the standard deviation of each set)

Figure 9: Average Power Consumption for TV 9 by Illuminance 0 20 40 60 80 100

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 5

Adjacent Distance

0 10 20 30 40 50

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 6

Adjacent Distance14 (Error bars reflect the standard deviation of each set)

Figure 10: Average Power Consumption for TV 7 by Illuminance

(Error bars reflect the standard deviation of each set)

Figure 11: Average Power Consumption for TV 8 by Illuminance

1.5.2 LAB EFFECTS

The WLS regression analysis (the only model allowing unbiased hypothesis testing)

indicated that neither test set-up had statistically significant lab effects, meaning that the lab

effects for both test set-ups could reasonably be attributed to random chance. The range of

predicted lab effects (the predicted systematic change in measured power consumption between

0 20 40 60 80 100 120

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 7

Adjacent Distance

0 50 100 150 200

0 10 12 35 50 75 100 300Power Consumption (W)

Illuminance (Lux)



## TV 8

Adjacent Distance15

the two labs whose measurements are furthest apart) is presented in Table 2. For more detailed

results see Appendix F: Coefficients and Hypothesis Testing of Statistical Significant of Lab and

Tester Effects in Regressions.

Table 2 : P redicted Range of Lab Effects under Each Model

Test Set

- up Adjacent Test Set

- up (W) Distance Test Set

- up (W)

WLS 0.6 1.8

OLS 0.7 2.4

OLS (Outliers Removed) 1.6 2.3

Table 2 indicates that the predicted difference between the lowest power consumption

measured at a given lab and the highest power consumption measured at a given lab is 0.6 W and

1.8 W for the adjacent and distance test set-ups, respectively, using the WLS model.

1.5.3 TESTER EFFECTS

The WLS regression indicated that neither test set-up had any labs with statistically

significant tester effects, meaning that the tester effects for both test set-ups could reasonably be

attributed to random chance. The average range of predicted tester effects across all four labs is

shown in Table 3. Much of the reason for the higher predicted tester effects for the adjacent test

se t-up is due to Lab B. For more detailed information see Appendix F: Coefficients and

Hypothesis Testing of Statistical Significant of Lab and Tester Effects in Regressions

Table 3 : Ave rage Predicted Range of Tester Effects under Each Model

Test Set

- up Adjacent Test Set

- up (W) Distance Test Set

- up (W)

WLS 1.38 0.97

OLS 1.45 0.94

OLS (Outliers Removed) 1.07 0.9416

1.5.4 COEFFICIENT OF VARIATION

Average CV for the distance test set-up is 2.2% compared to an average CV of 3.5% for

the adjacent test set-up. Figure 12 shows the average CV as a function of illuminance for the

dist ance and adjacent test set-ups. Figure 13 compares the CV by TV and test set-up averaged

across all lux values. CVs for each set are included in Appendix F: Coefficients and Hypothesis

Testing of Statistical Significance of Lab and Tester Effects in Regressions.

Figure 12: Average CV by Illuminance Value and Test Set- up

Figure 13: Average CV by TV and Test Set- up 0% 1% 2% 3% 4% 5% 6% 7%

0 10 12 35 50 75 100 300Average CV (%)

Illuminance (Lux)

Distance Adjacent

0% 1% 2% 3% 4% 5% 6%

TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8CV (%)

Distance Adjacent17

APPENDIX A: AVERAGE POWER CONSUMPTION BY TEST SET-UP, TV,

AND ILLUMINANCE VALUE

This appendix gives the average power consumption for each set in watts. It averages

each of the three observations from each of the labs which measured a TV, so there are twelve

observations for most of the sets. There are six observations for each set at 75 lux because Labs

B and D did not measure power consumption at 75 lux, and there are nine observations for each

set measuring the power consumption of TV 6 (excluding those at 75 lux) because Lab D did not

measure power consumption on TV 6.

Table 4 : Ave rage Power Consumption by Test Set-up and Illuminance Value

Test Set

up TV 0 lux 10 lux 12 lux 35 lux 50 lux 75 lux 100 lux 300 lux

Adjacent 1 71.5 156.9 150.8 157.9 157.8 157.6 157.6 157.5

2 64.9 67.2 67.1 96.8 112.1 122.6 126.7 126.3

3 46.1 92.6 97.7 101.2 99.0 100.6 101.0 101.0

4 48.2 78.3 78.3 78.3 78.3 77.6 78.2 78.2

5 31.8 75.3 76.3 76.5 76.4 76.0 76.3 76.3

6 28.6 30.8 32.0 39.4 40.4 42.6 42.7 43.0

7 44.2 53.3 54.9 73.4 84.6 85.8 88.7 91.1

8 115.6 125.6 129.6 157.7 156.0 166.5 164.0 167.0

Distance 1 71.6 120.2 132.1 157.2 157.1 155.8 156.8 156.8

2 64.8 71.0 74.9 114.3 126.5 126.4 126.2 126.0

3 46.1 64.8 68.0 92.8 100.1 100.6 101.0 100.9

4 48.1 69.2 70.0 76.8 78.0 77.7 78.2 78.2

5 31.9 45.2 48.8 68.0 73.2 76.3 76.5 76.5

6 28.6 28.9 30.0 40.5 42.3 43.4 45.1 46.5

7 44.0 44.1 44.3 46.1 47.6 52.2 53.9 75.9

8 115.7 115.8 115.9 117.9 123.3 135.3 144.7 167.618 APPENDIX B: BOX AND WHISKER PLOTS

An analysis was done using the range and average of a given set to analyze precision.

The box and whisker plots from this analysis are presented in the tables in this section. Box and

whisker plots are helpful tools to present the dispersion of data in a given set. In each set, the

minimum and maximum values are represented by the ends of the whiskers (the thin horizontal

li nes on the outside of each box). The 25 th percentile and 75 th percentile are represented by the

bottom and top of the box, respectively, and the median (middle value) is represented by the

middle horizontal line in the box.

60 80 100 120 140 160 180Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 119 60 70 80 90 100 110 120 130 140Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 2

40 50 60 70 80 90 100 110Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 320 40 45 50 55 60 65 70 75 80 85Power Consumption (Lux)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 4

30 35 40 45 50 55 60 65 70 75 80Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 521 25 30 35 40 45 50Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 6

40 50 60 70 80 90 100 110Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 722 110 120 130 140 150 160 170 180Power Consumption (Watts)

Test Set

- up and Illuminance



## Box and Whisker Plot

- TV 823 APPENDIX C: COEFFICIENT OF VARIATION BY TEST SET- UP

Table 5 : Coe fficient of Variation for Adjacent Test Set- up

Percent

Lux TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8 Grand

Mean

0 0.4 0.3 0.9 1.0 0.7 1.0 1.3 0.7 0.8

10 3.2 4.1 10.7 1.3 3.4 4.5 5.1 5.0 4.7

12 13.5 4.5 4.7 1.3 1.0 6.8 5.9 8.7 5.8

35 3.0 9.0 0.7 1.3 0.5 7.0 8.9 9.2 5.0

50 2.9 8.5 7.0 1.3 0.5 4.7 3.8 12.4 5.1

75 2.6 3.7 0.3 0.4 0.3 0.7 0.1 0.8 1. 2

100 2.8 0.4 0.7 1.3 0.5 2.8 8.2 6.2 2.8

300 2.8 0.5 0.7 1.3 0.5 1.5 10.2 1.3 2.4

Grand

Mean 3.9 3.9 3.2 1.1 0.9 3.6 5.4 5.6 3.5

Table 6 : Coe fficient of Variation for Distance Test Set- up

Percent

Lux TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8 Grand

Mean

0 0.5 0.3 0.7 0.7 0.9 1.0 0.4 0.9 0.7

10 4.6 2.6 3.1 2.8 3.4 2.0 0.6 0.9 2.5

12 5.9 3.5 5.0 2.1 2.1 2.4 1.7 0.9 2.9

35 3.1 2.2 5.3 1.8 2.2 1.2 3.7 2.8 2.8

50 3.1 0.8 3.3 1.5 2.4 1.0 4.7 2.5 2.4

75 2.5 0.2 0.3 0.4 0.5 0.7 6.8 4.9 2.0

100 3.1 0.8 0.6 1.2 0.6 1.1 6.9 3.2 2.2

300 3.1 0.8 0.6 1.1 0.6 0.8 11.2 1.2 2.4

Grand

Mean 3.3 1.4 2.4 1.4 1.6 1.3 4.5 2.2 2.224 APPENDIX D: REGRESSION ANALYSIS

The regression analysis combined the sets q to increase the number of observations and to

evaluate information across sets while minimizing bias and assumptions. White’s test 3 was performed on the data and indicated a high level of heteroskedasticity, so Weighted Least

Squares (WLS) and Ordinary Least Squares (OLS) with heteroskedasticity robust standard errors

were used to make estimates more statistically efficient and to reduce potential bias in the

standard errors (and thus hypothesis tests).

WLS is more robust to heteroskedasticity than OLS, so it was the primary model used for

hypothesis testing. The OLS model with heteroskedasticity-consistent standard errors was

primarily used for estimating effect size, as the heteroskedasticity does not bias effect size. The

main difference between the three models in how they estimate effect size is the allocation of

weights to each observation. The OLS model with outliers included gives equal weight to all

observations; the OLS model with outliers removed gives equal weight to all observations but

excludes outliers; and, the WLS model gives weight equal to the reciprocal of the variance of an

observation to each observation. In other words, the WLS model gives less weight to higher

variance observations than to lower variance observations. Most hypothesis testing used the

WLS model, while all three models were used for estimating effect size.

The full equation for all three models is provided in Equation 1 . This equation was

applied to the adjacent and distance test set-up samples separately. Note that the results from the

OLS and WLS regressions were usually very similar, although the predicted lab effects were

somewhat lower for the distance test set-up than the adjacent test set-up under the WLS

regression, both for effect size and statistical significance.

Equation 1

Va riable Definitions:

 Power – measured power (in Watts) of an observation

 α

– coefficient of the predicted equation

 β, δ, π, ρ – coefficients of the explanatory variables

 ε

– error term or residual needed to balance the equation. Can be calculated by

subtracting the predicted value for an observation based on the rest of the equation from

the actual value

 percsat – illuminance as a percentage of saturation illuminance for a given TV under the

distance method

q “Set” refers to data obtained by different testers and different labs for a given TV, tested with a given test set-up (distance or adjacent), tested at a given illuminance value.25

 δ variables – binary variables for whether a test was performed on a given TV

 π variables – binary variables for whether a test was performed by a given lab

 ρ variables – binary variables for whether a test was performed by a given tester

After carefully reviewing a variety of scatterplots for subsamples, this equation was

chosen to adequately account for different effects while minimizing over-fitting the data. The

variables omitted in the equation to prevent collinearity are:

 TV 6

 Lab C

 Tester A4

 Tester B2

 Tester C3

 Tester D2

These variables are included in the analysis but are omitted from the equation to provide

a reference case. For example, when investigating lab effects all observations are observed at

Lab A, Lab B, Lab C, or Lab D. The equation assigns a 1 to a lab variable if an observation

occurred at that lab and a 0 if it did not. However, if all four labs had binary variables associated

with them, then the equation would not be solvable. For example, suppose there were only two

labs, Lab E and Lab F, and Lab E measured 1 W higher on average than Lab F. If only Lab E

was included in the equation it would have a coefficient of 1 which indicates that it measures one

watt higher than the lab which is not included – Lab F. If only Lab F was included, then it would

ha ve a coefficient of -1, which indicates that it measures 1 W lower than the lab that is not

included – Lab E. If both were included, however, their coefficients would reference each other

a nd the equation could not be solved.

When evaluating lab effects, the tester effect coefficients (ρ coefficients) are held at zero,

and when examining tester effects the model is restricted to a particular lab. This allows lab and

teste r effects to be examined separately.

The models make the following assumptions common to all regression analysis 4 :

 The equation is correctly specified. In other words, the explanatory variables do not pick

up the effects of omitted variables.

 The expected value of the error term is zero, and the expected value of the error term

subject to the explanatory variables is zero. This assumption is necessary to achieve a

best fit given the model and equation, and the models do this automatically.

 Variables are not linear combinations of each other. This is why the variables mentioned

as omitted were omitted.

 Errors are normally distributed around 0. This assumption is necessary for accurate

hypothesis testing and the determination of standard errors.

It should also be noted that the models do not include interaction effects. This means, for

example, that when comparing two TVs at zero lux under the distance test set-up the difference

between them will be predicted to be the average difference between them predicted in the

model, but not the average difference predicted between them at zero lux under the distance test26

set-up. There still are coefficients to account for the fact that the TVs are measured at zero lux

and under the distance test set-up but they are independent of which TV is used. This assumption

is more reasonable for some TVs than others but was decided on for a variety of reasons

including :

 Using interaction terms would cause a much larger number of subpopulations for

comparison, removing many of the benefits of combining the data.

 The coefficients would have been drawn from a much smaller effective number of

observations, making each coefficient much more susceptible to outliers.

 The benefits and drawbacks of using the no interactions models would apply equally to

the distance and adjacent test set-ups and each lab and tester, and so were not expected to

change the relevant conclusions.

 The interaction terms experimented with exhibited an especially high degree of

heteroskedasticity and increased heteroskedasticity throughout the models.27 APPENDIX E: OUTLIER ANALYSIS

Outliers were identified and removed from the OLS regression based on the criteria that

they either deviated from the median observation of their set by 20 W or by 20% of the median.

This corresponds to being outside the 99 th percentile for either watts or percentage. These criteria

were chosen by balancing the need to remove gross outliers for a good regression equation

against the risks of no longer having a representative data sample. Figure 14 shows the

obser vations in one set for TV 1 and indicates the level of deviation necessary to be considered

an outlier.

Eighteen outliers were removed from the 1398 total observations in the round robin. Of

the eighteen outliers under the criteria in this section, seventeen corresponded with the adjacent

test set-up. Also of the eighteen, sixteen occurred at Lab B. This suggests that Lab B had a

greater propensity to measure outliers, particularly under the adjacent test set-up.

(T he outlier was removed based on the criterion of a 20 W deviation from the median)

Figure 14: Observations for TV 1 under the Distance Test Set-up at 12 Lux 120 125 130 135 140 145 150Power Consumption (W)

Normal Observation Median of Observations Outlier28

APPENDIX F: COEFFICIENTS AND HYPOTHESIS TESTING OF STATISTICAL

SIGNIFICANT OF LAB AND TESTER EFFECTS IN REGRESSIONS

This appendix presents the coefficients and statistical significance of lab and tester

effects. As noted in Ap pendix D: Regression Analysis, the WLS is the only regression used for

hypothesis testing of joint statistical significance due to its robustness to heteroskedasticity.

In this appendix, each coefficient uses the appropriate omitted variable as outlined in

Appendix D: Regression Analysis as a reference value. In choosing the reference tester, the tester

who pe rformed the most tests was used as the reference case. At Lab C, where each tester

performed the same number of tests, the reference tester was chosen arbitrarily. The reference

lab was chosen arbitrarily to be Lab C in all cases.

In this appendix the coefficient for a particular lab or tester is the change in expected

measured power consumption for that lab in relation to the reference lab or tester. To compare

two labs or testers when neither is the reference case, the coefficient of one is subtracted from the

other. For example, under the WLS regression for the distance test set-up in Table 6, Lab B has a

coefficient of 0.72 while Lab A has a coefficient of -1.03. This means that Lab B measured 0.72

W higher than Lab C on average, and Lab A measured 1.03 W lower than Lab C on average.

Therefore Lab B measured 0.72 – (-1.03) = 1.75 W higher than Lab A. Note that the choice of

reference lab and tester has no effect on the calculation of these results.

The joint statistical significance of the lab and tester effects are presented as P values,

which can be interpreted as the probability of obtaining a test statistic at least as extreme as the

one observed assuming that there are no lab or tester effects. As the 95% confidence interval was

used, P values lower than 0.05 are statistically significant. Joint statistical significance is used

because it is not affected by choice of reference lab or tester.

Table 7 : Coe fficient of Lab Effects by Test Set-up under WLS

Test Set

- up Adjacent (W) Distance (W)

Lab A

- 0.21

- 1.03

Lab B

- 0.13 0.72

Lab C Reference Reference

Lab D

- 0.65

- 1.0829

Table 8 : Coe fficient of Lab Effects by Test Set-up under OLS with Outliers Removed

Test Set

- up Adjacent (W) Distance (W)

Lab A

- 0.44

- 1.19

Lab B 0.63 0.94

Lab C Reference Reference

Lab D

- 1.01

- 1.32

Table 9 : Coe fficient of Lab Effects by Test Set-up under OLS with Outliers

Test Set

- up Adjacent (W) Distance (W)

Lab A

- 0.22

- 1.19

Lab B

- 0.18 1.13

Lab C Reference Reference

Lab D

- 0.69

- 1.27

Table 10 : Join t Statistical Significance of Lab Effects by Test Set-up (WLS Regression)

Test Set

- up Adjacent (P Value) Distance ( P Value )

Lab Effects 0.96 0.11

Table 11 : Coe fficient of Tester Effects by Test Set-up and Lab under WLS

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.14 0.36

Tester A2

- 0.23

- 0.88

Tester A3

- 0.28 0.29

Tester A4 Reference Reference

Lab B Tester B1

- 4.25 0.81

Tester B2 Reference Reference

Lab C Tester C1

- 0.73 0.13

Tester C2

- 0.48 1.38

Tester C3 Reference Reference

Lab D Tester D1 0.11

- 0.44

Tester D2 Reference Reference30

Table 12 : Coe fficient of Tester Effects by Test Set-up and Lab under OLS with Outliers

Removed

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.13 0.29

Tester A2

- 0.24

- 0.71

Tester A3

- 0.30 0.21

Tester A4 Reference Reference

Lab B Tester B1

- 2.95 0.68

Tester B2 Reference Reference

Lab C Tester C1

- 0.12 0.55

Tester C2

- 0.74 1.57

Tester C3 Reference Reference

Lab D Tester D1 0.14

- 0.49

Tester D2 Reference Reference

Table 13 : Coe fficient of Tester Effects by Test Set-up and Lab under OLS with Outliers

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.13 0.29

Tester A2

- 0.24

- 0.71

Tester A3

- 0.30 0.21

Tester A4 Reference Reference

Lab B Tester B1

- 4.14 0.68

Tester B2 Reference Reference

Lab C Tester C1

- 1.10 0.54

Tester C2

- 0.46 1.57

Tester C3 Reference Reference

Lab D Tester D1 0.14

- 0.49

Tester D2 Reference Reference

Table 14 : Join t Statistical Significant of Tester Effects by Test Set-up and Lab (WLS

Regression)

Test Set

- up Adjacent (P Value) Distance (P Value)

Lab A 1.00 0.97

Lab B 0.17 0.77

Lab C 0.93 0.56

Lab D 0.96 0.7331

1 http://www.energystar.gov/ia/partners/product_specs/program_reqs/Televisions_Program_Requ

irements_V5_3.pdf?0082-7f02

2 Stata Analysis Tools: Weighted Least Squares Regression. UCLA : Academic Technology

Services, Statistical Consulting Group. From

http://www.ats.ucla.edu/stat/stata/ado/analysis/wls0.htm (accessed September 2012).

3 White, H. (1980). "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity". Econometrica 48 (4): 817–838

4 Consulting Stock, J. H., & Watson, M. W. (2007). Introduction to econometrics . (2 ed.). New

York: Addison-Wesley.