1

ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMINING 

POWER CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROLENABLED TELEVISIONS: 

ENERGY EFFICIENCY PROGRAM FOR CONSUMER PRODUCTS: 

Television Sets 

Revised February 2013 

U.S. Department of Energy 

Assistant Secretary 

Office of Energy Efficiency and Renewable Energy 

Building Technologies Program 

Appliances and Commercial Equipment Standards 

Washington, DC 20585 
2

This Document was prepared for the Department of Energy 

by staff members of 

Navigant Consulting Inc. 
3

ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMINING POWER 

CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROL-ENABLED 

TELEVISIONS 

TABLE OF CONTENTS 

1.1 INTRODUCTION ...............................................................................................................5

1.2 TEST PROCEDURE ...........................................................................................................6

1.2.1 ADJACENT TEST SET-UP ................................................................................................6

1.2.2 DISTANCE TEST SET-UP .................................................................................................7

1.3 TEST LABS AND PRODUCTS TESTED .........................................................................8

1.4 STATISTICAL ANALYSIS METHODS ...........................................................................8

1.4.1 REGRESSION ANALYSIS ................................................................................................9

1.4.2 COEFFICIENT OF VARIATION ANALYSIS ..................................................................9

1.5 RESULTS ..........................................................................................................................10

1.5.1 SATURATION POINT .....................................................................................................10

1.5.2 LAB EFFECTS ..................................................................................................................14

1.5.3 TESTER EFFECTS ...........................................................................................................15

1.5.4 COEFFICIENT OF VARIATION .....................................................................................16

APPENDIX A: AVERAGE POWER CONSUMPTION BY TEST SET-UP, TV, AND 

ILLUMINANCE VALUE .................................................................................................17

APPENDIX B: BOX AND WHISKER PLOTS ...........................................................................18

APPENDIX C: COEFFICIENT OF VARIATION BY TEST SET-UP .......................................23

APPENDIX D: REGRESSION ANALYSIS ................................................................................24

APPENDIX E: OUTLIER ANALYSIS ........................................................................................27

APPENDIX F: COEFFICIENTS AND HYPOTHESIS TESTING OF STATISTICAL 

SIGNIFICANT OF LAB AND TESTER EFFECTS IN REGRESSIONS .......................28
4

LIST OF TABLES 

Table 1: Characteristics of TVs used in DOE Round Robin Study .................................................8

Table 2: Predicted Range of Lab Effects under Each Model .........................................................15

Table 3: Average Predicted Range of Tester Effects under Each Model ......................................15

Table 4: Average Power Consumption by Test Set-up and Illuminance Value ............................17

Table 5: Coefficient of Variation for Adjacent Test Set-up ..........................................................23

Table 6: Coefficient of Variation for Distance Test Set-up ...........................................................23

Table 7: Coefficient of Lab Effects by Test Set-up under WLS....................................................28

Table 8: Coefficient of Lab Effects by Test Set-up under OLS with Outliers Removed ..............29

Table 9: Coefficient of Lab Effects by Test Set-up under OLS with Outliers ..............................29

Table 10: Joint Statistical Significance of Lab Effects by Test Set-up (WLS Regression) ...........29

Table 11: Coefficient of Tester Effects by Test Set-up and Lab under WLS ................................29

Table 12: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers 

Removed ................................................................................................................30

Table 13: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers ...........30

Table 14: Joint Statistical Significant of Tester Effects by Test Set-up and Lab (WLS 

Regression) ............................................................................................................30

LIST OF FIGURES 

Figure 1: Adjacent Test Set-up ........................................................................................................7

Figure 2: Distance Test Set-up .........................................................................................................7

Figure 3: Average Saturation Point by Test Set-up and TV ..........................................................10

Figure 4: Average Power Consumption for TV 1 by Illuminance.................................................11

Figure 5: Average Power Consumption for TV 2 by Illuminance.................................................11

Figure 6: Average Power Consumption for TV 3 by Illuminance.................................................12

Figure 7: Average Power Consumption for TV 4 by Illuminance.................................................12

Figure 8: Average Power Consumption for TV 5 by Illuminance.................................................13

Figure 9: Average Power Consumption for TV 9 by Illuminance.................................................13

Figure 10: Average Power Consumption for TV 7 by Illuminance...............................................14

Figure 11: Average Power Consumption for TV 8 by Illuminance...............................................14

Figure 12: Average CV by Illuminance Value and Test Set-up ....................................................16

Figure 13: Average CV by TV and Test Set-up ............................................................................16

Figure 14: Observations for TV 1 under the Distance Test Set-up at 12 Lux ...............................27
5

ROUND ROBIN EVALUATION OF TEST SET-UPS FOR DETERMINING POWER 

CONSUMPTION OF AUTOMATIC BRIGHTNESS CONTROL-ENABLED 

TELEVISIONS 

1.1 INTRODUCTION 

 The U.S. Department of Energy (DOE) initiated a television round robin test study to 

assess the precision of distance and adjacent test set-ups for measuring television (TV) power 

consumption. Both test set-ups measure power consumption of TVs with Automatic Brightness 

Control (ABC)atechnology at various illuminance levels. The distance test set-up was derived 

from the ENERGY STAR V. 6.0 Displays test method investigation and uses a specified lamp to 

illuminate the TV sensor from a distance of 5 feet. The adjacent test set-up places the same lamp 

in a closed apparatus directly in front of the TV’s sensor to illuminate the TV sensor. 

DOE evaluated the round robin data in two ways: 

 Performed a regression analysisbusing three regression models to predict lab and tester 

effectsc

for each test set-up, considering all observations recorded during round robin 

testing. This analysis provides a robust estimate of lab and tester effects across all labs 

and all TVs, while using the largest available sample size. The report considers both the 

size of the systematic effect (in watts) caused by a given variable (e.g. a lab or tester) on 

predicted power consumption as well as the statistical significanced

of this effect size. 

Lab and tester effects were determined for the distance and adjacent test set-ups by 

separately performing regression analysis on the observations from each test set-up. 

 

a ABC is a technology which allows TVs to adjust screen brightness based on a room’s ambient 

light. It both reduces power consumption and improves picture quality in low-light situations. 

b Regression analysis refers to the overall analysis performed using different regression models 

(WLS, OLS with heteroskedasticity consistent standard errors, etc.) 

c

 Lab effects are defined as the predicted effect a particular lab testing a TV will have on that 

TV’s measured power consumption when compared to other labs; tester effects are defined as the 

predicted effect a particular tester testing a TV will have on that TV’s measured power 

consumption when compared to other testers in the same lab. If lab effects were zero then all labs 

would on average measure the same power consumption, and if tester effects were zero then all 

testers in a lab would on average measure the same power consumption. One of the main 

purposes of the regression analysis was to determine whether there is enough statistical evidence 

to say whether different labs or different testers within a lab have differences in measured power 

consumption which cannot be attributed to random chance. 

d

The phrase ‘statistically significant’ is used to indicate that 95% of the time, the absolute 

difference between two test results will not exceed a calculated limit of statistical significance, 

except when another percentage is mentioned. This is a standard method of assessing whether a 

given pattern is unlikely to have occurred by chance. 
6

 Performed a precision analysis by calculating the Coefficients of Variation (CVs)efor 

each set (defined throughout the paper as the set of observations under a given test set-up 

on a given TV at a given lux pointf

). 

 Lab and tester effects are considered in the regression analysis, and the precision analysis 

focuses on lab and tester variability for a given TV, test set-up and illuminance value.g

1.2 TEST PROCEDURE 

 Four labs conducted round robin testing using the two test set-ups described in this 

section. Each test set-up measured the power consumption (in watts) of different TV models as a 

function of the ambient illuminance of a room (in lux). For each test set-up, each lab tested each 

TV three times.hTests used the primary HDMI input to play an International Electrotechnical 

Commission (IEC) test clip with no surfaces other than the table the TV was on within two feet 

of the sensor. For both test set-ups, testers used a standard spectrum halogen incandescent 

reflector lamp with a conditioned power supply as the source of illuminance. To minimize 

autocorrelation,i

testers powered down each TV and all testing equipment; TVs were physically 

removed from the test station, and then set up again for the next testing round. Testers also 

stabilized each TV for one hour prior to testing. Labs were instructed to have a different tester 

perform each individual test, if possible. Requiring multiple testers also helps minimize 

autocorrelation and allows for the analysis of tester effects within labs where ever possible. 

1.2.1 ADJACENT TEST SET-UP

 Prior to testing using the adjacent test set-up, each lab constructed an apparatus to 

illuminate the sensor in a closed system. The apparatus had a diameter at one end which fit 

completely around the light source and a diameter at the other end of two inches. The tester 

secured the light source so no light escaped between the light source and the apparatus. The 

tester placed the two inch diameter end of the apparatus flush against the TV bezel, completely 

covering the ABC sensor. The tester aligned the center of the lamp at the same height as the 

center of the ABC sensor with respect to the floor. Figure 1 illustrates the adjacent test set-up.

 

 

e CV is the ratio of the standard deviation to the mean. 

f

 Lux is the SI unit for illuminance. 

g

Set refers to data obtained by different testers and different labs for a given TV, tested with a 

given test set-up (distance or adjacent), and tested at a given illuminance value. 

h

TV 6 was not tested by Lab D. 

i

The correlation which commonly occurs between values measured in sequence. 
7

(Not to scale) 

Figure 1: Adjacent Test Set-up

1.2.2 DISTANCE TEST SET-UP

 For the distance test set-up, testers placed a light source at a distance of five feet from the 

surface of the ABC sensor. To ensure the light entered the sensor at a zero degree vertical angle, 

the distance from the floor to the center of the lamp was set to be equal to the distance from the 

floor to the center of the sensor. Each TV was placed with all four corners equidistant from a 

reference plane (e.g., wall). Testers aligned the center of the lamp to be perpendicular with 

respect to the sensor. Figure 2 shows the set-up for the distance test set-up. 

(Not to scale) 

Figure 2: Distance Test Set-up
8

1.3 TEST LABS AND PRODUCTS TESTED 

 Four laboratories, Lab A, Lab B, Lab C, and Lab D, participated in the television round 

robin test study and DOE selected eight different TV models for testing. Each participating lab 

tested both the adjacent and distance test set-ups three times on each of the eight TVs, except for 

Lab D which did not perform the adjacent test set-up on TV 6 due to technical issues with that 

TV. All four labs took measurements at 0, 10, 12, 35, 50, 100, and 300 lux on each TV, using 

both test set-ups; Lab A and Lab C also took measurements at 75 lux. Table 1 lists the 

characteristics of the eight televisions tested. All TVs meet the ENERGY STAR specification 

Version 5.3.1

Table 1: Characteristics of TVs used in DOE Round Robin Study 

Test Unit Screen Type Screen Size (inches) Manufacture Date

TV 1 LED Backlit 55 March 2010

TV 2 LCD 46 September 2010

TV 3 LED Backlit 42 May 2010

TV 4 LED 46 February 2012

TV 5 LED 47 March 2012

TV 6 LED 32 August 2010j

TV 7 LED 46 March 2011k

TV 8 CCFL 53 May 2009l

1.4 STATISTICAL ANALYSIS METHODS 

To analyze the round robin test results, two primary methods were used: 

 A regression analysis estimated the size and joint statistical significancemof lab and tester 

effects for both the entire sample of round robin data and a variety of subsamples. Three 

regression models were used: 

o Weighted Least Squares regression2(WLS regression) 

o Ordinary Least Squares regression with heteroskedasticity-consistent standard 

errors (OLS regression)n

 

j

Estimated based on release date of TV. 

k

 Id. 

l

 Id. 

m

Joint Statistical Significance is a measure of whether the labs in the round robin are statistically 

different from each other, or whether the testers in a lab are statistically different from each 

other. This is different from individual statistical significance because individual statistical 

significance relies on a reference value to compare against. This is one of the primary ways of 

analyzing lab and tester effects. 

n Heteroskedasticity occurs when the errors in a regression model are correlated with the 

explanatory variables of that model. Heteroskedasticity-consistent standard errors are a method 

of correcting for this in hypothesis testing. Note that most hypothesis testing was performed on 

the WLS model. 
9

o Ordinary Least Squares regression with heteroskedasticity-consistent standard 

errors and outliers removed (OLS regression with outliers removed). 

 A coefficient of variation was calculated for each set to evaluate in-set precision.

1.4.1 REGRESSION ANALYSIS 

 The regression analysisoexamines the round robin data to predict power consumption of 

a given TV as a function of the TV, illuminance value, test set-up, lab, and tester. Each 

regression model predicts the average effect size in watts for each of these variables on power 

consumption and the standard errorp

of that prediction. The regression analysis helped DOE 

understand differences between the distance and adjacent test set-ups, such as whether the 

observations at different labs or the observations of different testers at the same lab are 

statistically different from each other. For more information on the methodology of the 

regression analysis see Appendix D: Regression Analysis. 

The different regression models were used to account for the different properties of the models. 

 The WLS regression gives weights to each observation which are inversely proportional 

to the variance of that observation, which means that most of the weight is given to the 

most typical observation. This helps the model avoid trying to incorporate outliers and is 

more appropriate for hypothesis testing but may underestimate the unidirectional effects 

of values with higher variance. 

 The OLS regression gives equal weights to all observations, which provides a broader 

sample of the observation range but is more susceptible to extreme values and is less 

appropriate for hypothesis testing. 

 The OLS regression with outliers removed was used to better account for the 

observations with moderate variance from the predicted value without being affected by the most 

extreme values. See Appendix E: Outlier Analysis for details on the methodology by which 

outliers were removed.

1.4.2 COEFFICIENT OF VARIATION ANALYSIS 

 The coefficient of variation is calculated as the ratio of the sample standard deviation of a 

given set to that set’s mean. In other words, a set’s mean and sample standard deviation were 

calculated, and then the standard deviation was divided by the mean. Outliers were not removed 

in this analysis due to sample size concerns and because outliers affected the analysis less than in 

the OLS regression. 

 

o

For more detail on the nature of these regressions, see the appropriate regression subsections in 

1.5 Results and in Appendix D: Regression Analysis. 

p

Standard Error is a measure of the error of the sample mean (the average of the observations for 

a given test set-up, TV, and lux value) as an estimator of the population mean (the actual average 

power consumption of a TV). It is calculated by dividing the sample standard deviation of a set 

by the square root of the sample size of that set. Note this is not to be confused with standard 

deviation. 
10

1.5 RESULTS 

1.5.1 SATURATION POINT 

 Sensor saturation occurs when an increase in illuminance does not lead to an increase in 

display brightness (or alternatively power consumption). It is defined in this report as the 

illuminance value at which a TV reaches average power consumption greater than 95% of its 

power consumption at 300 lux. Figure 3 shows the average saturation value for each TV by test 

set-up and Figures 4 through 11 shows the average measured power consumption for each TV as 

a function of illuminance value. For more information see Appendix A: Average Power 

Consumption by Test Set-up, TV and Illuminance Values 

Figure 3: Average Saturation Point by Test Set-up and TV 

0 50 100 150 200 250 300

TV 8

TV 7

TV 6

TV 5

TV 4

TV 3

TV 2

TV 1

Saturation Value (Lux)

Average Saturation Value vs. TV 

Adjacent Distance
11

(Error bars reflect the standard deviation of each set)

Figure 4: Average Power Consumption for TV 1 by Illuminance 

(Error bars reflect the standard deviation of each set)

Figure 5: Average Power Consumption for TV 2 by Illuminance

0

50

100

150

200

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 1

Adjacent Distance

0

20

40

60

80

100

120

140

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 2

Adjacent Distance
12

(Error bars reflect the standard deviation of each set)

Figure 6: Average Power Consumption for TV 3 by Illuminance

(Error bars reflect the standard deviation of each set)

Figure 7: Average Power Consumption for TV 4 by Illuminance

0

20

40

60

80

100

120

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 3

Adjacent Distance

0

20

40

60

80

100

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 4

Adjacent Distance
13

(Error bars reflect the standard deviation of each set) 

Figure 8: Average Power Consumption for TV 5 by Illuminance 

(Error bars reflect the standard deviation of each set) 

Figure 9: Average Power Consumption for TV 9 by Illuminance 

0

20

40

60

80

100

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 5

Adjacent Distance

0

10

20

30

40

50

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 6

Adjacent Distance
14

(Error bars reflect the standard deviation of each set)

Figure 10: Average Power Consumption for TV 7 by Illuminance 

(Error bars reflect the standard deviation of each set) 

Figure 11: Average Power Consumption for TV 8 by Illuminance 

1.5.2 LAB EFFECTS 

 The WLS regression analysis (the only model allowing unbiased hypothesis testing) 

indicated that neither test set-up had statistically significant lab effects, meaning that the lab 

effects for both test set-ups could reasonably be attributed to random chance. The range of 

predicted lab effects (the predicted systematic change in measured power consumption between 

0

20

40

60

80

100

120

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 7

Adjacent Distance

0

50

100

150

200

0 10 12 35 50 75 100 300

Power Consumption (W)

Illuminance (Lux)

TV 8

Adjacent Distance
15

the two labs whose measurements are furthest apart) is presented in Table 2. For more detailed 

results see Appendix F: Coefficients and Hypothesis Testing of Statistical Significant of Lab and 

Tester Effects in Regressions. 

Table 2: Predicted Range of Lab Effects under Each Model 

Test Set-up Adjacent Test Set-up(W) Distance Test Set-up (W)

WLS 0.6 1.8

OLS 0.7 2.4

OLS (Outliers Removed) 1.6 2.3

 Table 2 indicates that the predicted difference between the lowest power consumption 

measured at a given lab and the highest power consumption measured at a given lab is 0.6 W and 

1.8 W for the adjacent and distance test set-ups, respectively, using the WLS model. 

1.5.3 TESTER EFFECTS 

 The WLS regression indicated that neither test set-up had any labs with statistically 

significant tester effects, meaning that the tester effects for both test set-ups could reasonably be 

attributed to random chance. The average range of predicted tester effects across all four labs is 

shown in Table 3. Much of the reason for the higher predicted tester effects for the adjacent test 

set-up is due to Lab B. For more detailed information see Appendix F: Coefficients and 

Hypothesis Testing of Statistical Significant of Lab and Tester Effects in Regressions

Table 3: Average Predicted Range of Tester Effects under Each Model 

Test Set-up Adjacent Test Set-up (W) Distance Test Set-up (W)

WLS 1.38 0.97

OLS 1.45 0.94

OLS (Outliers Removed) 1.07 0.94
16

1.5.4 COEFFICIENT OF VARIATION 

 Average CV for the distance test set-up is 2.2% compared to an average CV of 3.5% for 

the adjacent test set-up. Figure 12 shows the average CV as a function of illuminance for the 

distance and adjacent test set-ups. Figure 13 compares the CV by TV and test set-up averaged 

across all lux values. CVs for each set are included in Appendix F: Coefficients and Hypothesis 

Testing of Statistical Significance of Lab and Tester Effects in Regressions. 

Figure 12: Average CV by Illuminance Value and Test Set-up

 Figure 13: Average CV by TV and Test Set-up

0%

1%

2%

3%

4%

5%

6%

7%

0 10 12 35 50 75 100 300

Average CV (%)

Illuminance (Lux)

Distance Adjacent

0%

1%

2%

3%

4%

5%

6%

TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8

CV (%)

Distance Adjacent
17

APPENDIX A: AVERAGE POWER CONSUMPTION BY TEST SET-UP, TV, 

AND ILLUMINANCE VALUE 

 This appendix gives the average power consumption for each set in watts. It averages 

each of the three observations from each of the labs which measured a TV, so there are twelve 

observations for most of the sets. There are six observations for each set at 75 lux because Labs 

B and D did not measure power consumption at 75 lux, and there are nine observations for each 

set measuring the power consumption of TV 6 (excluding those at 75 lux) because Lab D did not 

measure power consumption on TV 6. 

Table 4: Average Power Consumption by Test Set-up and Illuminance Value 

Test SetupTV 0 lux 10 lux 12 lux 35 lux 50 lux 75 lux 100 lux 300 lux

Adjacent 1 71.5 156.9 150.8 157.9 157.8 157.6 157.6 157.5

2 64.9 67.2 67.1 96.8 112.1 122.6 126.7 126.3

3 46.1 92.6 97.7 101.2 99.0 100.6 101.0 101.0

4 48.2 78.3 78.3 78.3 78.3 77.6 78.2 78.2

5 31.8 75.3 76.3 76.5 76.4 76.0 76.3 76.3

6 28.6 30.8 32.0 39.4 40.4 42.6 42.7 43.0

7 44.2 53.3 54.9 73.4 84.6 85.8 88.7 91.1

8 115.6 125.6 129.6 157.7 156.0 166.5 164.0 167.0

Distance 1 71.6 120.2 132.1 157.2 157.1 155.8 156.8 156.8

2 64.8 71.0 74.9 114.3 126.5 126.4 126.2 126.0

3 46.1 64.8 68.0 92.8 100.1 100.6 101.0 100.9

4 48.1 69.2 70.0 76.8 78.0 77.7 78.2 78.2

5 31.9 45.2 48.8 68.0 73.2 76.3 76.5 76.5

6 28.6 28.9 30.0 40.5 42.3 43.4 45.1 46.5

7 44.0 44.1 44.3 46.1 47.6 52.2 53.9 75.9

8 115.7 115.8 115.9 117.9 123.3 135.3 144.7 167.6
18

APPENDIX B: BOX AND WHISKER PLOTS 

 An analysis was done using the range and average of a given set to analyze precision. 

The box and whisker plots from this analysis are presented in the tables in this section. Box and 

whisker plots are helpful tools to present the dispersion of data in a given set. In each set, the 

minimum and maximum values are represented by the ends of the whiskers (the thin horizontal 

lines on the outside of each box). The 25th percentile and 75th percentile are represented by the 

bottom and top of the box, respectively, and the median (middle value) is represented by the 

middle horizontal line in the box. 

60

80

100

120

140

160

180

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 1
19

60

70

80

90

100

110

120

130

140

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 2

40

50

60

70

80

90

100

110

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 3
20

40

45

50

55

60

65

70

75

80

85

Power Consumption (Lux)

Test Set-up and Illuminance

Box and Whisker Plot - TV 4

30

35

40

45

50

55

60

65

70

75

80

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 5
21

25

30

35

40

45

50

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 6

40

50

60

70

80

90

100

110

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot - TV 7
22

110

120

130

140

150

160

170

180

Power Consumption (Watts)

Test Set-up and Illuminance

Box and Whisker Plot- TV 8
23

APPENDIX C: COEFFICIENT OF VARIATION BY TEST SET-UP

Table 5: Coefficient of Variation for Adjacent Test Set-up

Percent

Lux TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8 Grand 

Mean

0 0.4 0.3 0.9 1.0 0.7 1.0 1.3 0.7 0.8

10 3.2 4.1 10.7 1.3 3.4 4.5 5.1 5.0 4.7

12 13.5 4.5 4.7 1.3 1.0 6.8 5.9 8.7 5.8

35 3.0 9.0 0.7 1.3 0.5 7.0 8.9 9.2 5.0

50 2.9 8.5 7.0 1.3 0.5 4.7 3.8 12.4 5.1

75 2.6 3.7 0.3 0.4 0.3 0.7 0.1 0.8 1. 2

100 2.8 0.4 0.7 1.3 0.5 2.8 8.2 6.2 2.8

300 2.8 0.5 0.7 1.3 0.5 1.5 10.2 1.3 2.4

Grand 

Mean

3.9 3.9 3.2 1.1 0.9 3.6 5.4 5.6 3.5

Table 6: Coefficient of Variation for Distance Test Set-up

Percent

Lux TV 1 TV 2 TV 3 TV 4 TV 5 TV 6 TV 7 TV 8 Grand 

Mean

0 0.5 0.3 0.7 0.7 0.9 1.0 0.4 0.9 0.7

10 4.6 2.6 3.1 2.8 3.4 2.0 0.6 0.9 2.5

12 5.9 3.5 5.0 2.1 2.1 2.4 1.7 0.9 2.9

35 3.1 2.2 5.3 1.8 2.2 1.2 3.7 2.8 2.8

50 3.1 0.8 3.3 1.5 2.4 1.0 4.7 2.5 2.4

75 2.5 0.2 0.3 0.4 0.5 0.7 6.8 4.9 2.0

100 3.1 0.8 0.6 1.2 0.6 1.1 6.9 3.2 2.2

300 3.1 0.8 0.6 1.1 0.6 0.8 11.2 1.2 2.4

Grand 

Mean

3.3 1.4 2.4 1.4 1.6 1.3 4.5 2.2 2.2
24

APPENDIX D: REGRESSION ANALYSIS 

 The regression analysis combined the setsqto increase the number of observations and to 

evaluate information across sets while minimizing bias and assumptions. White’s test3 was 

performed on the data and indicated a high level of heteroskedasticity, so Weighted Least 

Squares (WLS) and Ordinary Least Squares (OLS) with heteroskedasticity robust standard errors 

were used to make estimates more statistically efficient and to reduce potential bias in the 

standard errors (and thus hypothesis tests). 

 WLS is more robust to heteroskedasticity than OLS, so it was the primary model used for 

hypothesis testing. The OLS model with heteroskedasticity-consistent standard errors was 

primarily used for estimating effect size, as the heteroskedasticity does not bias effect size. The 

main difference between the three models in how they estimate effect size is the allocation of 

weights to each observation. The OLS model with outliers included gives equal weight to all 

observations; the OLS model with outliers removed gives equal weight to all observations but 

excludes outliers; and, the WLS model gives weight equal to the reciprocal of the variance of an 

observation to each observation. In other words, the WLS model gives less weight to higher 

variance observations than to lower variance observations. Most hypothesis testing used the 

WLS model, while all three models were used for estimating effect size. 

 The full equation for all three models is provided in Equation 1. This equation was 

applied to the adjacent and distance test set-up samples separately. Note that the results from the 

OLS and WLS regressions were usually very similar, although the predicted lab effects were 

somewhat lower for the distance test set-up than the adjacent test set-up under the WLS 

regression, both for effect size and statistical significance. 

Equation 1 

 

 

 

 

Variable Definitions: 

 Power – measured power (in Watts) of an observation 

 α – coefficient of the predicted equation 

 β, δ, π, ρ – coefficients of the explanatory variables 

 ε – error term or residual needed to balance the equation. Can be calculated by 

subtracting the predicted value for an observation based on the rest of the equation from 

the actual value 

 percsat – illuminance as a percentage of saturation illuminance for a given TV under the 

distance method 

 

q

“Set” refers to data obtained by different testers and different labs for a given TV, tested with a 

given test set-up (distance or adjacent), tested at a given illuminance value. 
25

 δ variables – binary variables for whether a test was performed on a given TV 

 π variables – binary variables for whether a test was performed by a given lab 

 ρ variables – binary variables for whether a test was performed by a given tester 

 After carefully reviewing a variety of scatterplots for subsamples, this equation was 

chosen to adequately account for different effects while minimizing over-fitting the data. The 

variables omitted in the equation to prevent collinearity are: 

 TV 6 

 Lab C 

 Tester A4 

 Tester B2 

 Tester C3 

 Tester D2 

 These variables are included in the analysis but are omitted from the equation to provide 

a reference case. For example, when investigating lab effects all observations are observed at 

Lab A, Lab B, Lab C, or Lab D. The equation assigns a 1 to a lab variable if an observation 

occurred at that lab and a 0 if it did not. However, if all four labs had binary variables associated 

with them, then the equation would not be solvable. For example, suppose there were only two 

labs, Lab E and Lab F, and Lab E measured 1 W higher on average than Lab F. If only Lab E 

was included in the equation it would have a coefficient of 1 which indicates that it measures one 

watt higher than the lab which is not included – Lab F. If only Lab F was included, then it would 

have a coefficient of -1, which indicates that it measures 1 W lower than the lab that is not 

included – Lab E. If both were included, however, their coefficients would reference each other 

and the equation could not be solved. 

 When evaluating lab effects, the tester effect coefficients (ρ coefficients) are held at zero, 

and when examining tester effects the model is restricted to a particular lab. This allows lab and 

tester effects to be examined separately. 

The models make the following assumptions common to all regression analysis4: 

 The equation is correctly specified. In other words, the explanatory variables do not pick 

up the effects of omitted variables. 

 The expected value of the error term is zero, and the expected value of the error term 

subject to the explanatory variables is zero. This assumption is necessary to achieve a 

best fit given the model and equation, and the models do this automatically. 

 Variables are not linear combinations of each other. This is why the variables mentioned 

as omitted were omitted. 

 Errors are normally distributed around 0. This assumption is necessary for accurate 

hypothesis testing and the determination of standard errors. 

 It should also be noted that the models do not include interaction effects. This means, for 

example, that when comparing two TVs at zero lux under the distance test set-up the difference 

between them will be predicted to be the average difference between them predicted in the 

model, but not the average difference predicted between them at zero lux under the distance test 
26

set-up. There still are coefficients to account for the fact that the TVs are measured at zero lux 

and under the distance test set-up but they are independent of which TV is used. This assumption 

is more reasonable for some TVs than others but was decided on for a variety of reasons 

including: 

 Using interaction terms would cause a much larger number of subpopulations for 

comparison, removing many of the benefits of combining the data. 

 The coefficients would have been drawn from a much smaller effective number of 

observations, making each coefficient much more susceptible to outliers. 

 The benefits and drawbacks of using the no interactions models would apply equally to 

the distance and adjacent test set-ups and each lab and tester, and so were not expected to 

change the relevant conclusions. 

 The interaction terms experimented with exhibited an especially high degree of 

heteroskedasticity and increased heteroskedasticity throughout the models. 
27

APPENDIX E: OUTLIER ANALYSIS 

 Outliers were identified and removed from the OLS regression based on the criteria that 

they either deviated from the median observation of their set by 20 W or by 20% of the median. 

This corresponds to being outside the 99th percentile for either watts or percentage. These criteria 

were chosen by balancing the need to remove gross outliers for a good regression equation 

against the risks of no longer having a representative data sample. Figure 14 shows the 

observations in one set for TV 1 and indicates the level of deviation necessary to be considered 

an outlier. 

 Eighteen outliers were removed from the 1398 total observations in the round robin. Of 

the eighteen outliers under the criteria in this section, seventeen corresponded with the adjacent 

test set-up. Also of the eighteen, sixteen occurred at Lab B. This suggests that Lab B had a 

greater propensity to measure outliers, particularly under the adjacent test set-up. 

(The outlier was removed based on the criterion of a 20 W deviation from the median)

Figure 14: Observations for TV 1 under the Distance Test Set-up at 12 Lux 

120

125

130

135

140

145

150

Power Consumption (W)

Normal Observation Median of Observations Outlier
28

APPENDIX F: COEFFICIENTS AND HYPOTHESIS TESTING OF STATISTICAL 

SIGNIFICANT OF LAB AND TESTER EFFECTS IN REGRESSIONS 

 This appendix presents the coefficients and statistical significance of lab and tester 

effects. As noted in Appendix D: Regression Analysis, the WLS is the only regression used for 

hypothesis testing of joint statistical significance due to its robustness to heteroskedasticity. 

 In this appendix, each coefficient uses the appropriate omitted variable as outlined in 

Appendix D: Regression Analysis as a reference value. In choosing the reference tester, the tester 

who performed the most tests was used as the reference case. At Lab C, where each tester 

performed the same number of tests, the reference tester was chosen arbitrarily. The reference 

lab was chosen arbitrarily to be Lab C in all cases. 

 In this appendix the coefficient for a particular lab or tester is the change in expected 

measured power consumption for that lab in relation to the reference lab or tester. To compare 

two labs or testers when neither is the reference case, the coefficient of one is subtracted from the 

other. For example, under the WLS regression for the distance test set-up in Table 6, Lab B has a 

coefficient of 0.72 while Lab A has a coefficient of -1.03. This means that Lab B measured 0.72 

W higher than Lab C on average, and Lab A measured 1.03 W lower than Lab C on average. 

Therefore Lab B measured 0.72 – (-1.03) = 1.75 W higher than Lab A. Note that the choice of 

reference lab and tester has no effect on the calculation of these results. 

 The joint statistical significance of the lab and tester effects are presented as P values, 

which can be interpreted as the probability of obtaining a test statistic at least as extreme as the 

one observed assuming that there are no lab or tester effects. As the 95% confidence interval was 

used, P values lower than 0.05 are statistically significant. Joint statistical significance is used 

because it is not affected by choice of reference lab or tester. 

Table 7: Coefficient of Lab Effects by Test Set-up under WLS 

Test Set-up Adjacent (W) Distance (W)

Lab A -0.21 -1.03

Lab B -0.13 0.72

Lab C Reference Reference

Lab D -0.65 -1.08
29

Table 8: Coefficient of Lab Effects by Test Set-up under OLS with Outliers Removed 

Test Set-up Adjacent (W) Distance (W)

Lab A -0.44 -1.19

Lab B 0.63 0.94

Lab C Reference Reference

Lab D -1.01 -1.32

Table 9: Coefficient of Lab Effects by Test Set-up under OLS with Outliers 

Test Set-up Adjacent (W) Distance (W)

Lab A -0.22 -1.19

Lab B -0.18 1.13

Lab C Reference Reference

Lab D -0.69 -1.27

Table 10: Joint Statistical Significance of Lab Effects by Test Set-up (WLS Regression) 

Test Set-up Adjacent (P Value) Distance (P Value)

Lab Effects 0.96 0.11

Table 11: Coefficient of Tester Effects by Test Set-up and Lab under WLS 

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.14 0.36

Tester A2 -0.23 -0.88

Tester A3 -0.28 0.29

Tester A4 Reference Reference

Lab B Tester B1 -4.25 0.81

Tester B2 Reference Reference

Lab C Tester C1 -0.73 0.13

Tester C2 -0.48 1.38

Tester C3 Reference Reference

Lab D Tester D1 0.11 -0.44

Tester D2 Reference Reference
30

Table 12: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers 

Removed 

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.13 0.29

Tester A2 -0.24 -0.71

Tester A3 -0.30 0.21

Tester A4 Reference Reference

Lab B Tester B1 -2.95 0.68

Tester B2 Reference Reference

Lab C Tester C1 -0.12 0.55

Tester C2 -0.74 1.57

Tester C3 Reference Reference

Lab D Tester D1 0.14 -0.49

Tester D2 Reference Reference

Table 13: Coefficient of Tester Effects by Test Set-up and Lab under OLS with Outliers 

Lab Tester Adjacent (W) Distance (W)

Lab A Tester A1 0.13 0.29

Tester A2 -0.24 -0.71

Tester A3 -0.30 0.21

Tester A4 Reference Reference

Lab B Tester B1 -4.14 0.68

Tester B2 Reference Reference

Lab C Tester C1 -1.10 0.54

Tester C2 -0.46 1.57

Tester C3 Reference Reference

Lab D Tester D1 0.14 -0.49

Tester D2 Reference Reference

Table 14: Joint Statistical Significant of Tester Effects by Test Set-up and Lab (WLS 

Regression) 

Test Set-up Adjacent (P Value) Distance (P Value)

Lab A 1.00 0.97

Lab B 0.17 0.77

Lab C 0.93 0.56

Lab D 0.96 0.73
31

 

1

http://www.energystar.gov/ia/partners/product_specs/program_reqs/Televisions_Program_Requ

irements_V5_3.pdf?0082-7f02 

2

Stata Analysis Tools: Weighted Least Squares Regression. UCLA: Academic Technology 

Services, Statistical Consulting Group. From 

http://www.ats.ucla.edu/stat/stata/ado/analysis/wls0.htm (accessed September 2012).

3 White, H. (1980). "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct 

Test for Heteroskedasticity". Econometrica 48 (4): 817–838

4 Consulting Stock, J. H., & Watson, M. W. (2007). Introduction to econometrics. (2 ed.). New 

York: Addison-Wesley.