Earlier Versus Later Respondent Assessments on Customer Surveys: Is 

There a Difference? 

Howard Bradsher-Fredrick, William Weinig1

Energy Information Administration 

1000 Independence Ave., S.W. Washington, D.C. 20085 

Howard.Bradsher-Fredrick@eia.doe.gov, William.Weinig@eia.doe.gov

Introduction 

Federal agencies are increasingly using customer surveys as one measure among several to evaluate their services and 

products. This is due in part to the more widespread use of performance measures and strategic planning as mechanisms to 

improve the quality of their products and effectiveness in providing services. Toward that end, the Energy Information 

Administration (EIA) administered customer surveys on a sample of customers twice in 2004. With the utilization of 

customer surveys the issue always arises concerning response rates and whether or not the results obtained are truly 

representative of the results if all of the initially surveyed persons had responded. 

While it is impossible or impractical to actually receive a completed survey from every person initially asked to complete a 

survey, it is widely recognized that a reasonable amount of non-response follow-up be undertaken and/or other measures 

taken to minimize non-response bias. However, without actually obtaining surveys from all of the respondents, it is never 

possible to determine (without estimating) the actual non-response bias. In effect, how large is the difference in the opinions 

of those who responded versus those who did not respond? 

While this paper does not presume to provide an answer to that question, the multiple iterations of non-response follow-ups 

enabled the researchers to study the issue of differences in early versus late responses provided by respondents. It is 

generally thought that respondents to customer surveys tend toward having more extreme positions than non-respondents, 

and are thus not representative of the overall customer base. The data obtained through multiple iterations could shed light 

on the issue of whether or not earlier respondents are more extreme in their opinions than later respondents. 

This paper will generally present the findings of the customer surveys and specifically present what was learned concerning 

the responses of earlier versus later respondents. 

Methodology 

The first study was conducted in March 2004; the products evaluated were EIA’s annual publications, The Annual Energy 

Outlook (AEO) and The International Energy Outlook (IEO). The population surveyed was the registrants to the National 

Energy Modeling System Conference. This is an annual conference in which the latest edition of The Annual Energy 

Outlook is discussed. All of the non-EIA pre-registrants to the conference were surveyed; this amounted to a first mailing of 

3972

. The e-mail system was employed as the vehicle for distribution. 

The second study was conducted in September 2004; the product evaluated was the Short Term Energy Outlook (STEO). 

The target population was the non-EIA users of that product. A random sample of 5003

 was drawn from approximately 4,000 

users registered on the Listserv. Again, e-mail was employed as the vehicle for distribution. 

The third and fourth studies were conducted in May-June 2005. The products evaluated again were the AEO and the IEO. 

However, this time the population surveyed were the non-EIA registrants listed on the EIA Listservs for those respective 

 

1

 The authors would like to acknowledge the technical and analytical support provide by Ms. Melissa Marie Draggett. 

2

 The number of registrants was 450. However, some registrants did not provide e-mail addresses and some of the e-mail 

addresses were not usable. 

3

 The number of usable e-mail addresses was 472. 
publications. A random sample of 500 was drawn from the 1,500 users registered on the AEO Listserv4

 and an additional 

500 from the 3,500 users registered on the IEO Listserv5

. 

In all of the studies, a total of four mailings were administered; in the cases of the 2004 AEO/IEO and the STEO studies, this 

was actually due to a time limitation. In the first case, this time limitation involved the conference for which the respondents 

were pre-registered actually taking place. It was thought that actual attendance at the conference6

 by respondents could 

contaminate the results of the survey. In the second case, this time limitation also involved a conference7

 that is widely 

covered in the media, also potentially contaminating the results of the survey. 

Results 

The total number of respondents for the AEO/IEO 2004 study, the STEO 2004 study, the AEO 2005 study and the IEO 2005 

study were 132 (33.2%), 122 (25.9%), 115 (24.0%) and 68 (17.4%), respectively. The cumulative breakdown by mailing is 

shown in Table 1. 

Table 1: Number of Responses and Response Rate Stratified by Mailing Number 

Mailing 

Number 

AEO/IEO (2004) STEO (2004) AEO (2005) IEO (2005) 

1 42 (10.6%) 36 (7.6%) 54 (11.3%) 25 (6.4%) 

2 81 (20.4%) 83 (17.6%) 79 (16.5%) 38 (9.7%) 

3 128 (32.2%) 111 (23.6%) 98 (20.4%) 53 (13.6%) 

4 132 (33.2%)8 122 (25.9%) 115 (24.0%) 68 (17.4%) 

The number of questions employed in evaluating each of the three products varied, but was approximately 14 for each. The 

evaluation questions employed a five-point Likert scale where 1 was the least favorable assessment and 5 the most favorable 

assessment for a particular question. 

For each study, the survey data were stratified into two strata: respondents to the first mailing and respondents to the second

through fourth mailings. It was thought that the highly motivated respondent would quickly respond to the survey without 

requiring further reminders. These first mailing respondents then became operationally defined as early respondents, while 

the respondents to the second through fourth mailings became defined as late respondents. 

Tables 2-6 show the results obtained for early and late respondents for the AEO 2004, IEO 2004, STEO, AEO 2005, and IEO 

2005, respectively. 

 Table 2: AEO 2004 Rating Results 

 N = 132 

Question Early Avg 

(n=42) 

Late Avg 

(n=90) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 4 

& 5 

Late % 4 

& 5 

1. The AEO is used extensively 

in my work. 

4.00 3.78 18.4% 9.5% 78.9% 63.5% 

2. The AEO is clearly written. 4.16 4.04 0% 2.7% 84.2% 78.7% 

3. The AEO is relevant. 4.42 4.27 5.3% 1.4% 86.8% 88.7% 

5. The AEO incorporates the 

changing industry structure. 

3.33 3.60 12.5% 4.4% 41.7% 53.3% 

 

4

 The actual number of usable e-mail addresses was 480 for the AEO. 

5

 The actual number of usable e-mail addresses was 391 for the IEO. 

6

 The conference at issue here was the 2004 National Energy Modeling Conference (NEMS) 

7

 The conference at issue here was the 2004 Winter Fuels Conference. 

8

 The last mailing on the AEO/IEO customer evaluation was conducted on a sample of 40 of the remaining non-respondents. 

This last iteration involved changing the mode to the telephone. 

2
Question Early Avg 

(n=42) 

Late Avg 

(n=90) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 4 

& 5 

Late % 4 

& 5 

6.Appropriate assumptions are 

used in the AEO. 

3.50 3.72 12.5% 4.7% 54.2% 62.8% 

The methodology is suitable to 

the analysis of the following: 

 

7a. End-use Sectors: Residential 3.71 3.68 14.3% 10.5% 64.3% 68.4% 

Commercial 3.86 3.74 7.1% 5.3% 64.3% 68.4% 

Industrial 3.53 3.40 20.0% 5.0% 53.3% 45.0% 

Transportation 3.33 3.50 26.7% 7.1% 53.3% 57.1% 

7b. Supply: Petroleum 3.80 3.72 13.3% 0% 66.7% 66.7% 

Natural Gas 3.71 3.45 14.3% 18.2% 50% 54.5% 

Coal 4.07 3.72 0% 0% 71.4% 66.7% 

Renewables 3.50 3.40 28.6% 13.3% 50.0% 53.3% 

Electric Power Generation 3.88 3.75 18.8% 8.3% 75.0% 70.8% 

7c. Petroleum Refining 3.45 3.31 18.2% 12.5% 36.4% 43.8% 

8. The conclusions reached in 

the report are supported by the 

analysis in the AEO. 

3.67 4.07 16.7% 0% 66.7% 81.0% 

9. The information in the AEO 

is high quality. 

4.10 4.14 9.5% 0% 85.7% 80.9% 

Table 3: IEO 2004 Rating Results 

N = 36 

Question Early 

Avg 

(n=12) 

Late 

Avg 

(n=24) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 

4 & 5 

Late % 4 & 5 

1. The IEO is used extensively 

in my work. 

3.42 3.30 16.7% 17.4% 50% 34.8% 

2. The IEO is clearly written. 4.08 3.88 0% 4.2% 83.3% 75% 

3. The IEO is relevant. 4.27 4.04 0% 0% 81.8% 84% 

5. The IEO incorporates the 

changing industry structure. 

4.00 3.55 0% 9.1% 66.7% 54.5% 

6.Appropriate assumptions are 

used in the IEO. 

4.33 3.91 0% 0% 66.7% 81.8% 

8. The conclusions reached in 

the report are supported by the 

analysis in the IEO. 

4.67 3.83 0% 0% 100% 66.7% 

9. The information in the IEO is 

high quality. 

4.67 4.00 0% 0% 100% 83.3% 

Table 4: STEO 2004 Rating Results 

N = 122 

Question Early 

Avg 

(n=36) 

Late 

Avg 

(n=86) 

Early %1 

& 2 

Late 

%1 & 2 

Early % 

4 & 5 

Late % 4 & 5 

1. The STEO is used 

extensively in my work. 

3.28 3.48 27.8% 18.1% 47.2% 56.6% 

2. The STEO is clearly written. 4.08 4.23 2.8% 2.5% 86.1% 89.9% 

3. The STEO is relevant. 4.28 4.37 0% 1.2% 94.4% 92.8% 

4. The STEO incorporates the 

most up-to-date market 

information 

3.88 4.08 2.9% 8.0% 76.5% 84.0% 

3
Question Early 

Avg 

(n=36) 

Late 

Avg 

(n=86) 

Early %1 

& 2 

Late 

%1 & 2 

Early % 

4 & 5 

Late % 4 & 5 

5. Appropriate data inputs are 

used for the projections in the 

STEO 

3.679

 4.01 0% 0% 50.0% 81.2% 

6.Appropriate assumptions are 

used in the STEO. 

3.88 3.96 0% 0% 75.0% 82.2% 

7. The methodology is suitable 

to the analysis of the following: 

 

 

a. Summer gasoline supply 4.17 3.95 0% 4.8% 83.3% 85.7% 

b. Winter heating fuel supply 4.00 4.00 0% 4.8% 80% 81.0% 

c. Natural gas demand and 

supply 

3.80 4.30 0% 5.0% 60% 95% 

d. Summer and winter 

electricity market 

3.33 4.00 0% 5.6% 33.3% 88.9% 

8. The projections provided in 

the report are in-line with other 

independent sources. 

3.69 3.70 0% 8.6% 61.5% 74.3% 

9. The information in the STEO 

is of high quality. 

4.00 4.13 0% 1.3% 82.3% 81.3% 

10. It would be useful to add 

regional data for prices, 

demand, and supply to STEO. 

3.94 4.24 8.8% 6.0% 64.7% 80.7% 

Table 5: AEO 2005 Rating Results 

 N = 115 

Question Early 

Avg 

(n=54) 

Late 

Avg 

(n=61) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 4 

& 5 

Late % 4 & 5 

1. The AEO is used extensively 

in my work. 

3.49 3.65 15.1% 8.3% 49.1% 56.7% 

2. The AEO is clearly written. 3.89 4.10 3.8% 3.3% 77.4% 86.7% 

3. The AEO is relevant. 4.21 4.22 1.9% 3.3% 86.5% 93.3% 

4. The AEO shows accurate 

results. 

3.79 3.71 2.1% 3.6% 70.2% 64.3% 

7. The AEO incorporates the 

changing industry structure. 3.35 3.53 12.9% 5.9% 54.8% 55.9% 

8.Appropriate assumptions are 

used in the AEO. 3.69 3.65 7.7% 5.9% 65.4% 64.7% 

9. The methodology is suitable 

to the analysis of the following: 

9a.End-use Sectors: Residential 

Sector 3.61 

 

3.83 7.1% 0% 64.3% 66.7% 

Commercial Sector 3.86 3.84 0% 0% 72.4% 68.0% 

Industrial Sector 3.77 3.79 10.0% 3.6% 70.0% 75.0% 

Transportation 3.73 3.75 3.3% 3.4% 63.3% 69.0% 

9b. Supply: Petroleum 4.03 3.9 3.4% 3.3% 79.3% 73.3% 

Natural Gas 3.73 4.07 9.7% 3.3% 67.7% 76.7% 

Coal 3.88 4.04 3.7% 0% 74.1% 72.4% 

 

9

 Significantly lower at the 0.05 significance level. 

4
Question Early 

Avg 

(n=54) 

Late 

Avg 

(n=61) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 4 

& 5 

Late % 4 & 5 

Renewables 3.52 3.76 11.1% 0% 59.3% 63.3% 

Electric Power Generation 3.86 3.97 0% 3.3% 75.9% 66.7% 

9c. Petroleum Refining 3.83 3.81 4.2% 0% 70.8% 74.1% 

10. The conclusions reached in 

the report are supported by the 

analysis in the AEO. 3.77 3.94 10.0% 0% 70.0% 75.8% 

11. The information in the AEO 

is high quality. 4.18 4.18 0% 0% 87.9% 89.5% 

Table 6: IEO 2005 Rating Results 

N = 68 

Question Early 

Avg 

(n=25) 

Late 

Avg 

(n=43) 

Early % 

1 & 2 

Late 

% 1 & 2 

Early % 

4 & 5 

Late % 4 & 5 

1. The IEO is used extensively 

in my work. 

3.65 3.90 8.7% 5.0% 56.5% 80.0% 

2. The IEO is clearly written. 4.33 4.10 0% 7.5% 100% 87.5% 

3. The IEO is relevant. 4.43 4.34 0% 2.4% 100% 92.7% 

4. The IEO shows accurate 

results. 

3.80 3.61 0% 11.1% 63.2% 55.6% 

7. The IEO incorporates the 

changing industry structure. 

3.36 3.81 7.1% 4.8% 42.9% 76.2% 

8.Appropriate assumptions are 

used in the IEO. 

3.64 3.68 7.1% 4.5% 71.4% 63.6% 

9. The conclusions reached in 

the report are supported by the 

analysis in the IEO. 

4.00 4.04 0% 0% 86.7% 87.0% 

10. The information in the IEO 

is high quality. 

4.13 4.13 0% 0% 81.3% 82.6% 

A non-parametric analysis was conducted to determine whether significant differences existed in the responses of the early 

versus late respondents to each question. For the AEO and IEO, no significant differences were found for the responses to 

any question in either 2004 or 2005. For the STEO, the evaluations showed a generally lower score to the questions for the 

early respondents, but only the responses to one question showed a significant difference at the 0.05 level. This was 

Question #5: “Appropriate data inputs are used for the projections in the STEO.” The early respondents scored this 

significantly lower than the later respondents. Thus, our findings were not definitive on the issue of whether or not early 

respondents are different from the late respondents in terms of their evaluations of the product. 

Tables 2 – 4 also show the percentage of early and late respondents who disagreed10 with the designated statements in the 4th

and 5th columns and who agreed11 with the designated statements in the 6th and 7th columns. These computed data are shown 

in order to shed light on the issue of whether early respondents are more extreme in their responses than the late respondents 

(i.e., a higher percentage of 1s and 2s and a higher percentage of 4s and 5s). Again, the data are not conclusive. For the 2004 

AEO, for 13 of the 16 evaluation questions, the early respondents disagreed with the statement more frequently (as a 

percentage) than the late respondents. However, this pattern does not hold for the frequency of respondents agreeing with the 

 

10 Responded to individual statements with either of the following: 1 = Strongly disagree or 2 = Disagree 

11 Responded to individual statements with either of the following: 4 = Agree or 5 = Strongly agree 

5
statements. For the 2004 IEO, 2005 AEO, 2005 IEO and for the STEO data, there appears to be no pattern with respect to 

either the percentage disagreeing or agreeing with the statements. 

Summary and Conclusions 

This analysis of the data that was conducted due to the availability of the data from the customer survey research did not 

provide definitive answers concerning differences in the responses given by early in comparison to late respondents. For the 

AEO and IEO in 2004 and 2005, there were no significant differences between the average responses given to a statement by 

early versus late respondents. For the 2004 STEO, on only one question was there a significant difference at the 0.05 level; 

early respondents scored significantly lower. However, on 11 of 12 questions, the average score given by the early 

respondents was lower than those given by the late respondents. 

When studying the table showing the percentage of responses given that were low, stratified by early and late respondents, no 

noticeable differences were shown, except where a pattern of lower scores was shown for the 2004 AEO. This indicated that 

there was a slight tendency on the part of the early respondents to be harsher on the evaluation statement responses than the 

late respondents. 

When studying the percentage of responses that were high, no noticeable differences between early and late respondents were 

shown. Due to the relatively small numbers of elements in most of the cells, a statistical analysis of cell differences was not 

conducted for either the lower or higher scores. 

6