USABILITY TESTING FOR THE ENERGY

INFORMATION ADMINISTRATION'S INFO-CENTRIC

WEB SITE

Colleen Blessing, Howard Bradsher-Fredrick, Herbert Miller, Renee Miller, Robert Rutchik

Abstract:

In the fall of 2000, the Energy Information Administration (EIA) launched a new “info-centric" design of its Web

site1

. The info-centric design organized EIA’s information by energy topic or channel rather than by the agency's

organizational structure.

The redesign was the product of cognitive testing on the previous EIA Web site and an EIA Joint Application

Development (JAD) session with participants from all parts of EIA who incorporated the results of the cognitive

tests.

In May of 2001, EIA conducted additional cognitive tests on the info-centric site to determine if users found

information more easily and with greater confidence on the new site. This paper presents the results of those tests

and an outline of a design for info-centric sites.

Keywords: cognitive testing, usability testing, participants, Web site, info-centric,

paper-centric, channel

Purpose

This paper shows how EIA has applied cognitive testing principles to usability testing of its

Web site in order to measure the agency’s progress towards its goal implementing an “infocentric” Web site in which EIA’s information is organized by energy topic not organizational

structure or paper publications. The paper will discuss how the agency has used the findings

from each test to improve its testing methodology and Web site in order to move towards its

goal of an “info-centric” site. The paper will cover:

• Background of the EIA Web Site and Implementing an “info-centric” Site

• 1999 and 2000 Usability Tests

• May 2001 Usability Test

• Next Steps

• Summary

Background of The EIA Web Site and Implementing an “Info-centric” Site

In 1993 and 1994, the Energy Information Administration (EIA) began the move from a paper

publication agency to a “paperless” or electronic disseminator of information. When EIA

rolled out its Web site in July 1995, the philosophy was to get information and data on the site

and “out there” quickly. Visual Web site design and the organization of material on the site

1

John Pearson and John Weiner, “A More Info-centric EIA,” paper presented at the semi-annual meeting of the

American Statistical Association Advisory Committee on Energy Statistics (Washington, DC, October, 1999).
 

to make it more usable for users would be addressed later. Now that the site has gone from a

little over 5,000 user sessions in July 1995 to nearly 700,000 users sessions in May 2001,

improved usability has become imperative.

Since 1995, EIA has collected Web site statistics and systematically categorized and studied

user calls and other feedback. Using this information, EIA has twice made major redesigns to

its home page in attempts to allow customers to more easily and quickly find the information

for which they are looking. In early 1999, EIA decided to modify the design and organization

of its Web site from a “publication centric” design that reflected the agency’s organization

structure to an “information-centric” design. EIA’s goal:

Our [Web] processes have been systematically designed to make it easy for

customers to obtain, understand, and use our information conveniently for their

purposes. From their vantage point, EIA appears seamless. For example, to

find the information they need on our Web site, customers will not need to

know how EIA is organized, on which surveys the data were collected or which

individuals to call. If they want to talk directly with an EIA person, they can

easily contact people familiar with the information they seek. 2

EIA’s long-term plan was to redesign its Web site, beginning with the Home Page and moving

to the second and lower level pages, around this principle.

1999 and 2000 Usability Tests

Literature Review and Methodological Guidance

To start the process of turning this principle into reality, EIA asked its Web Cognitive and

Usability Evaluation Team to test the [then] current EIA Web site and future designs to

determine:

• The ability of users to locate information on the EIA Web site.

• Whether users unknowingly locate incorrect information when performing specific

searches.

• Why users locate incorrect information.

• The navigation behavior of users.

For guidance in formulating possible experimental designs to test the EIA Web site, the Team

reviewed the available literature on usability. Specifically, we looked at:

• Research on the Canadian Statistics Area of the Statistics Canada Website,

(http://www.statcan.ca) presented by Phase 5 Consulting Group Inc. and sponsored by

Statistics Canada. In this 1997 study, Statistics Canada conducted an evaluation of their

Web site to describe the interactions of their users with the site and to identify any

issues regarding content, structure, or organization that caused users problems in

2

Ibid., p. 1.
 

 

locating information. They completed a total of 97 think-aloud interviews3 with a

seven-minute limit on each exercise. EIA, after experimenting with other approaches,

would adopt the think-aloud approach with a time (ten minutes) limit on each exercise.

• Facilitating Statistical Information Seeking on Websites: Intermediaries, Organizational

Tools, and Other Approaches, a report to the Bureau of Labor Statistics in August 1998

by Carol Hert. This study builds on the prior work done by Gary Marchionini and Carol

Hert. It investigated user groups of the Fedstats Web site and the tasks attempted by

end-users and intermediaries.

• The team also studied, Web Site Usability: A Designer’s Guide, by Jared M. Spool of

User Interface Engineering. This study describes how well information-rich Web sites

actually work when people use them to find specific answers.

American Statistical Association (ASA) Advisory Committee on Energy Statistics Test and

Advice

The first test the Team conducted was at the 1999 American Statistical Association (ASA)

Advisory Committee on Energy Statistics meetings.4 The purpose of the ASA Committee is

to provide EIA with technical advice on survey methodology and modeling issues. All

participants were familiar with EIA data; some were familiar with the EIA Web site. The team

wanted the ASA Committee to provide advice on which of two experimental designs, a focus

group format or an individual think-aloud format, would be optimal for usability tests.

At the end of the exercise period, committee members made a number of suggestions.

• Use the “think-aloud approach.” It will provide richer information.

• Phrase the test questions as people would normally ask them, rather than using EIA

jargon.

• Videotape the tests.

• Cut down the length of the introduction protocol; it is too long and too formal.

• Revise a demographic survey in which the Team asks participants about their level of

Web expertise and familiarity with energy terms prior to the exercises.

Based on the results of the tests of the ASA committee and their advice, the team decided to

use the “think aloud” approach for EIA testing. We also incorporated most of the other

suggestions into its testing design.

3

In the think-aloud procedure, test participants are asked by the interviewer to say what they are thinking,

feeling, and doing during the test. Hence “think-aloud.”

4 Colleen Blessing, Howard Bradsher-Fredrick, Renee Miller, Robert Rutchik, Antoinette Ware-Martin

“Cognitive Interviewing: Applications to Evaluating the Energy Information Administration’s Web Site,”

paper presented at the annual meeting of the American Statistical Association (Baltimore, MD, August 1999)
 

July 1999 Test

The next test of the EIA Web site was conducted in July 1999 . Seventeen participants were

tested. Each was given four consecutive exercises in which they were asked to find a particular

piece of data on the EIA Web Site and a maximum of ten minutes to complete each exercise.

The participants were a mix of EIA and DOE employees and industry, the media, and the

general public. The tests were conducted at the Bureau of Labor Statistics (BLS) cognitive

lab, a makeshift cognitive lab at EIA, and private homes.

5

Methodologically:

• The think-aloud procedure worked.

• Phrasing the questions in common English, instead of agency jargon, made them

understandable to the participants.

• Ten minutes provided sufficient time to complete an exercise.

Substantitively, several common themes emerged:

• EIA uses a lot of technical terms, creating its own language, making it difficult for users

to find the information they need.

• Search strategies were nearly as varied as the number of participants.

• The team also observed, and some participants said this explicitly, that the larger type

for the links on the right side of the pages signified to them that EIA placed more

importance on that information than on the links on the left with smaller type. (This

was not EIA’s intention.) This is consistent with, for example, Dillman’s work on

survey designs that “words, numbers, and graphic symbols can be increased in size to

attract the respondent’s eyes, or decreased in size to avoid commanding the

respondent’s attention. ”

6

Findings to the debriefing questions (Attachment 1) were:

• The respondents were asked if they were confused while using EIA’s site. Several

responded that the site was confusing rather than their being confused by the site.

• Participants reported widely varying degrees of confidence that they found the correct

information.

• Participants who reported being “totally confident” did not necessarily find correct

answers, while some subjects who chose the neutral response, “in between,” did in fact

locate some correct answers.

5

Ibid.

6 Don A. Dillman, Mail and Internet Surveys: The Tailored Design Method, 2nd Ed. (New York: John Wiley &

Sons, Inc.), p. 106.
 

 

 

2000 Tests

As a result of the July 1999 test, EIA realized that substantial changes were needed to improve

the site. A Joint Application Development (JAD) session was held with participants from all

parts of EIA. From this session, the “all roads lead to Rome” principle emerged. We wanted

there to be several ways to find any particular piece of information.

Based on that principle, we designed a new home page. Before doing formal laboratory

testing of an on-line prototype, the Web Usability Team conducted two prototype tests of the

new home page at EIA’s March 2000 National Energy Modeling System Conference in

Crystal City, Virginia. EIA tested 69 people. The first test displayed to participants two paper

designs. One was a “clean” screen with only the information headings, or channels, displayed.

The second design listed all the subheadings under their channels. In the second test, EIA

showed participants an electronic mockup of the design with subheadings to see if people

would notice them and if they liked them. The Web team found that participants preferred a

mouse-over version in which users placed the cursor on or channels to show a pop-up menu of

the choices under that channel.

Encouraged by and building upon the NEMS tests, EIA decided to develop the on-line

prototype of the Home Page with mouse-overs and to test it formally. The tests were

conducted in July 2000 at the Bureau of Labor Statistics Cognitive Lab. EIA tested 15

participants. They were a cross section of EIA data users: Media (4), associations (3), library

(1), government (2), academics (4), and research (1). The setting was:

• Participant and interviewer in an office with a workstation.

• Participants were asked to find answers on the EIA Web site to four consecutive

randomly selected questions with a ten-minute maximum to answer each question.

• EIA note takers behind the one-way mirror in order to observe without being a

distraction.

• Videotaping of the session, with the participant’s permission.

Findings:

• Most users who had used the current EIA site preferred the new Home Page design

• Most users realized the “meat” of the page was in the middle, the Channels.

• Users seemed able to select an appropriate first button very easily.

• More people were successful in finding the correct answers in this test than in the July

1999 test.

• Data appears to support a positive view of the new design. There was a much higher

correlation, r-squared = 0.42, between participants expressing confidence that they

found the right answers and actually finding them than in July 1999, r-squared = 0.10.
May 2001 Tests

EIA implemented the new Home Page in October 2000. EIA also redesigned the second level

pages to have the same look and feel as the Home Page, in short to make navigation more

transparent. The purpose of the May 2001 tests was to evaluate the effectiveness of the new

design.

Methodology

The Web Usability team used the same methodology that it had used in the previous year’s

tests. Like the 2000 tests, the 2001 tests were conducted at BLS Cognitive Lab. Again, they

were videotaped, with the participants’ permission, and team members observed carefully and

took notes. Also, as with the 2000 tests, 15 participants were tested. Though not randomly

selected, they were, the team believes, a cross section of EIA users. The team recruited

participants at:

• The 2001 National Energy Modeling System Conference (an EIA-sponsored

conference)

• From personal contacts

• Media (an increasing consumer of EIA data)

• Bureau of Labor Statistics volunteer list

• New EIA employees

The breakdown of participants was 9 non-EIA Federal and an EIA employee plus a consultant,

an oil company person, an undergraduate student, and one each from the trade press and a

state agency. Participants were given four or five randomly selected exercise questions, with a

ten-minute maximum to answer each question (Attachment 2). Total interview time (including

debriefing) was 35 to 55 minutes.

Findings

First, participants navigated through the Home Page and second level pages more quickly than

in the previous tests. Several reported, unprompted, that they like the info-centric approach.

The participants also reported themselves to be:

• Slightly less expert in using the Internet than year 2000 participants. However, the team

thought that they actually showed more expertise in using the Net than 2000 users.

Maybe this shows that, though people are becoming more familiar with the Internet, the

bar of what is Net expertise is higher.

• Slightly more familiar with the energy industry and energy terms than the 2000

participants
 

The percent of correct answers and unanswered questions was instructive.

• Percent of correct answers dropped from 54 percent in 2000 to 40 percent in 2001. In

2001, we used many of the questions on which subjects performed poorly on the 2000

testing, so this lower success rate was not a surprise.

• Percent of all questions left unanswered increased from 15 percent to 35 percent.

• The 2001 participants were much more aware of obtaining correct or incorrect answers

than the 2000 participants. As reported, the correlation between participants' actual

number of correct answers versus their reported "confidence that they found the correct

information" was r-square = .68. This is surprisingly high and much higher than last

year’s .32. Maybe the large number of unanswered questions can partially explain this.

Finally, the average time to answer a question was 6 minutes and 38 seconds. This varied from

3 minutes and 15 seconds for “What was the projected price of gasoline this summer” and 4

minutes and 19 seconds for “How much profit did U.S. oil companies make in 1999” to 10

minutes and 26 seconds for “How much electricity was generated from solar energy in the

United States in 1993?”

Another finding of note was that “all roads do not necessarily lead to Rome.” Earlier, the

report discussed that one of the guiding principles in the Channel design of the EIA Web site

was that “all road should lead to Rome;” that there should be multiple paths to the same data.

Several users explicitly noted that they were considering several ways to begin their search.

Yet, all roads still did not lead to Rome on the EIA site. For example: If you click forecasts

>natural gas, you go to a different place than if you click natural gas> forecasts.

The Team’s most important finding was at the third and lower level pages. Web users expect to

find information fast. Our participants had problems quickly finding the information they were

asked to look for. The reason appeared to be because the pages often were paper publication

pages transposed onto the Web. Looking for and viewing information on a Web monitor is

fundamentally different from looking for and viewing it on a printed page in a paper publication.

The tests demonstrated this problem in several ways:

• Participants often found text reports when they wanted a table. Encountering text when

looking for a table, they were often stymied, frustrated, and confused. “Expecting a table.

It is here somewhere.”

• Many participants said there was too much text to read and that the paragraphs were too

long. “Reading text is hard to do. ”

7

• Participants reported that reading was made even harder by:

4”Creative wallpaper” backgrounds

4Lack of high contrast between the text and background

4Small, almost unreadable, font sizes

7

Paul Muter, “Interface Design and Optimization of Reading of Continuous Text,” H. van Oostendorp and S. de

Mul, Eds., Cognitive Aspects Of Electronic Text Processing (Norwood, N.J.: Ablex Publishing Corporation,

1996). Muter cites research that says reading from a videotex terminal is about 28 percent slower than reading

from paper.
 

• Participants did not like text tables with lots of horizontal and vertical scrolling. For

example, they could not remember row and column headings and would have to scroll

back to the top of a table to see if the column heading there was for the total they had

found at the bottom of the table.

• Participants would get to a page, especially through the search, and not know where

they were. There was no context, like there would be for a printed report. There was no

table of contents, index, executive summary, or title page for the data. Participants also

noted that in many places on the site the lack of context was reinforced by:

4The titles of tables and graphs (and even text sections) not being intuitive

4Jargon words not being defined

4Acronyms not being spelled out

4The source of the data not being cited

4Links to supporting or explanatory information not being supplied

Participants, often unsolicited, made a number of recommendations to make navigation

quicker, data extraction easier, and tables and graphs more “stand alone.” The latter is a

necessity for presenting data on the Web. There is often only one table or graph on the screen

at a time. Users view it independently of others on the site. To make Web pages, particularly

tables and graphs, more “stand alone” or self-explanatory than paper publication ones:

• Use bullets, headers, high lights, jump links into documents.

• Put column totals and current data points at the top of table, rather than the bottom. On

paper you can see the whole page and your eye can go to the total at the bottom. This is

often not so on Web.

• Do not use text tables.

Implementing these suggestions would help EIA achieve compliance with Section 508, an

amendment to the Rehabilitation Act of 1973. Section 508 requires that the public and Federal

employees with disabilities have access to, and the use of, electronic and information

technologies in a way comparable to individuals without disabilities. For example, “creative

wallpaper” behind low contrasting text would not be in compliance with Section 508.

American Customer Satisfaction Index

Support for the 2001 test findings also comes the American Customer Satisfaction Index

Survey done for EIA in early 20018. The ACSI survey conducted telephone interviews with

260 of EIA’s information users. They asked about three areas of electronic services:

• Dissemination services

• Information quality

• Web Site structure

Of these three areas, the quality of the information on the EIA Web site scored the highest,

with a calculated index rating of 83. In the ACSI survey, any score over 80 on a driver of

8 American Customer Satisfaction Index, “Report on the Energy Information Administration,” (April 2001).
 

 

satisfaction is considered a very strong score, an excellent score. Clearly, Web site users find

the information to be very useful and accurate. Users gave a reasonably good rating to EIA

for updating web pages with current information in a timely manner and for the variety of

electronic information services, such as the Web site, listserv, references to experts and the

help desk. The Web site structure scored about the same as dissemination, although users

gave the lowest relative rating among the components to the effectiveness of the navigation

tools on the Web, such as the search engine, the A to Z search, and menus, links, and other

buttons 9.

ASCI concluded that “the organization and navigation of the Web site is the prime candidate

for focusing resources, as it had the lowest score and does have a significant impact on

quality. ”

10

Next Steps

EIA is at a crossroads in Web development in moving from paper/office-centric configuration

to an electronic information-centric configuration. EIA's customers are weighing in heavily in

favor of more consistent accessibility. They are telling our usability testers (us) that it is not

good enough simply to have everything on the Web, because they still cannot find what they

are looking for on our Web site many times.

To cross the Rubicon to a fully implemented info-centric Web site, EIA’s Information

Products and Services Committee, the EIA Committee that sets EIA Web site policy and

manages the site, has developed a three-stage plan. Stage 1 will be changes, many developed

from recommendations from our team’s testing, that can be implemented easily before the end

of the year.

Stage 2 will be changes that will be implemented in by the end of 2002. An example of this

would be to develop a “My EIA” page, similar to “My Yahoo,” that will give users the ability

to create a custom information service.

Stage 3 will be fully implemented after the end of 2002. Its goals are to:

• Eliminate all “publications” except those called for the EIA Strategic Plan

• Replace “publications and tables” with information packages designed for specific user

classes or make them customizable by individual users. For example, there would no

longer by a Petroleum Marketing Monthly but a information topic called Gasoline

The complete transformation to an info-centric site where all roads lead quickly to Rome will

not be easy. The plan will need to be a top priority with the agency in both words and

resources. It will take money, time, a great deal of effort, and a fundamental change in how

EIA thinks about structuring its data for dissemination.

9

Ibid., p. 9.

10 Ibid., p. 10.
 

Summary

In the past EIA thought we were successful if we had all our data and information available

electronically on the Web. Organizations that put statistical information on the Web can no

longer just make a Portable Document Format (PDF) or text file of a paper publication, post it

on a Web, and expect readers to find and extract information just like they did from the paper

publication. Our usability research has helped illustrate that this definition of a successful

Web site no longer is valid.

People read material differently on a computer monitor than from a paper publication. In a

paper publication, when a reader wants to see a total, a unit of measure, a footnote, definition

for a term, all he or she needs to do is scan to the top or bottom of the page or look in a

glossary or appendix in the back of the publication. This is much more difficult on a computer

monitor. Our team has seen these differences repeatedly in our usability tests.

Customer expectations are rising. Users are increasingly becoming aware that the process of

reading and extracting information from a computer monitor is more difficult than from a

paper publication and they want agencies and establishments to make their Web products

Web-friendly. They want to see quickly and clearly on the monitor all the information they

need. Presentation of the content is as important as the content itself.

The best way to know for sure how customers are using your site and how successful they are

in finding information is to watch them work. EIA plans to continue to conduct usability tests

to determine our progress in moving toward a user-friendly, understandable Web site.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Attachment 1:

July Debriefing Findings

Feel About Site N

Completely confused 0

Somewhat confused 7

In between 3

Most things made sense 6

Everything made sense 1

Ease of Searching N

Very Difficult 3

Difficult 2

In between 7

Fairly Easy 4

Very Easy 1

Confidence N

Not at all 3

Not very 2

In between 6

Fairly 4

Totally 2

Site Rating N

One of worst 0

Not as good as most 4

About same 8

Better than most 3

One of best 2
Attachment 2:

Questions for May 1-3, 2001, Usability Test

BLS Cognitive Laboratory

1. How much energy was consumed in the United States in 1999?

96.6 quadrillion Btus

http://www.eia.doe.gov/pub/pdf/multi.fuel/aer1999/sec1_5.pdf

2. What was the price of electricity in Massachusetts in 1985?

$24.34 (nominal dollars) per million Btus

http://tonto.eia.doe.gov/FTPROOT/multifuel/037697.pdf

3. How much carbon dioxide was emitted in the United States from using coal to

generate electricity in 1999?

1,918,146 thousand short tons (11,295 + 6,547 + 1,900,304)

http://www.eia.doe.gov/cneaf/electricity/epav2/html_tables/epav2t25p1.html

1,787,910 metric tons

http://www.eia.doe.gov/cneaf/electricity/page/co2_report/co2emiss.pdf

4. What percentage of oil used in the U.S. in 2000 was imported?

49.6% 9.61 (net imports)/19.39 barrels per day

54.4% 10.55 total imports/19.39

55.6% (from petroleum page using total imports)

http://www.eia.doe.gov/pub/pdf/multi.fuel/aer1999/sec5_5.pdf

57%

5. How much electricity was generated from solar energy in the United States in 1993?

0.9 billion kilowatthours 

http://www.eia.doe.gov/pub/pdf/multi.fuel/aer1999/sec8_7.pdf

6. What state had the highest residential electricity prices in 1999?

Hawaii 14.30 cents http://www.eia.doe.gov/cneaf/electricity/esr/t12.txt

7. How much oil did Russia export in 1999?
3.96 million barrels per day http://www.eia.doe.gov/emeu/cabs/rusexp.html

8. How much electricity was generated using natural gas in the United States in 1999?

565 billion kilowatthours

http://www.eia.doe.gov/cneaf/electricity/epav1/elecprod.html#tab5

9. How much natural gas will be produced in the U.S. in the year 2020?

About 30 (29.79 to be exact) quadrillion Btus

http://www.eia.doe.gov/oiaf/aeo/images/figure_5.jpg

10. How much profit did US oil companies make in 1999?

22.9 billion dollars Http://www.eia.doe.gov/emeu/perfpro/btab02.html

11. How much did the demand for electricity grow in California between 1990 and

1999?

11.3 percent

12. What was the year of the highest gasoline prices in the U.S.?

1981 /pub/energy.overview/monthly.energy/txt/mer9-4

13. How much oil was in the Strategic Petroleum Reserve in January 2001?

541.7 million barrels

http://www.eia.doe.gov/pub/oil_gas/petroleum/data_publications/weekly_petroleum

_status_report/current/pdf/table03.pdfWhat was the average residential price for

heating oil in the United States as March 19, 2001?

$1.407 per gallon

http://www.eia.doe.gov/oil_gas/petroleum/special/heating_update/heating_update.ht

ml

14. What percent of U.S. electricity was used in California in 1999?

6.5 or 7 percent

http://www.eia.doe.gov/cneaf/electricity/esr/t06.txt
15. What is the projected price for gasoline for this summer?

$1.49 per gallon

http://www.eia.doe.gov/emeu/steo/pub/special/summout/2001/summer_gas_outlook.

pdf

16. How many commercial buildings were there in the Northeast in 1999?

677,000 buildings

http://www.eia.doe.gov/emeu/cbecs/char99/prelim1.htm#table1

Navigation Questions

17. What does “click for the text menu” mean to you?

You get all the drop-down menus printed on the screen

18. What do you think you would get if you clicked on the A-Z button?

A list of all major topic items