PERFORMANCE MEASURES FOR FEDERAL AGENCY

WEBSITES:

FINAL REPORT

To Sponsoring Agencies:

Defense Technical Information Center

Energy Information Administration

Government Printing Office

by

Charles R. McClure, Ph. D. <cmcclure@lis.fsu.edu>

Francis Eppes Professor, and Director

Information Use Management and Policy Institute

School of Information Studies, Florida State University

Tallahassee, Florida 32306

Voice: 850-644-8109; Fax 850-644-9763

J. Timothy Sprehe, President <jtsprehe@jtsprehe.com>

Sprehe Information Management Associates, Inc.

4201 Connecticut Ave., NW, Suite 610

Washington, DC 20008

Voice: 202-364-5300; Fax: 202-364-5302

and

Kristen Eschenfelder, Ph.D.

Assistant Professor

University of Wisconsin

Madison, WI 53706

October 1, 2000
PREFACE

The U.S. Energy Information Administration, the Government Printing Office, and the

Defense Technical Information Center funded the study that resulted in the information

reported here. The study was conducted between March, 2000 and September, 2000.

Given the federal government’s emphasis on developing E-services, E-commerce, and Eresources via the networked environment, these three agencies took a leadership position

to support a study that would assist them (and hopefully other agencies) to better assess

the services provided via their websites.

The authors appreciate the assistance participants from these agencies provided in

developing assessment techniques, reviewing their agency procedures and activities, and

working directly with the investigators. This effort, however, should be considered as a

first step in developing assessment techniques and specific performance measures to

assess web based services.

The investigators wish to acknowledge the assistance of others who contributed to

completion of this study. Bruce Fraser, Bruce Smith, and Goldie Burton, all at Florida

State University, Information Use Management and Policy Institute, contributed to early

drafts of portions of this report.

The investigators believe that the findings presented in this study will be an important

means to move forward in developing future assessment techniques and performance

measures. The rapidly changing networked and web environment means that assessment

techniques and performance measures must also continue to evolve. Beyond the efforts

reported here, much work remains to be done in this area.

Charles R. McClure

J. Timothy Sprehe

Kristen Eschenfelder

October 1, 2000
TABLE OF CONTENTS

Preface

Executive Summary E-1

Chapter 1 Introduction 1-1

Chapter 2 Federal Policies Affecting Website Development

and Management 2-1

Chapter 3 Review of Agency Practices 3-1

Chapter 4 Proposed Performance Measures 4-1

Chapter 5 Key Issues and Next Steps 5-1

Appendices

Appendix A Criteria for the Evaluation of Federal Websites A-1

Appendix B Website Performance Measurement and Evaluation

for GPO Access B-1

Appendix C Example: Proposal to Study Agency Compliance

with Federal Website Policies C-1
McClure, Sprehe & Eschenfelder E-1 Federal Website Performance Measures

Performance Measures for Federal Agency Websites

EXECUTIVE SUMMARY

1.0 INTRODUCTION

As federal agencies move to the Web environment as a primary means to disseminate

information to the public, they require performance measures concerning the extent to

which their websites are successfully presenting and conveying the government

information the public needs to access and use. Performance measures for rating the

success of federal agency websites in meeting their goals are consistent with the

provisions of the Paperwork Reduction Act Amendments of 1995, the Clinger-Cohen

Act, the Government Performance and Results Act (GPRA), and OMB Circular No. A130. Foremost among these is GPRA, which mandates the adoption of a strategic and

annual planning process tied to budget and authorization cycles and based on established

and measurable performance indicators for every program.

The purpose of this study was to develop performance measures that will assist agencies

to assess the quality and usefulness of their websites and to improve public information

access services of those websites. The study was sponsored jointly by three federal

agencies: the Defense Technical Information Center, the Energy Information

Administration, and the Government Printing Office. The investigators conducted a

literature review, reported in Chapter 1. They undertook an analysis of laws and policies

affecting website performance measures, reported in Chapter 2. They met with agency

representatives individually and collectively and analyzed documents provided by the

agencies, as Chapter 3 explains. Chapter 4 describes the performance measures that

emerged from the study. Chapter 5 sets forth key issues and recommends next steps.

The study employed a multi-method approach to data collection and analysis. Given the

relatively short time line for the study and the limited resources available the following

approaches were employed:

• Literature Review. The study reviewed selected literature and reports from both

inside and outside the government.

• Group Interviews with Participating Agency Representatives. The

investigators (1) met individually with representatives of each participating

agency, (2) met with the participants as a group, and (3) conducted a number of

individual follow-up discussions with representatives of the three agencies about

activities and procedures for assessment in their agency.

• Best Practices. Each of the three participating agencies provided the

investigators with (1) descriptions of their assessment techniques, and (2)

documentation and reports related to these assessment techniques; these materials

were reviewed and analyzed by the investigators.

• Policy Analysis. The investigators reviewed existing federal policies, laws, and
McClure, Sprehe & Eschenfelder E-2 Federal Website Performance Measures

guidelines to produce an overview of policies affecting the development and

management of federal websites.

• Survey. The investigators conducted a survey at the FedWeb 2000 conference

held in March, 2000 at Bethesda, MD, that provided a snapshot of current

practices and key issues of concern to federal officials regarding the assessment of

federal websites.

• Participant Review. Officials from the three participating agencies had a

number of opportunities to review and discuss preliminary findings and reports

developed by the investigators.

Website evaluation can be defined as the use of research or investigative procedures to

systematically investigate the effectiveness of a web based information system on an

ongoing basis. Evaluation is used in conjunction with a number of activities:

• Planning, Goal Setting and Determining the Degree to Which an Information

Systems Organization is Meeting Set Goals

• Decision-making and Resource Allocation

• Determine the Effectiveness of a Previous Decision

• Determine the Degree to which the System/Service Adds Value to the

Organization

• Provide Trend Data to Assess Change Over Time

• Contribute to Continuous Improvement Efforts and Benchmarking

• Identify Problems and Possible Solutions

• Empower Organizational Actors to Seek and Enact Solutions

• Develop Accountability

• Organizational Learning

Performance measures are the tools that enable evaluation. Performance measures

typically provide measurements of the following:

• Inputs and Efficiency. The use of resources in providing or accessing web

services.

• Effectiveness. How well the networked information service met the objectives of

the provider, user or customer.

• Outputs. Indicators of the products or services resulting from the use of those

resources.

• Extensiveness. How much of a service the network provides.

• Service Quality. How well a service or activity is done.

• Impact. How a service made a difference in some other activity or situation.

• Usefulness. The degree to which the services are useful or appropriate for

individual.

• Adoption. The extent to which institutions or users integrate and adopt web

resources in organizational or individual activities.

One can see federal agency web system evaluation as a special subset of web information

systems (IS) evaluation that in turn is a subset of general IS evaluation. IS evaluation has
McClure, Sprehe & Eschenfelder E-3 Federal Website Performance Measures

become an increasingly important topic within the competitive U.S. business

environment. The last five years has seen a surge of interest in website evaluation.

Federal website evaluation has been ongoing since the inception of federal websites. One

early landmark was the World Wide Web Federal Consortium publication of suggested

guidelines for federal website development.

In summary, website evaluation is an important part of the ongoing management and

enhancement of federal agency websites. Evaluation, whether as part of a formal

planning process or as a stand-alone activity, can provide managers with key information

that can aid decision making. Although web sites are a relatively new medium, IS

evaluation has a long history, providing tools and lessons learned that federal web

managers can apply to their web evaluation efforts. Current academic research on federal

agency web evaluation also provides strategies and tools that agencies can adapt for their

efforts.

2.0 FEDERAL POLICIES AFFECTING WEBSITE DEVELOPMENT AND

MANAGEMENT

Federal information policies establish the legal and procedural framework in which

government information and services are made available to the public. The study provides

an introductory review of selected U.S. Federal information policy instruments that affect the

development and management of Federal websites. This listing is not comprehensive, but rather

offers a general sense of the range of existing Federal policies that should be considered in

the development, management, and evaluation of websites. Issues that Federal agencies need

to consider in developing websites include information security, information privacy,

information access, electronic records management, and intellectual property.

Table 2.1 is a summary presentation of the study’s elaboration of federal policies affecting

agency websites. For each topic the table shows the relevant statute, executive order, or

other key document, plus implementing policy guidance, if any. The right hand column

summarizes the implications for websites. The text of Chapter 2 summarizes the policy

instruments and includes pertinent quotations from the instruments themselves.

3.0 REVIEW OF AGENCY PRACTICES

The report provides an overview of evaluation and performance measures practices at the

Defense Technical Information Center (DTIC), the Energy Information Administration

(EIA), and the Government Printing Office (GPO) current as of July 2000. On March 15

and 16, 2000, the investigators interviewed agency personnel from each of the

participating federal agencies in site visits to the agencies. Each of these agencies

supplied documents in answer to the investigators' queries.

From each agency, the study team requested the following:

x

I. Set of documents by which the agency evaluates its website material.
McClure, Sprehe & Eschenfelder E-4 Federal Website Performance Measures

II. What types of statistics and performance measures is the agency currently

using for its website?

III. What overall evaluation methodologies are currently in place and how are

they conducted/operated?

IV. Can we determine the types of reports generated by the agency evaluation

efforts and the success or quality of these reports?

V. To what degree are information policy instruments/guidelines (as per the

compilation in I) being considered in the documents and reports?

To the above questions, the investigators added:

VI. Preliminary review of website

VII. Summary comments

The objectives of the review and assessment were to:

• Identify the basic evaluation methodologies, statistics, and performance

measures currently in use at each of the agencies

• Provide a composite view of the three agencies that would allow each to

compare its practices and techniques with others

• Consider which activities might constitute “best practices” for federal website

performance measures

• Establish a first step toward developing statistics and measures for federal

website performance measures.

DTIC. The DTIC website acts as a portal for the public to other sites designed for fuller

public use. Further, it provides fuller service to users who are technologically

sophisticated and familiar with the agency’s structure, context, and background. In the

past, DTIC has relied on technical self-evaluation sources such as log file analysis,

bandwidth usage and tracked web site response time. But recent efforts have also

included user based evaluation techniques such as the Secure STINET customer

satisfaction survey. DTIC is also involved in a larger CENDI Agency working group

effort to incorporate greater use of web evaluation tools.

EIA. The task faced by EIA in putting its information on the web is an organizational

conundrum. The EIA web committee has the unenviable assignment of posting highly

diverse technical information from many sources on one website suitable for use by the

layperson. To this end, and to their credit, the EIA web committee has recognized the

need for continual evaluation and redesign of the site in reaction to a constantly changing

environment. They have conducted studies to aid in the creation of the next version of

their home page to be unveiled in the near future. The new design takes a channel

approach, strives for less jargon, and is designed with the layperson in mind. To prepare

for this new design, EIA chose to perform cognitive usability testing. The results from

this usability testing suggest that EIA’s current web site design meets key usability

requirements as most users could find the required information.
McClure, Sprehe & Eschenfelder E-5 Federal Website Performance Measures

GPO. GPO established the GPO Access website to fulfill the mandate required by PL

103-40. Under the provisions of this law it has subsequently evolved to provide public

access to many key government documents. As of November 1999, the website provided

access to more than 104,000 titles on Government Printing Office servers and over

62,000 additional titles through links to other federal agency websites. GPO considers

Permanent Public Access to be an important performance measure for federal agency

websites. GPO Access maintains historical archives of information previously available

on its servers, ensuring permanent public access to its collection of electronic government

information products, and facilitating the same historical research that is possible through

printed media.

As a content oriented web site, GPO Access primarily measures the number of

documents retrieved by users as a selection of the units of content being distributed by

the site. In addition, staff constantly evaluate other aspects of the website through focus

groups, online user surveys, user comments, log analysis, trade shows and conferences.

Further, GPO conducts user-training sessions. GPO has ensured that the information on

its website is listed in commercial search engines. GPO also has conducted bandwidth

usage studies to determine peak user times and peak usage pages. GPO has a well

developed management approach for its website and has committed significant effort and

thought to evaluating it.

The evaluation and measurement efforts at DTIC, EIA, and GPO demonstrate the wide

range of strategies for measurement. The efforts also suggest that different strategies can

be useful in different agency settings. For example, some agencies may be better able to

employ regular and automatic log analysis techniques than to implement full-scale

usability studies. Others may find that the establishment of formal usability labs to “test”

the effectiveness of selected pages and content may be more effective. Ultimately, the

“best” strategies may be those that best accommodate agency mission and available

resources.

The review of agency practices described here may also be useful for other federal

agencies to compare themselves against. It is not that one agency is better or worse than

another; rather, that these agencies have taken different approaches to developing

evaluation and measurement techniques. A range of issues and situational factors affect

the success with which any agency can engage in an ongoing program of evaluation and

measurement.

Each of these three agencies is in the process of developing and evolving their assessment

techniques. Within this process each has struggled with a comprehensive planning

approach to formally establish responsibilities, tasking, schedules, statistics, and

measures to be employed to assess the quality of agency websites and their usefulness to

users of that website.
McClure, Sprehe & Eschenfelder E-6 Federal Website Performance Measures

4.0 PROPOSED PERFORMANCE MEASURES

The report describes and elaborates on the findings from the assessment of the materials

collected from the three sponsoring agencies concerning the types of statistics and

performance measures the agencies were using. Appendix A, Criteria for the Evaluation

of Federal Websites, and Appendix B, Website Performance Measurement and

Evaluation for GPO Access, supplement the information about performance measures.

As the investigators sifted through the data collected from the agencies and from their

own analyses, they realized that the term “performance measure” can be understood in

several ways. Many things lumped under the general heading of performance measures

might be considered conditions prerequisite to performance measures. In other cases, the

materials pertained to management or operational considerations rather than

measurement of the website’s performance. Performance measures necessarily imply

performance goals and the measures yield indicators of progress toward achieving the

goals.

The study divided performance measures into three types:

1. Legal and policy conditions affecting agency websites

2. Management and infrastructure factors

3. Performance measures in the stricter sense of the term

These three types of performance measures are summarized below in Tables 4.1, 4.2, and

4.3.
McClure, Sprehe & Eschenfelder E-7 Federal Website Performance Measures

Table 4.1. Checklist of Legal and Policy Conditions

Pertaining to Federal Agency Websites

Statute/Policy Checklist Question

A. Does the website contain a privacy notice that complies with the OMB guidance

and model language for federal websites?

B. Does the website avoid the use of “cookies” or observe OMB-stipulated

restrictions?

1. Privacy

C. Does the website comply with the Children’s Online Privacy Protection Act,

particularly with regard to collecting personal information from children? (Same as

9-B below)

A. Conventional FOIA: Does the website contain clear procedures for requesting

agency records under FOIA?

2. Freedom of

Information Act

B. Electronic FOIA: Does the website contain an Electronic FOIA Reading Room?

A. Does the website management include provisions to ensure that copyrighted

materials are not posted without permission from copyright holders?

3. Copyright

B. Reuse Restrictions: Can the site content be freely reused without restriction?

4. Accessibility Does the website make provision for accessibility for persons with disabilities in

accordance with §508 of Rehabilitation Act?

5. Security Does the website management include adequate provisions for protecting the

security of the website and other agency information systems?

A. Do information collections undertaken via the website have appropriate OMB

clearances?

6. Paperwork

Reduction Act

B. Does the website comply with provisions for the Government Information

Locator Service?

7. Depository

Library Program

Have publications posted to the website been made available to the Federal

Depository Library Program?

8. Government A. Does the website permit and encourage electronic information collection?

Paperwork

Elimination Act

B. Does the website permit use of digital signatures?

9. Federal Records

Act

Does the website management include adequate provision for identifying website

records and transferring records to agency record keeping systems?

A. Does the website comply with the President’s April 1997 guidance on

expanding Internet access for children, parents, and teachers?

10. Access for

Children

B. Does the website comply with the Children’s Online Privacy Protection Act,

particularly with regard to collecting personal information from children? (Same as

1-C above)
McClure, Sprehe & Eschenfelder E-8 Federal Website Performance Measures

Table 4.2. Management and Infrastructure Factors

Pertaining to Federal Agency Websites

Type of Factor Factor

A. Does the website have sufficient technology infrastructure (bandwidth,

server controller array, etc.) to ensure adequate response time?

B. Does the website have the appropriate software to achieve its purposes?

C. Does the website have the appropriate level of availability (e.g., 24 hours, 7

days a week)?

D. Does the website have adequate steps for emergency preparedness and

disaster recovery?

E. Does the website have adequate operating resources (funding and personnel)

to achieve its purposes?

1. Infrastructure

F. Are server errors and 404 errors (pages not found errors) at an acceptably

minimum level?

A. Does the website comport with industry best practices, such as the World

Wide Web Federal Consortium’s Home Page Guidelines, as regards content

design, navigation, organization, style, and markup.

B. Access control: Does the agency exert management control over who may

post to the website?

C. Quality control: Does the agency exert management control over the quality,

design, and style of website postings?

D. Awareness and Visibility: Has the agency ensured the site is easily reachable

by registering with search engines and portals and periodically searching for its

sites on these engines?

E. User Support: Has the agency provided an adequate user support staff for

technical and content queries?

F. Security: Is the agency providing adequate security to prevent intrusion into

the website?

G. Evaluation: Does the agency regularly evaluate the website through log file

analysis, focus groups, online user surveys, open public forums, and other

appropriate methodologies?

2. Management

H. Records Management: Has the agency made adequate provision for records

management on the website and integrated the website into the agency records

management program?
McClure, Sprehe & Eschenfelder E-9 Federal Website Performance Measures

Table 4.3. Performance Measures for Federal Agency Websites

Performance Goal Basic Measures Other Agency Specific Measures

Extensiveness:

Amount or extent

to which services

are used

-Information on content

unit retrievals; e.g., no. of

document downloads

-No. of user sessions per

time period (not no. of hits)

-No of User contact sessions

-Activity levels by time periods

-Ratio of unique to repeat (2 or more) user sessions

per time period.

Efficiency: Use of

resources in

providing services

-Cost of providing website

session per user

-Percent of operational

time when website is not

available

-No. of FTE hours or days devoted to website

creation/ maintenance by size of site in pages

-Cost per user help session

-Relation to diminishing costs of other publications

media (e.g., printing) as indicator that website may

be replacing other media.

Effectiveness: How

well the website

meets the general

governmental

objectives and

specific agency

objectives

-Completeness of coverage

of agency publications,

press releases, etc.

-Degree to which website

is increasing the timeliness

of access to agency pubs

-Permanent public access to agency publications

-Degree to which GILS is integrated into website

design/operations

-Degree to which website shows agency reaching

new constituent audiences

Service Quality:

How well the

website functions

-User success rate in

finding specific

information in a given time

period

-Average time between

user contact request and

agency response

-No. of customer

complaints/suggestions and

whether agency action

results.

-Whether agency has Help Desk dedicated to its

website

-24/7 availability measures

-Minimal 404 errors

-Courtesy, helpfulness of user support staff

-Increase in no. of repeat users per time period

Usefulness: How

well the website

meets the needs of

users

-Customer comments plus

surveys and focus groups

-Degree to which website

information increases user

productivity

-Degree to which website

information is incorporated

into other tasks inside and

outside the agency

-Cognitive and Usability Evaluation

-Measured user satisfaction with:

 *Clarity of homepage; organization of site

 *Timeliness of website information

 *Links to other useful information

-No. of referrals from other websites and sources of

referrals

In conclusion, the investigators offer the following recommendations:

1. Federal agencies should address the questions in Table 4.1 regarding agency

websites. These are basic questions about whether the practices of the agencies’

sites comply with federal law and policy.

2. Agencies should consider the questions in Table 4.2 as a useful but

nonexhaustive checklist of infrastructure and management questions they should

pose with respect to their websites.
McClure, Sprehe & Eschenfelder E-10 Federal Website Performance Measures

3. Agencies should ask themselves whether their website performance measures

programs address the six performance goals listed in Table 4.3 and their

associated basic measures. The investigators do not necessarily recommend that

websites use the exact measures listed in the table, but they do recommend that

all agencies employ measures at least similar to these.

5.0 ISSUES, RECOMMENDATIONS, AND NEXT STEPS

The issues discussed in this study are not intended to be comprehensive; rather, they suggest

important concerns that will need to be reviewed, discussed, and resolved at individual

agencies. The solution to these issues is likely to vary depending on a range of situational

factors in each agency.

Key issues agencies must address and resolve concerning websites include the following

• Top Level Administrative Support for Website Assessment

• Ongoing Program of Website Evaluation

• Organizing for Website Assessment

• Funding Website Assessment

• Agreement on Approach and Measures

• Training

• Recognizing the Policy Context

• Sharing Knowledge

• A Changing Web Evaluation Context

Federal agency website evaluation and development of performance measures is in its

infancy and much work lies ahead in designing, testing, and implementing evaluation

methods and measures. The investigators recommend two broad areas for next steps that

can build upon the work described in this report.

• Undertake a follow-on study to explore in greater depth some of the issues

surfaced in this research. Appendix C provides an example of a proposal for such

a study.

• Disseminate findings widely and increase awareness. This study is limited in scope

and yet its findings could prove useful to many other agencies.

Evaluation and performance measures to describe federal websites and resources in the

networked environment are important tools. But an ongoing program of evaluation that

regularly produces these performance measures is a tool that is essential for the long-term

success of the agency. Perhaps most importantly, an ongoing program of evaluation

contributes to the process of constant improvement – looking for ways to improve the

usefulness, impact, and benefits that can result from web-based resources and services. At

issue is the degree to which these web-based resources and services are cost effective, deliver

high-quality services, meet the needs of users, comply with existing policy, reduce agency

costs, and help accomplish agency mission and objectives – to name but a few criteria.
McClure, Sprehe & Eschenfelder 1-1 Federal Website Performance Measures

CHAPTER 1

INTRODUCTION

1.0 BACKGROUND

Federal agency web sites increasingly act as key information and service resources for

the agencies themselves, citizens, public interest groups, government employees,

contractors and other stakeholders. Agency web managers have seen web page use go

“up up up” as more people access more federal agency web pages more often. Five years

ago, in the nascent stages of federal web site development, web mangers could evaluate

their web system based on perceived usage and expansion. As long as usage was

increasing, and agency personnel were putting more information and services on line,

“things were good.”

Today’s environment, however, demands more sophisticated measures for web site

evaluation. IT resource demands are increasing, federal performance review guidelines

demand increased efficiency, users grow more sophisticated and demand more

information and more services. This environment of growing demands and constricting

resources requires evaluation and performance measures that allow web managers to do

more than show that usage is going “up up up” and that the overall size of the web site is

increasing. Today’s federal agencies need evaluation and performance measures to aid in

more complex tasks such as:

• Increasing site usability, content and service quality;

• Creating strategic plans and prioritizing projects;

• More efficiently and effectively allocating resources;

• Improving information and service quality and;

• Enhancing behind the scenes management processes.

In addition, a variety of federal laws, regulations, and guidelines prescribe how websites

are to be managed and evaluated (see Chapter 2). Foremost among these is the

Government Performance and Results Act of 1993 (GPRA), which mandates the

adoption of a strategic and annual planning process tied to budget and authorization

cycles and based on established and measurable performance indicators for every

program.

Websites are now a major vehicle through which federal agencies deliver information to

the public. While agencies have made some progress in establishing finding aids such as

the Government Information Locator Service (GILS) to assist in locating information, as

yet no systematic effort has been devoted to measuring the overall performance of

websites in successfully achieving adequate levels of public access to government
McClure, Sprehe & Eschenfelder 1-2 Federal Website Performance Measures

information.

Some agencies maintain a range of statistics describing web services while others have

undertaken only minimal or no data collection and analysis effort; some have devoted

substantial resources to “one-stop shopping” for information; many have developed

“frequently asked questions” to assist visitors to agency websites. Most agencies already

use weblog statistics and other software-based measures to examine aspects of their

websites' performance. But agencies still need a flexible approach that goes beyond web

statistics such as transaction logs to offer a variety of techniques by which agencies can

determine whether their websites are successfully achieving the information

dissemination missions for which they are intended.

1.1 STUDY PURPOSE AND REPORT STRUCTURE

The overall purposes of the project are to develop performance measures that will assist

agencies to assess the quality and usefulness of their websites and to improve public

access services of those websites. More specifically, the project had the following

objectives:

• Examine the professional literature, as well as government agency and

commercial experience, to identify and make use of measures already developed

by others.

• Identify those aspects of agency websites that are most important for effective

delivery of public information.

• Develop a set of objective performance measures for determining the degree to

which an agency website effectively provides public access to agency information

resources.

• Test and refine these measures in selected agencies.

• Propose a concise set of procedures that describes how these performance

measures can be further developed and applied by agencies.

Limited time and resources did not allow the investigators to test and refine the measures

as much as might have been desired. Such efforts can be continued after this study.

Ultimately, the investigators hope that both public user groups and agency officials will

be able to determine the degree to which a given agency website meets these performance

measures.

The study provides an overview of evaluation techniques and performance measures used

in three federal government agencies (see Chapter 3). Based on these summaries,

selected sources from the literature, and the investigators’ experience, the report then

suggests key federal website performance measures that can be developed and used in

federal agencies at this time (see Chapter 4).

The report is structured as follows. This first chapter provides an introduction to key

concepts and literature related to evaluation and performance assessment. The second
McClure, Sprehe & Eschenfelder 1-3 Federal Website Performance Measures

chapter reviews federal policy impacting federal agency web site evaluation. The third

chapter summarizes performance measures used at three federal agencies. The fourth

chapter suggests federal agency performance measures for the future. The report

concludes with a chapter describing key issues yet to be resolved and offers some

possible next steps in developing performance measures for federal websites.

1.2 OVERVIEW OF STUDY METHOD

The study employed a multi-method approach to data collection and analysis. Given the

relatively short time line for the study and the limited resources available the following

approaches were employed:

• Literature Review. The study reviewed selected literature and reports from both

inside and outside the government. While there is some useful background

information (see below) there is limited knowledge and information about

developing and computing a coherent set of performance measures for

government websites.

• Group Interviews with Participating Agency Representatives. The

investigators met (1) individually with representatives of each participating

agency, (2) the participants as a group, and (3) conducted a number of individual

follow-up discussions with representatives of the three agencies about activities

and procedures for assessment in their agency.

• Best Practices. Each of the three participating agencies provided the

investigators with (1) descriptions of their assessment techniques, and (2)

documentation and reports related to these assessment techniques; these materials

were reviewed and analyzed by the investigators.

• Policy Analysis. The investigators reviewed existing federal policies, laws, and

guidelines to produce an overview of policies affecting the development and

management of federal websites; this compendium identifies a number of

ambiguities and issues that will require additional assessment.

• Survey. The investigators conducted a survey at the FedWeb 2000 conference

held in March, 2000 at Bethesda, MD. This survey provided a snapshot of current

practices and key issues of concern to federal officials regarding the assessment of

federal websites.

• Participant Review. Officials from the three participating agencies had a

number of opportunities to review preliminary findings and reports developed by

the investigators.

The combination of these approaches provided the investigators with a unique

perspective on current practices, key issues, and possible assessment techniques and

performance measures. The approach demonstrated the importance of recognizing the

existing policy environment that affects agency website development and management

(see Chapter 4). The approach also resulted in a better understanding of the problems and

issues that agencies confront in attempting to conduct ongoing website evaluation and

assessment (see Chapter 5).
McClure, Sprehe & Eschenfelder 1-4 Federal Website Performance Measures

1.3 WHAT IS EVALUATION?

Website evaluation can be defined as the use of research or investigative procedures to

systematically investigate the effectiveness of a web based information system on an

ongoing basis. Evaluation is used in conjunction with a number of activities and has a

number of impacts on an organization.

1.3.1. Planning, Goal Setting and Determining the Degree to Which the IS

Organization is Meeting Set Goals

Evaluation plays a key role in organizational planning and goal

setting. This is known as “formative” evaluation. In contrast,

“summative” evaluation determines the degree to which the

organization is meeting set goals. Figure 1 illustrates this dual

role. On the left side of the diagram, information discovered as

part of the evaluation process feeds back into goal setting and

planning. Ongoing evaluation is a vital source of information for

agencies’ planning processes. For example, an evaluation of

current website user satisfaction may reveal usability issues with

the current page design or information architecture. Planners may

choose to change or modify goals based upon newly discovered

problems or the achievement of previously set goals.

Figure 1.1: Formative and Summative Evaluation

On the right side of the diagram, evaluation is used to determine the degree to which the

organization has met stated goals. Developing goals and objectives with no follow-up

effort to determine how well those objectives were actually accomplished significantly

reduces the overall value of both planning and the use of assessment techniques. Based

on the previous evaluation, if the organization had created a goal to improve site

usability, they would then use evaluation to determine the degree to which the site’s

usability had improved.

Goals or

Desired

State

Actual state as

revealed by

evaluation

EvaluationSummative

FormativeEvaluation
McClure, Sprehe & Eschenfelder 1-5 Federal Website Performance Measures

Both formative and summative evaluation efforts are important – although most

organizations tend to concentrate on summative approaches. But for monitoring and

ongoing improvement of services, formative evaluation (intended to improve, not prove)

is essential.

1.3.2. Decision-making and Resource Allocation

Often managers find themselves in problematic situations with multiple possible

solutions. It is difficult to know, up front, which solution will best meet overall goals.

Consider the situation where a web manager’s overall goal is to increase the user’s ability

to find product and service information. Designers might suggest a number of possible

web page designs. Which design will best increase usability? Evaluation of the possible

page designs will provide data to help answer the question.

Further, in an environment of restricted resources, organizations often cannot provide

resources to all programs or projects that request funding. Decision makers must choose

projects or programs to fund. Evaluation can provide data to assist in the decision

making process and justify requests for new resources.

1.3.3. Determine the Effectiveness of a Previous Decision

Similarly, organizations can use evaluation to determine the effectiveness of a previous

decision. Did the funded program or project live up to its promises? To what extent did

the funded program or project meet its stated goals? To what extent did the decision

successfully contribute to the organization’s goals? When agencies fail to ask these types

of questions they run the risk of maintaining programs and services that do not work well

and are not meeting user needs.

1.3.4. Determine the Degree to which the System/Service Adds Value to the

Organization

Evaluation can answer specific organizational questions about the degree to which a

system adds value to an organization. For instance, evaluation could show how a system

reduced overall costs or to determine if printing costs decline as on-line publication of

information increases.

Agencies are only beginning to explore how web based services add value or enhance

traditional services. But, agencies now see that being able to provide user services on a 7

days a week, 24 hours a day basis is a significant change in service provision. This

“value add,” however, may result in other impacts such as an increased need for real time

user support.

1.3.5. Provide Trend Data to Assess Change Over Time
McClure, Sprehe & Eschenfelder 1-6 Federal Website Performance Measures

Collection and comparison of assessment results over time can show trend data for

federal web sites. Trend results may show changes more effectively than individual

assessment results if the results change gradually over time and imply a continued trend

in a certain direction. For instance, if results were being used to justify more budget

dollars, trend data showing reduced costs over time would both better show the impact of

the service and more effectively convince decision makers that costs will continue to fall.

Trend data is especially important for formative evaluation efforts as it provides a basis

by which changes in the use of a services can be identified early on. In log analysis

techniques, a change in accessing the website via a second level page rather than the

homepage has significant implications for content development.

1.3.6. Contribute to Continuous Improvement Efforts and Benchmarking

Evaluation can serve as a first step in benchmarking. In benchmarking, organizations

identify best-practice performance. They then use that performance as a goal, investigate

the factors that led to the performance, and then try to replicate that level of performance.

Evaluation provides a base level reading of organizational performance on benchmark

measures and can also show longitudinal progress toward the goal.

1.3.7. Identify Problems and Possible Solutions

One impact of evaluation is the identification of previously undetected problems.

Evaluation can also indicate the extent or scope of a problem. Once such problems are

identified and described, a plan to address the problem can be developed. Identification

of unanticipated problems is a powerful tool that takes on increased important in a web

environment since oftentimes technical assessment (as opposed to user assessment) may

be the only type of assessment being conducted.

1.3.8. Empower Organizational Actors to Seek and Enact Solutions

Another beneficial side effect of evaluation is that it can lead to increased employee

involvement and action. As part of the evaluation process, employees identify problems

and potential solutions. Employees then can act on the problems, develop solutions and

improve web-based services directly and immediately. Seeing changes and

improvements to services can improve morale and increase confidence to tackle further

problems. Further, changes to services may improve work processes and reduce

inefficiency, thereby reducing work stress.

1.3.9. Develop Accountability

Evaluation may also lead to increased accountability as employees within the

organization identify and take responsibility for problems and solutions. Further, the goal

setting process in formative evaluation has the positive impact of increasing employees’

understanding of organizational goals and expectations. This better understanding may

lead to increased accountability in meeting the goals and expectations.
McClure, Sprehe & Eschenfelder 1-7 Federal Website Performance Measures

1.3.10. Organizational Learning

Evaluation is also an opportunity for organizations to learn about themselves and the

diversity of needs, goals and perceptions within the organization. First, evaluation tells

the organization about its own strengths and weaknesses. Second, it provides a snapshot

of current organizational infrastructure and resources. Third, evaluation can illuminate

differences between groups within the organization; how the groups may have different

goals, different interpretations of problems, and differing needs. Finally, the evaluation

process encourages these value judgments regarding appropriate levels or quality of

services to be made explicit.

These value judgments may provoke debate that educates members about the potential

impact of particular measurements on the organization, the costs and benefits of choosing

a particular goal, and the variety of needs and goals of different groups within the

organization.

1.4. PERFORMANCE MEASURES

Performance measures are the tools that enable evaluation. As a doctor needs to monitor

blood pressure and heartbeat to perform a check-up, evaluators need to look at

performance measures to conduct an evaluation. They are an essential means to assess

agency services and accomplishment of mission.

Performance measures typically provide measurements of the following:

• Inputs and Efficiency. The use of resources in providing or accessing web

services (e.g. staff hours required for site maintenance, technical training or

consulting services required for systems integration).

• Effectiveness. How well the networked information service met the objectives of

the provider, user or customer (e.g. success rate of identifying and accessing the

information needed by the user).

• Outputs. Indicators of the products or services resulting from the use of those

resources (e.g. number of agency documents available online, usage levels for a

given e-service).

• Extensiveness. How much of a service the network provides (e.g. number of

visits to a web site per week, how many document downloads per week.)

• Service Quality. How well a service or activity is done (e.g., percentage of

transactions which result in an order or a document download).

• Impact. How a service made a difference in some other activity or situation (e.g.

the degree to which network users saved time, resolved a decision or identified

new and innovative applications/services).

• Usefulness. The degree to which the services are useful or appropriate for

individual users (e.g. percentage of services of interest to different categories of

users).

• Adoption. The extent to which institutions or users integrate and adopt web

resources in organizational or individual activities (e.g. problem solving, using
McClure, Sprehe & Eschenfelder 1-8 Federal Website Performance Measures

online forms instead of print, etc.).

These categories can become blurred and are not necessarily mutually exclusive. In

addition, evaluation and performance measures reflect value judgments on the part of

evaluators regarding the adequacy, appropriateness, and success of a particular service or

activity. Use of performance measures can lead to beneficial organizational changes.

For example, a tendency of users stick to use of printed forms instead of online forms can

be an indicator that the organization needs some targeted business process reengineering.

1.5. OVERVIEW OF WEB EVALUATION

This section sets the context for federal web site evaluation, explaining its relation to

broader efforts to evaluate web information systems (web IS) and information systems

(IS) in general. One can see federal agency web system evaluation as a special subset of

web IS evaluation that in turn is a subset of general IS evaluation. This section first

provides a brief overview of IS evaluation in general. It then provides a select overview

of modern web IS evaluation research. It finishes with an overview of recent federal web

evaluation efforts.

1.5.1. IS Evaluation

IS evaluation has become an increasingly important topic within the competitive U.S.

business environment. Several factors have contributed to evaluation’s growing

importance. First, IS projects historically have had low success rates; some researchers

have suggested they are as low as 30-40 percent (Willcocks and Lester, 1993). Tom

Peters believes that 85 percent of companies trying to restructure around e-commerce will

“blow it.” He goes on to suggest that the move to e-commerce has more to do with

relationships and organization than with IT. (Infoworld, September 25, 2000, p. 200).

This increases pressure on managers to both justify their projects and show how their

projects can and will succeed. Second, vendors inundate mangers with a maelstrom of

hype surrounding new products and IT trends.

Managers need evaluation tools to help them determine the actual usefulness of these

products and trends for their organizations. Third, while organizations’ budgets have

generally increased allocations for IT, downsizing and streamlining demands require IT

managers to show how increased IT spending is adding value to the organization. For

examples of and resources about general IS evaluation guides see:

• The National Research Council report More Than Screen Deep: Toward EveryCitizen Interfaces to the Nation's Information Infrastructure,

http://www.nap.edu/readingroom/books/screen

• Assessing the Academic Networked Environment: Strategies and Options by

Charles McClure and Cynthia Lopata, available through the Coalitation on

Networked Information, http://www.cni.org

• “Managing Information Systems: A Practical Self Assessment Tool” (Preview
McClure, Sprehe & Eschenfelder 1-9 Federal Website Performance Measures

version) by the Information Technology Review Board, http://www.itrib.gov

• Performance-Based Management: Eight Steps to Developing and Using IT

Measures Effectively” by the GSA Office of Government wide Policy,

http://www.itpolicy.gsa.gov/mkm/pathways/evt8step.html

• Performance Pathways” part of the IT Ramp website of the GSA Office of

Information Technology http://itpolicy.gsa.gov

Readers should take note that the many resources developed for general IS evaluation can

be adapted for use in web site evaluation.

1.5.2. Web IS Evaluation

The last five years has seen a surge of interest in website evaluation. One result has been

the publication of plethora of web “do-it-yourself” books that include advice on both

design and evaluation (for example see Nielsen, 2000; Jacobson, 1999). At the same

time researchers from the business, education and library science fields have sought to

evaluate web sites based on the many criteria including:

• Interface design (Kopak and Cherry, 1998; Vanhouse, Butler, Ogle, Schiff, 1996)

• Usability (Benbunan-Fich, 1999)

• Comparison to peer organizations - benchmarking (Johnson and Misic, 1999)

• Fit with theoretical models (eg. Marketing model: von Dran; Zhang and Small,

1999; motivational model: Zhang and von Dran, 2000)

• Web site strategy (Simeon, 1999; Auger 1997)

• Information quality (McMurdo, 1998)

• Hypertext Structure (Bauer and Scharl, 2000)

Web site evaluation has also become a popular topic within the trade press (e.g. Dugan,

2000). A significant amount of web evaluation effort has been put into log analysis

techniques (Bertot, McClure and Rubin, 1997) and use of specific log analysis software

such as WebTrends (http://webtrends.com). Additional information on the use of log

analysis tools and software can be found at Yahoo! Under computers and the Internet:

Log Analysis tools at

http://www.yahoo.com/Computers_and_Internet/Software/Internet/World_Wide_Web/Se

rvers/Log-Analysis_Tools.

Once again, readers should note that information on general website evaluation is

applicable to the federal web environment with certain key allowances made for design

restraints imposed by regulation or statute.

1.5.3. Federal Website Evaluation

Federal website evaluation has been ongoing since the inception of federal websites. One

early landmark was the World Wide Web Federal Consortium1 publication of suggested

 

1

 The original guidelines is still available at http://www.dtic.mil/staff/cthomps/guidelines (last visited

August 2000).
McClure, Sprehe & Eschenfelder 1-10 Federal Website Performance Measures

guidelines for federal website development (draft 1996). The guidelines have been

periodically updated in recent years.2 Many federal agencies conduct periodic

evaluations to maintain and enhance the quality of their sites. See, for example, “Love

That Site, Love That Agency” (Government Computer News, 1999) for a description of

an evaluation of the Defenselink website. Recently published guidelines include the

Federal Webmaster Forum Website Privacy Guidelines.3

There has also been a substantial and increasingly sophisticated academic evaluation

research stream. Current web evaluation research has looked at federal websites in terms

of a variety of evaluation criteria including information content and ease of use (e.g.

Eschenfelder et al., 1997; McClure and Wyman, 1997; Hert and Marchionini, 1997) and

compliance with federal records guidelines (McClure and Sprehe, 1999).

Further, some studies have looked at specific aspects of websites. For instance, Hert

(1998) evaluated web site finding aids and Moen and McClure (1997) examined the

government information locator service (GILS). Other evaluation efforts have taken a

more holistic approach. For instance, Hert, Eschenfelder and McClure (2000) included a

usability, management, technical and policy analysis. Finally, these studies have been

varied in methodologies, with some relying on mainly one method (e.g. log file analysis

Redalen & Miller (2000) and Bertot et al. (1997)) while others have taken a multi-method

approach (e.g. Hert et al., 2000).

1.6. IMPORTANCE OF WEBSITE EVALUATION

In summary, web site evaluation is an important part of the ongoing management and

enhancement of federal agency websites. Evaluation, whether as part of a formal

planning process or as a stand-alone activity, can provide managers with key information

that can aid decision making. Although web sites are a relatively new medium, IS

evaluation has a long history, providing tools and lessons learned that federal web

managers can apply to their web evaluation efforts. Current academic research on federal

agency web evaluation also provides strategies and tools that agencies can adapt for their

efforts.

Interest in evaluation of web based services continues to increase, although, few

comprehensive approaches assess federal websites on an ongoing basis. As agencies

continue to be pressed to provide additional web based services with limited resources

and as implementation of the Government Performance and Results Act presses forward,

ongoing evaluation and use of performance measures is likely to take on increased

importance.

 

2

 The current version of the guidelines, dated July 1999, can be found at

http://diggov.org/library/current/general/www.ojp.usdog.gov^oa^fedwebguide^execsum.htm. (Visited

August 2000)

3

 See the Federal Webmaster Forum homepage at http://itpolicy.gsa.gov/mke/fedwebm/fedwebm.html for

current Forum website privacy guidelines.
McClure, Sprehe & Eschenfelder 1-11 Federal Website Performance Measures

References

Auger, P. (1997). Marketing on the World Wide Web: An Empirical Investigation of the

Relationship Between Strategy and the Performance of Corporate Web Sites

[Diss] (01A, Vol. 59, p. 0241), Syracuse, NY: Syracuse University.

Bauer, C.; Scharl, A. “Quantitive evaluation of Web site content and structure” Internet

Research ; 10:1 2000; pp. 31-44.

Benbunan-Fich, R. (1999, August). Methods for Evaluating the Usability of Web Based

Systems. AMCIS Americas Conference on Information Systems. Milwaukee, WI.

Bertot, J.C., McClure, C.R., and Rubin, J. Web Usage Statistics: Measurement Issues

and Analytical Techniques. Government Information Quarterly; 14:4 1997; pp.

373-395.

Computer Science and Telecommunications Board of the National Research Council

More Than Screen Deep: Toward Every-Citizen Interfaces to the Nation's

Information Infrastructure, http://www.nap.edu/readingroom/books/screen.

Washington, DC. Also partially available as “Design and Evaluation: A Review

of the State-of-the-Art” D-Lib Magazine, July-August 1998.

Dugan, Sean “Learn, know, and live them: The seven metrics of highly successful Web

sites” Infoworld.com, January 23, 2000.

Eschenfelder, K.R.; Beachboard, J.C.; McClure, C.R.; Wyman, S.K. “Assessing U.S.

Federal Government Websites” Government Information Quarterly,14(2), pp.

173-189, 1997.

Government Computer News “Love That Site, Love That Agency” Supplement to the

Technology in Excellence Seminar of June, 1999. http://www.gcn.com

Harter, S.P.; Hert, C.A. “Evaluation of information retrieval systems. Approaches,

issues, and methods” In Annual Review of Information Science and Technology

(ARIST) Volume 32, 1997 p. 3-94 1998 Information Today Medford, NJ

Hert, C.A.; Eschenfelder, K.R., McClure, C.R. Evaluation Of Selected Websites At The

U.S. Department Of Education: Increasing Access To Web-Based Resources.

Information Institute of Syracuse, 2000.

Hert, C.A. Facilitating Statstical Information Seeking on Websites: Intermediaries,

Organizational Tools and Other Approaches. Final Report to the Bureau of Labor

Statistics, 1998. Available at http://istweb.syr.edu/~hert/BLSphase2.html

Hert, C.A. and Marchionini, G. Seeking Statistical Information on Federal Websites:

Users, Tasks, Strategies, and Design Recommendations. Final Report to the

Bureau of Labor Statistics. Available at

http://ils.unc.edu/~march/blsreport/mainbls.hmtl

Jacobson, R. Information Design MIT Press: Cambridge, MA: 1999

Johnson, K.L.; Misic, M.M. “Benchmarking: a tool for Web site evaluation and

improvement” Internet Research ; 09:5 1999; pp. 383-392.

Kopak, R.W. Cherry, J.M. “Bibliographic displays and web catalogues: user

evaluations of three prototype displays” Electronic Library, 16(5)p. 309-323,

1998.

Nielsen, J Designing Web Usability:The Practice of Simplicity New Riders Publishing:

2000.

McClure, C.R. and Cynthia Lopata, Assessing the Academic Networked Environment:
McClure, Sprehe & Eschenfelder 1-12 Federal Website Performance Measures

Strategies and Options by Charles available through the Coalitation on

Networked Information: Washington DC) 1996.

McClure, C.R.; Sprehe, T. Analysis and Development of Model Quality Guidelines for

Electronic Records Management on State and Federal Websites: Final Report

The National Historical Publications and Records Commission: Washington DC,

1999.

McClure, C.R.; Wyman, S. Quality Criteria For Evaluating Information Resources And

Services Available From Federal Websites Based On User Feedback Online

Computer Library Center: Dublin, OH, 1997.

McMurdo, G. “Evaluating Web information and design” Journal of Information Science,

24(3), p. 192-204, May 1998

Moen, W.E. and McClure, C.R. An Evaluation of the Federal Government’s

Implementation of the Government Information Locator Service (GILS): Final

Report. 1997. Available at: http://www.unt.edu/slis/research/gilseval/titpag.htm

Redalen, A.; Miller, N. “Evaluating Website Modifications at the National Library of

Medicine through Search Log Analysis” D-Lib Magazine, 6(1), January 2000.

Tague-Sutcliffe, J.M. “Some perspectives on the evaluation of information retrieval

systems” Journal of the American Society for Information Science, 47(1), January

1996, Pages: 1-3.

Van House, N.A.; Butler, M.H.; Ogle, V.; Schiff, L. “User-Centered Iterative Design for

Digital Libraries: The Cypress Experience” D-Lib Magazine, February 1996

Von Dran, G., Zhang, P., & Small, R. “Quality Websites: An Application of the Kano

Model to Website Design” AMCIS Americas Conference on Information

Systems. Milwaukee, WI, 1999.

Willcocks, L.; Margetts, H. “Risk Assessment and Information Systems” European

Journal of Information Systems 3(2), 127-138, 1994.

Zhang, P.; von Dran, G. “Understanding Website Users: A Two-Factor Model for

Website Design and Evaluation” Journal of the American Society for Information

Science (forthcoming).
McClure, Sprehe & Eschenfelder 2-1 Federal Website Performance Measures

CHAPTER 2

FEDERAL POLICIES AFFECTING WEBSITE DEVELOPMENT

AND MANAGEMENT

2.0 INTRODUCTION

Federal information policies establish the legal and procedural framework in which

government information and services are made available to the public. "An information

policy instrument is a written law, guideline, regulation, or other official statement that

describes how information will be collected, managed, protected, accessed, disseminated,

and used" (Kent 1999, 307). The purpose of this chapter is to provide an introductory

review of selected U.S. Federal information policy instruments that affect the

development and management of Federal websites. While specific references to the

Internet or the World Wide Web may not appear in a given policy instrument, this does

not necessarily mean the policy has no application to Federal websites. These

instruments are directed toward Federal government procedures and operations in the

context of Federal information and/or information technology, and thus would apply to

Federal websites unless special rules require otherwise.

This listing is not comprehensive, but rather offers a general sense of the range of

existing Federal policies that should be considered in the development, management, and

evaluation of websites. Where possible, citations to Web-based sources of information

regarding a particular policy are provided, as are selected excerpts of the policy that may

be of interest to Federal Web designers, managers, and evaluators. The selection of

policies could be extended much more broadly. For example, the chapter leaves out the

area of electronic commerce. Federal agencies do use their websites for selling goods

and services, and electronic buying of goods and services is a major thrust in the Federal

procurement community. Readers who wish to explore the area of Federal e-commerce

are referred to http://www.ecommerce.gov.

The policy instruments are organized into a schematic outline, moving from general

government policy to general information policy and then to specific information

policies. Within each heading occurs a listing of statutory policies and implementing

policy guidance. It is within this context that Federal agencies develop and manage their

websites for providing government information and services, which must also meet the

agencies’ mission statements, objectives, and goals.

Issues that Federal agencies need to consider in developing websites include information

security, information privacy, information access, electronic records management, and

intellectual property. Information security instruments concern risks to the ongoing

operation of government computer systems, their integrity, and the protection of

classified or confidential materials they contain. Information privacy instruments seek to

protect personal information that may be collected from agency website users. Internet

access policy instruments are concerned with ensuring the equitable access for U.S.
McClure, Sprehe & Eschenfelder 2-2 Federal Website Performance Measures

citizens to electronic information contained on Federal government websites. Electronic

records management policy instruments concern issues regarding the creation,

maintenance, use, and disposal of Federal records. Intellectual property policy

instruments include a wide variety of ownership rights in intangible products, such as

copyrights, patents, trademarks, and trade secrets.

Federal information policy and agency website development occur in a dynamic

environment. Technological changes impacting on established information policies or

creation of new ones occur rapidly, but Federal agencies often must adjust their

operations almost immediately. Policy tends to follow technology and practice.

Sometimes the lag between policy and practice can be great, so that agencies must craft

their own policies to rationalize practices before Congress enacts new laws.

2.1 GENERAL GOVERNMENT POLICY

2.1.1. Government Performance and Results: Performance Plans and

Measures

2.1.1.1. Statute: Government Performance and Results Act of 1993.

(Public Law 103-62).

Available at: <http://thomas.loc.gov/cgi-bin/bdquery/

z?d103:SN00020:|TOM:bss/d103query.html>.

Application to Federal Information Technology and Web

Management:

The Government Performance and Results Act of 1993 (GPRA) is

intended to help reduce waste and inefficiency in Federal programs and

Federal agency operations. It is also designed to assist Congress and the

Executive in their oversight, legislative, and administrative tasks related to

authorizing, appropriating, and implementing Federal services. GPRA

mandates the adoption of a strategic and annual planning process, which is

tied to budget and authorization cycles and will be based on established

and measurable performance indicators for every program. Although this

act was made law prior to the expansion and wide public use of the

Internet, particularly the Web, the mandate for the development of

performance indicators tied to annual budgeting and strategic planning

applies to services offered in an electronic environment. Performance

indicators for Federal websites consist of measures that permit an agency

to demonstrate whether its websites are or are not meeting the

performance goals set forth for the sites. Performance measures could

include things such as log transaction files and impact measurements.

Excerpt of Original Text from Policy Instrument:

Section 3. Strategic Planning
McClure, Sprehe & Eschenfelder 2-3 Federal Website Performance Measures

Chapter 3 of title 5, United States Code, is amended by adding after

section 305 the following new section:

Sec. 306. Strategic plans

(a) No later than September 30, 1997, the head of each agency shall

submit to the Director of the Office of Management and Budget and to the

Congress a strategic plan for program activities. Such plan shall contain--

* * *

(3) a description of how the goals and objectives are to be achieved,

including a description of the operational processes, skills and technology,

and the human, capital, information, and other resources required to meet

those goals and objectives;

* * *

(6) a description of the program evaluations used in establishing or

revising general goals and objectives, with a schedule for future program

evaluations.

* * *

Section 4. Annual Performance Plans and Reports

* * *

(b) PERFORMANCE PLANS AND REPORTS- Chapter 11 of title 31,

United States Code, is amended by adding after section 1114 the following

new sections:

Sec. 1115. Performance plans

(a) In carrying out the provisions of section 1105(a)(29), the Director of

the Office of Management and Budget shall require each agency to

prepare an annual performance plan covering each program activity set

forth in the budget of such agency. Such plan shall--

(1) establish performance goals to define the level of performance to be

achieved by a program activity;

(2) express such goals in an objective, quantifiable, and measurable form

unless authorized to be in an alternative form under subsection (b);

(3) briefly describe the operational processes, skills and technology, and

the human, capital, information, or other resources required to meet the

performance goals;

(4) establish performance indicators to be used in measuring or assessing

the relevant outputs, service levels, and outcomes of each program

activity;

(5) provide a basis for comparing actual program results with the

established performance goals; and

(6) describe the means to be used to verify and validate measured values.

2.1.1.2. Implementing Guidance: OMB Circular No. A-11, Part

II, Preparation and Submission of Strategic Plans, Annual

Performance Plans, and Annual Program Performance

Reports
McClure, Sprehe & Eschenfelder 2-4 Federal Website Performance Measures

Available at <http://www.whitehouse.gov/omb/circulars/a11/99toc.html>.

Application to Federal Information Technology and Web

Management:

OMB Circular No. A-11, issued annually, instructs federal agencies how

they are to submit their budget requests for two years hence; that is, in

fiscal year (FY) 2000 agencies submit budget requests for FY 2002. Part

II was added to Circular A-11 after the passage of the Government

Performance and Results Act. It sets forth detailed procedural guidance

for formulating the agencies’ annual performance plans and describes

what agencies are to include in their annual program performance report.

Included in Part II is the requirement for program performance goals and

the enumeration of the performance measures agencies will use to assess

their relative success or failure in achieving performance goals. Part II is

the basic Federal policy guidance regarding performance measures in

general, as applied to all Executive Branch programs, and has applicability

to Federal websites.

2.1.2. Customer Service Standards: Setting Customer Service Standards.

(Executive Order 12862). 1993.

Available at: <http://www.pub.whitehouse.gov/uri-res/

I2R?urn:pdi://oma.eop.gov.us/1993/9/14/3.text.2>.

Application to Federal Information Technology and Web Management:

Although this Executive order does not specifically address Customer

Service Standards in an electronic environment, Federal agency website

development teams still need to identify their customers, their customers’

needs, and set standards and benchmarks.

Excerpt of Original Text from Policy Instrument:

Sec. 1. Customer Service Standards. * * *

All executive departments and agencies (hereinafter

referred to collectively as "agency" or "agencies") that provide

significant services directly to the public shall provide those

services in a manner that seeks to meet the customer service

standard established herein and shall take the following

actions:

(a)

identify the customers who are, or should be, served by the

agency;

(b) survey customers to determine the kind and quality of

services they want and their level of satisfaction with

existing services;

(c) post service standards and measure results against them;
McClure, Sprehe & Eschenfelder 2-5 Federal Website Performance Measures

(d) benchmark customer service performance against the best in

business;

(e) survey front-line employees on barriers to, and ideas for,

matching the best in

business;

(f) provide customers with choices in both the sources of service

and the means of

delivery;

(g) make information, services, and complaint systems easily

accessible; and

(h) provide means to address customer complaints.

Provisions of the Executive Order also require agencies

to have a “Customer Service Plan,” to develop assessment

techniques to gauge the success of the plans, and to report

on the degree to which the plan is being accomplished.

2.1.3. Accessible Electronic and Information Technology

Rehabilitation Act of 1973 (Pub. L. 93-112), section 508 (added by the

Rehabilitation Act Amendments of 1986, Pub. L. 99-506, section

603(a), codified as amended at 29 U.S.C. § 794d). 1998.

Available at: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1994_uscode_suppl_4&docid=29usc794d>

Application to Federal Information Technology and Web

Management:

Section 508 requires that Federal agencies’ electronic and information

technology be accessible to people with disabilities, including employees

and members of the public. It also establishes requirements for any

electronic and information technology developed, maintained, procured, or

used by the Federal government. The Attorney General must determine

and communicate what information is necessary from Federal agencies to

conduct evaluations of their current electronic and information technology

systems’ accessibility. Agencies must then evaluate the accessibility of

their IT to individuals with disabilities. In addition, by February 7, 2000,

the Architectural and Transportation Barriers Compliance Board is

required to issue standards that define which electronic and information

technology is covered and describe what is meant by “accessible

technology” by setting forth technical and functional performance criteria.

The deadline for issuing of the Standards has been extended to August 7,

2000. Federal agency website development teams can find additional

information at the Center for IT Accommodation (CITA) to aid in their
McClure, Sprehe & Eschenfelder 2-6 Federal Website Performance Measures

design of websites accessible to people with disabilities. CITA is

available at <http://www.itpolicy.gsa.gov/cita/index.htm>.

Excerpt of Original Text from Policy Instrument:

(a) Requirements for Federal departments and agencies

(1) Accessibility

(A) Development, procurement, maintenance, or use of

electronic and information technology

When developing, procuring, maintaining, or using

electronic and information technology, each Federal department or

agency, including the United States Postal Service, shall ensure,

unless an undue burden would be imposed on the department or

agency, that the electronic and information technology allows,

regardless of the type of medium of the technology—

(i) individuals with disabilities who are Federal employees

to have access to and use of information and data that is

comparable to the access to and use of the information and data by

Federal employees who are not individuals with disabilities; and

(ii) individuals with disabilities who are members of the

public seeking information or services from a Federal department

or agency to have access to and use of information and data that is

comparable to the access to and use of the information and data by

such members of the public who are not individuals with

disabilities.

* * *

(c) Agency evaluations

Not later than 6 months after the date of enactment of the

Rehabilitation Act Amendments of 1998, the head of each Federal

department or agency shall evaluate the extent to which the

electronic and information technology of the department or agency

is accessible to and usable by individuals with disabilities

described in subsection (a)(1), compared to the access to and use of

the technology by individuals described in such subsection who are

not individuals with disabilities, and submit a report containing the

evaluation to the Attorney General.

2.1.4. President’s Memorandum on Electronic Government. December

17,1999.

Available at:

 <http://www.pub.whitehouse.gov/urires/I2R?urn:pdi://oma.eop.gov.us/1999/12/20/5.text.1>

Application to Federal Information Technology and Web

Management:
McClure, Sprehe & Eschenfelder 2-7 Federal Website Performance Measures

President Clinton announced twelve actions that Federal agencies

can take in conjunction with private industry to provide American

citizens with improved access to government services and information.

The President desired Federal agencies to provide standardized access

to government information and services. People should be able to find

needed government information without needing to know what agency

is responsible for disseminating the information. People should also

be confident that that their communication with government is secure

and their privacy protected.

Excerpt of Original Text from Policy Instrument:

2. The heads of executive departments and agencies

(agencies) shall, to the maximum extent possible, make

available online, by December 2000, the forms needed for

the top 500 Government services used by the public. Under

the Government Paperwork Elimination Act, where

appropriate, by October 2003, transactions with the Federal

Government should be available online for online

processing of services. To achieve this goal, the Director of

the Office of Management and Budget shall oversee agency

development of responsible strategies to make transactions

available online.

4. The heads of agencies shall continue to build good privacy

practices into their web sites by posting privacy policies as

directed by the Director of the Office of Management and

Budget and by adopting and implementing information

policies to protect children's information on web sites that

are directed at children.

5. The head of each agency shall permit greater access to its

officials by creating a public electronic mail address

through which citizens can contact the agency with

questions, comments, or concerns. The heads of each

agency shall also provide disability access on Federal web

sites.

7. The Secretaries of Health and Human Services, Education,

Veterans Affairs, and Agriculture, the Commissioner of

Social Security, and the Director of the Federal Emergency

Management Agency, working closely with other Federal

agencies that provide benefit assistance to citizens, shall

make a broad range of benefits and services available

though private and secure electronic use of the Internet.
McClure, Sprehe & Eschenfelder 2-8 Federal Website Performance Measures

9. The heads of agencies shall develop a strategy for

upgrading their respective agency's capacity for using the

Internet to become more open, efficient, and responsive,

and to more effectively carry out the agency's mission. At a

minimum, this strategy should involve: (a) expanded

training of Federal employees, including employees with

policy and senior management responsibility; (b)

identification and adoption of "best practices" implemented

by leading public and private sector organizations; (c)

recognition for Federal employees who suggest new and

innovative agency applications of the Internet; (d)

partnerships with the research community for

experimentation with advanced applications; and (e)

mechanisms for collecting input from the agency's

stakeholders regarding agency use of the Internet.

2.2. FEDERAL INFORMATION POLICY

2.2.1. U.S. Information Infrastructure Task Force (IITF). 1993 The National

Information Infrastructure: Agenda for Action.

Available at: <http://metalab.unc.edu/nii/toc.html>.

Application to Federal Information Technology and Web

Management:

This document has served as a blueprint for President Clinton’s National Information

Infrastructure Initiative. The document outlined the President’s map for

developing a National Information Infrastructure (NII) listing goals and

the necessary actions to accomplish the goals. Goal nine called for an

increase in access to government information and to improve government

procurement. This goal called for Federal agencies in conjunction with

state and local governments to use the NII to expand the information

available to the public, and make that information so it can be accessed

easily and equitably.

Excerpt of Original Text from Policy Instrument:

9. Provide Access to Government Information and Improve

Government Procurement

Thomas Jefferson said that information is the currency of

democracy. Federal agencies are among the most prolific

collectors and generators of information that is useful and valuable

to citizens and business. Improvement of the nation's information

infrastructure provides a tremendous opportunity to improve the

delivery of government information to the taxpayers who paid for
McClure, Sprehe & Eschenfelder 2-9 Federal Website Performance Measures

its collection; to provide it equitably, at a fair price, as efficiently

as possible.

The Federal government is improving every step of the

process of information collection, manipulation, and

dissemination. The Administration is funding research programs

that will improve the software used for browsing, searching,

describing, organizing, and managing information. But it is

committed as well to applying those tools to the distribution of

information that can be useful to the public in their various roles as

teachers, researchers, businesspeople, consumers, etc.

Action: Improve the accessibility of government

information. IITF working groups will carefully consider the

problems associated with making government information broadly

accessible to the public electronically. Additionally, several interagency efforts have been started to ensure that the right

information is stored and available. Finally, to help the public find

government information, an inter-agency project has been formed

to develop a virtual card catalogue that will indicate the availability

of government information in whatever form it takes.

Action: Enhance citizen access to government information.

In June 1993, OMB prescribed new polices pertaining to the

acquisition, use, and distribution of government information by

Federal agencies. Among other things, the policies mandate that, in

distributing information to the public, Federal agencies should

recoup only those costs associated with the dissemination of that

information, not with its creation or collection. Moreover, a

number of inter-agency efforts are under way to afford greater

public access to government information. One project seeks to turn

thousands of local and field offices of various Federal agencies

into Interactive Citizen Participation Centers, at which citizens can

communicate with the public affairs departments of all Federal

agencies.

2.2.2 Privacy and the National Information Infrastructure: Principles for

Providing and Using Personal Information. Privacy Working Group.

Information Policy Committee. Information Infrastructure Task

Force. (June 6, 1995.)

Available at: <http://www.iitf.nist.gov/ipc/ipc/ipcpubs/niiprivprin_final.html>.

Application to Federal Information Technology and Web

Management:

The Privacy Working Group (PWG), part of the Information Infrastructure Task Force

(IITF) was formed to develop guidelines or principles for providing and

using personal information in the electronic environment or within the
McClure, Sprehe & Eschenfelder 2-10 Federal Website Performance Measures

National Information Infrastructure (NII). Personal information refers to

information such as identifying information, social security number, name,

address, and phone number; financial records or statements; and health

records. The principles were designed to provide guidelines to personal

information users and personal information providers. These principles are

not legally binding, and only serve to guide. In developing the standards,

PWG generally followed international standards such as the OECD

guidelines written in 1980.

Federal website developers and administrators need to be concerned with

• Acquisition Principle: only collect and keep information that supports

current or planned activities, and assess the impact on privacy in

collecting, disclosing, or using a person's personal information.

• Protection Principle: use appropriate technical and managerial

controls to protect the integrity, confidentiality, and quality of the

personal information.

• Notice Principle: personal information providers should be informed

by the website as to why information collection is occurring; how the

information is to be used; what steps are taken to protect the

information; consequences of providing or withholding information;

any rights of redress. What happens if information loses integrity, or

is leaked.

• Fairness principle: Information should be used in the manner in which

the user was told it would be used.

• Education principle: Information users should educate themselves and

the public about how information privacy can be maintained.

2.2.3. COPYRIGHT ACT

2.2.3.1 Statute: Digital Millennium Copyright Act. (Public Law 105-

304, Title 1, Sec. 103(a), codified at 17 U.S.C. §§ 1201-1205).

1998.

Available at: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1994_uscode_suppl_4&docid=17usc1201

>

Application to Federal Information Technology and Web

Management:

This law amended Title 17 of the U.S. Code to implement the World

Intellectual Property Organization Copyright Treaty and Performance and

Phonograms Treaty. The Digital Millennium Copyright Act (DMCA)

incorporated into Title 17 provisions preventing the circumvention of

technological protections of works. The DMCA also added provisions

prohibiting the removal or alteration of “copyright management

information.” As such, the DMCA tailors protection to holders of
McClure, Sprehe & Eschenfelder 2-11 Federal Website Performance Measures

copyright in electronic media. It prohibits the circumvention of

technological measures that control access to protected works and/or

manufacturing or trafficking in technology designed to circumvent

measures that control access to, or protect rights of copyright owners in,

such works. The act provides exemptions for nonprofit libraries, archives,

or educational institutions that gain access to a commercially exploited

copyrighted work solely to make a good faith determination of whether to

acquire such work, subject to certain restrictions; for purposes of

achieving interoperability of computer programs; and for authorized

investigative, protective, information security, or intelligence activities of

the United States, a state, or political subdivision of a state.

2.2.3.2 Implementing Guidance: The Copyright Office completed a

statutorily mandated study on the Digital Millennium Copyright Act in

December 1998. The study is available at

<http://www.loc.gov/copyright/legislation/dmca.pdf>.

2.2.4 RIGHTS OF ACCESS TO INFORMATION

2.2.4.1 FREEDOM OF INFORMATION

2.2.4.1.1. Statute: Freedom of Information Act. (Public Law 89-487,

codified as amended, 5 U.S.C. § 552). 

Available at: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1994_uscode_suppl_4&docid=5usc552>

Application to Federal Information Technology and Web

Management:

The purpose of the Freedom of Information Act (“FOIA”) is to allow

public access to agency records, thereby promoting an informed citizenry.

Unless exempted under section 552a, agency records are subject to proper

request. EFOIA eliminated doubt that information stored on computers is

a “record” within the definition of the statute. Because digital

information, including information posted by a Federal agency on its

website, falls under the statutory definition of a record, Federal agencies

are responsible for providing information in this format just as they are for

providing print materials.

Certain government bodies are exempted from the requirements of

FOIA because they fall outside the definition of “agency” set forth in the

statute, and the law contains nine exemptions for various classes of

records. As noted in the section dealing with the Privacy Act, certain

types of information are protected from public disclosure and

consequently their disclosure is not required under FOIA. Each section is

defined in terms of the other: records required under FOIA include
McClure, Sprehe & Eschenfelder 2-12 Federal Website Performance Measures

everything not protected under the Privacy Act.

2.2.4.1.2. Statute: Electronic Freedom of Information Act

Amendments of 1996. (Public Law 104-231, amending 5 U.S.C. §552).

Available at: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=104_cong_public_laws&docid=f:publ231.104.pdf

>

Application to Federal Information Technology and Web

Management:

The Electronic Freedom of Information Act Amendments of 1996 (EFOIA) encourage

government agencies to use new technology to enhance public access to

agency records and information. The act amends the FOIA to define

“record” to mean information in records maintained by an agency in any

format. Agencies responding to a request for records also must make

reasonable efforts to search for the records in electronic form or format.

All agencies are to make available electronically frequently

requested records obtainable under FOIA and are to maintain an

Electronic Reading Room. These Electronic Reading Rooms present

special interest collections as well as “policy statements, administrative

rulings and manuals, and other materials that affect members of the

public.” For a report on agency compliance see the OMB Watch study

report at <http://www.ombwatch.org/site/info/efoia99/efoiareport.html>.

Excerpt of Original Text from Policy Instrument:

Section 3. Application of Requirements to Electronic Format

Information

(f) For purposes of this section, the term--

(2) “record” and any other term used in this section in

reference to information includes any information that would

be an agency record subject to the requirements of this section

when maintained by an agency in any format, including an

electronic format.

Section 5. Honoring Form or Format Requests

(C) In responding under this paragraph to a request for

records, an agency shall make reasonable efforts to search for

the records in electronic form or format, except when such

efforts would significantly interfere with the operation of the

agency's automated information system.

2.2.4.1.3. Statute: Privacy Act of 1974. (Public Law 93-579, Sec. 3,

codified as amended at 5 U.S.C. § 552a).

Available at: < http://frwebgate.access.gpo.gov/cgi-
McClure, Sprehe & Eschenfelder 2-13 Federal Website Performance Measures

bin/getdoc.cgi?dbname=1994_uscode_suppl_4&docid=5usc552a >.

Application to Federal Information Technology and Web

Management:

Congress intended the Privacy Act to protect citizens against

disclosure of their personal information unless required by the

Freedom of Information Act, section 552. Congress was especially

concerned with the automation of information retrieval and

technological advances in information storage when it enacted this

legislation. The legislative history of the Privacy Act shows it

contemplates the problem of individuals’ increased personal exposure

through the automation of government operations and development of

computerized government systems.4

The Privacy Act describes the manner in which personal

information of individuals can be disclosed by the government. No

record containing personal information may be disclosed without the

prior written consent of the individual to whom the information

pertains. The act also requires agencies to allow individuals access to

their records.

Excerpt of Original Text from Policy Instrument:

(b) Conditions of Disclosure.

No agency shall disclose any record which is contained in a system of

records by any means of communication to any person, or

to another agency, except pursuant to a written request by,

or with the prior written consent of, the individual to whom

the record pertains, unless disclosure of the record would

be -

(e) Agency Requirements. - Each agency that maintains a

system of records shall

(1) maintain in its records only such information about an

individual as is relevant and necessary to accomplish a

purpose of the agency required to be accomplished by

statute or by executive order of the President;

 

4

See Senator Ervin’s speech on June 11, 1974: “It is a rare person who has escaped the quest of modern

government for information. ... When this quite natural tendency of Government to acquire and keep and

share information about citizens is enhanced by computer technology and when it is subjected to the

unrestrained motives of countless political administrators, the resulting threat to individual privacy make it

necessary for Congress to reaffirm the principle of limited, responsive Government on behalf of freedom.”

Senate Rpt. (of Government Operations Committee) No. 93-1183, page 6919; Thomas v. U.S. Dept. of

Energy, 719 F3d 342 (1983).]
McClure, Sprehe & Eschenfelder 2-14 Federal Website Performance Measures

(2) collect information to the greatest extent practicable

directly from the subject individual when the information

may result in adverse determinations about an individual's

rights, benefits, and privileges under Federal programs;

(3) inform each individual whom it asks to supply

information, on the form which it uses to collect the

information or on a separate form that can be retained by

the individual.

2.2.4.1.3.1. Implementing Guidance: Appendix I to OMB

Circular No. A-130 - Federal Agency Responsibilities for

Maintaining Records About Individuals

“This Appendix describes agency responsibilities for

implementing the reporting and publication requirements of

the Privacy Act of 1974, 5 U.S.C. 552a, as amended

(hereinafter "the Act"). It applies to all agencies subject to

the Act. Note that this Appendix does not rescind other

guidance OMB has issued to help agencies interpret the

Privacy Act's provisions, e.g., Privacy Act Guidelines (40

FR 28949-28978, July 9, 1975), or Final Guidance for

Conducting Matching Programs (54 FR at 25819, June 19,

1989).”

2.2.4.1.3.2. Implementing Guidance: Instructions on complying

with President’s Memorandum of May 14, 1998, “Privacy and

Personal Information in Federal Records.”

Available at:

<http://www.pub.whitehouse.gov/urires/I2R?urn:pdi://oma.eop.go

v.us/1998/5/14/8.text.1>.

Application to Federal Information Technology and Web

Management:

In this Memorandum the President directed Federal

agencies to review their current information practices and

ensure that they are being conducted in accordance with the

Privacy Act of 1974. The President also directed OMB to issue

instructions to the agencies on how to conduct this review.

Excerpt of Original Text from Policy Instrument:

1. Designate a Senior Official for Privacy Policy

Each agency head should have already designated a senior

official within the agency to assume primary responsibility

for privacy policy, in accordance with the President's

Memorandum. This individual will not necessarily be the
McClure, Sprehe & Eschenfelder 2-15 Federal Website Performance Measures

same person who is responsible for implementation of the

Privacy Act. For most Cabinet agencies, the appropriate

official would probably be a policy official at the Assistant

Secretary level, or equivalent, who in a position within the

agency to consider privacy policy issues on a national level.

2. Review and Improve the Management of Privacy Act

Systems of Records

Each agency shall conduct a thorough review of its systems

of records, system of records notices, and routine uses in

accordance with the criteria and guidance below. Because

the President directed agencies to review systems of

records, we have provided guidance on a subset of the

Privacy Act's requirements that are particularly relevant to

systems of records.

2.2.4.1.3.3. Implementing Guidance: M-99-18. Director Jacob

J. Lew’s Memorandum for the Heads of Executive

Departments and Agencies on Privacy Policies on Federal Web

Sites. 1999.

Available at: <http://cio.gov/docs/webprivl.htm>.

Application to Federal Information Technology and Web

Management:

This memo instructs Departments and Agencies to display

their privacy policies on their websites, and to comply with various

laws and regulations governing privacy policies, such as the

Privacy Act, OMB Circular No. A-130, and Principles for

Providing and Using Personal Information published by the

Information Infrastructure Task Force on June 6, 1995. Federal

agency web developers must include a privacy statement on their

departmental websites.

2.2.4.1.3.4. Implementing Guidance: M-00-13, Director Jacob

J. Lew’s Memorandum for the Heads of Executive

Departments and Agencies on Privacy Policies and Data

Collection on Federal Web Sites, June 22, 2000

Available at: <

http://www.whitehouse.gov/OMB/memoranda/m00-13.html>.

Application to Federal Information Technology and Web

Management

“Particular privacy concerns may be raised when uses of web

technology can track the activities of users over time and across
McClure, Sprehe & Eschenfelder 2-16 Federal Website Performance Measures

different web sites. . . “Cookies” – small bits of software that are

placed on a web user’s hard drive – are a principal example of

current web technology that can be used this way. . . Because of

the unique laws and traditions about government access to citizens’

personal information, the presumption should be that “cookies”

will not be used a Federal web sites. Under this new Federal

policy, “cookies” should not be used at Federal web sites, or by

contractors when operating web sites on behalf of agencies, unless,

in addition to clear and conspicuous notice, the following

conditions are met: a compelling need to gather the data on the

site; appropriate and publicly disclosed privacy safeguards for

handling of information derived from “cookies”; and personal

approval by the head of the agency.”

Also, “"It is federal policy that all Federal web sites and

contractors when operating on behalf of agencies shall comply

with the standards set forth in the Children's Online Privacy

Protection Act of 1998 with respect to the collection of personal

information online at web sites directed to children."

2.2.4.1.4. Statute: Public Printing and Documents: Depository

Library Program (44 U.S.C. 1902).

Available at: http://frwebgate1.access.gpo.gov/cgibin/waisgate.cgi?WAISdocID=040147185+0+0+0&WAISaction=r

etrieve

Application to Federal Information Technology and Web

Management:

Agencies are to make their publications available to depository

libraries through the Superintendent of Documents.

Excerpt of Original Text from Policy Instrument:

Sec. 1902. Government publications, except those

determined by their issuing components to be required for official

use only or for strictly administrative or operational purposes

which have no public interest or educational value and publications

classified for reasons of national security, shall be made available

to depository libraries through the facilities of the Superintendent

of Documents for public information.

2.2.5. PAPERWORK REDUCTION ACT

2.2.5.1. Statute: Paperwork Reduction Act of 1995. (Public Law 104-

13, codified at 44 U.S.C. 3501 et seq.).
McClure, Sprehe & Eschenfelder 2-17 Federal Website Performance Measures

Available at:

P.L. 104-13: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=104_cong_public_laws&docid=f:publ13.104.

pdf >.

44 U.S.C. 3501: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1994_uscode_suppl_3&docid=44usc3501>.

Application to Federal Information Technology and Web

Management:

The Paperwork Reduction Act (PRA) intends to minimize the

paperwork burden for individuals, small businesses, educational and

nonprofit institutions, Federal contractors, state, local, and tribal

governments, and other persons which results from the collection of

information by or for the Federal government. PRA is also designed

to coordinate, integrate, and make uniform Federal information

resources management policies and practices in order to improve the

efficiency of government programs. The 1995 amendments to the

PRA added extensive agency responsibilities for information

dissemination, responsibilities that are applicable to agency websites.

Suggested implementation of this law in conjunction with the

Government Performance and Results Act of 1993 and the Information

Technology Management Reform Act of 1996 occurs in Executive

Order 13011, Federal Information Technology. This implementation

was followed up with another memorandum issued by the President on

December 17, 1999, Memorandum on E-Government, to improve the

interoperability of government agencies and the sharing of information

resources.

Excerpt of Original Text from Policy Instrument:

Sec. 2. Coordination of Federal Information Policy.

Chapter 35 of title 44, United States Code, is amended to

read as follows:

* * *

“Section 3504. Authority and Functions of Director

“(a)(1) The Director shall oversee the use of

information resources to improve the efficiency and

effectiveness of governmental operations to serve agency

missions, including burden reduction and service delivery

to the public. In performing such oversight, the Director

shall—

“(A) develop, coordinate and oversee the

implementation of Federal information resources
McClure, Sprehe & Eschenfelder 2-18 Federal Website Performance Measures

management policies, principles, standards, and guidelines;

and

“(B) provide direction and oversee—

* * *

“(v) privacy, confidentiality, security, disclosure,

and sharing of information; and

“(vi) the acquisition and use of information

technology.

* * *

“Section 3506. Federal Agency Responsibilities

* * *

"(d) With respect to information dissemination, each

agency shall--

(1) ensure that the public has timely and equitable

access to the agency's public information, including

ensuring such access through--

(A) encouraging a diversity of public and

private sources for information based on government public

information;

(B) in cases in which the agency provides

public information maintained in electronic format,

providing timely and equitable access to the underlying

data (in whole or in part); and

(C) agency dissemination of public

information in an efficient, effective, and economical

manner;

(2) regularly solicit and consider public input on the

agency's information dissemination activities;

(3) provide adequate notice when initiating,

substantially modifying, or terminating significant

information dissemination products;"

* * *

Section 3511. Establishment and Operation of the Government

Information Locator System

* * *

(a) In order to assist agencies and the public in

locating information and to promote information sharing

and equitable access by the public, the Director shall - (1)

cause to be established and maintained a distributed

agency-based electronic Government Information Locator

Service (hereafter in this section referred to as the

''Service''), which shall identify the major information

systems, holdings, and dissemination products of each

agency;

2.2.5.1. Implementing Guidance: Office of Management and
McClure, Sprehe & Eschenfelder 2-19 Federal Website Performance Measures

Budget (OMB), Circular A-130, Management of Federal

Information Resources. 1996. (61 FR 6428)

 Available at:

<http://www.whitehouse.gov/OMB/circulars/a130/a130.html>.

Application to Federal Information Technology and Web

Management:

Circular A-130 provides uniform government-wide

information resources management policies as required by the

Paperwork Reduction Act of 1980 and as amended by the

Paperwork Reduction Act of 1995 (44 U.S.C. 35). This policy

instrument provides a framework for the development of an

information management plan to aid Federal agencies in the

development of their websites.

Excerpt of Original Text from Policy Instrument:

8. Policy. a. Information Management Policy

1. Information Management Planning. Agencies shall

plan in an integrated manner for managing information

throughout its life cycle. Agencies shall:

(a) Consider, at each stage of the information life cycle,

the effects of decisions and actions on other stages of

the life cycle, particularly those concerning information

dissemination

(e) Integrate planning for information systems with

plans for resource allocation and use, including

budgeting, acquisition, and use of information

technology;

8. Policy. b. Information Systems and Information

Technology Management

1. Evaluation and Performance Measurement.

Agencies shall promote the appropriate application of

Federal information resources as follows:

(a) Seek opportunities to improve the effectiveness and

efficiency of government programs through work

process redesign and the judicious application of

information technology;

(b) Prepare, and update as necessary throughout the

information system life cycle, a benefit-cost analysis for

each information system:

9. Assignment of Responsibilities. c. Department of

Commerce. The Secretary of Commerce shall:
McClure, Sprehe & Eschenfelder 2-20 Federal Website Performance Measures

4. Conduct studies and evaluations concerning

telecommunications technology, and concerning the

improvement, expansion, testing, operation, and use of

Federal telecommunications systems and advise the

Director, OMB, and appropriate agencies of the

recommendations that result from such studies;

2.2.6. CLINGER-COHEN ACT

2.2.6.1. Statute: Information Technology Management Reform Act of

1996. (Public Law 104-106, Division E). 1996. Amended by

Public Law 104-208, Division A, Title I, Sec. 101(f) (Title VIII,

Sec. 808(b)), providing that this act (and the Federal

Acquisition Reform Act of 1996 (Public Law 104-106, Division

D)) may be cited as the "Clinger-Cohen Act of 1996."

Available at:

P.L. 104-106: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=104_cong_public_laws&docid=f:publ106.104.pdf

>

P.L. 104-208: < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=104_cong_public_laws&docid=f:publ208.104.pdf

>

Application to Federal Information Technology and Web

Management:

The Clinger-Cohen Act is intended to address the management of IT in the

Federal government. It is to accomplish this goal through a variety of

methods, including the use of capital planning for IT acquisitions and

investments, the establishment of Chief Information Officer (CIO)

positions in Federal departments and agencies, and the requirement of

performance measurements of IT. The CIO, through IT, would also be

responsible for making government more effective, efficient, and

productive, and include the implementation of policies affecting Federal

agency Web development. The CIO has agency responsibility for IT

management and development and reports directly to the head of the

agency. As regard performance measures, the Clinger-Cohen Act

represents the specific application of the Government Performance and

Results Act to the area of IT.

Excerpt of Original Text from Policy Instrument:

Section 5112. Capital Planning And Investment Control

(b) USE OF INFORMATION TECHNOLOGY IN FEDERAL

PROGRAMS- The Director shall promote and be responsible for

improving the acquisition, use, and disposal of information technology by

the Federal Government to improve the productivity, efficiency, and
McClure, Sprehe & Eschenfelder 2-21 Federal Website Performance Measures

effectiveness of Federal programs, including through dissemination of

public information and the reduction of information collection burdens on

the public

(f) USE OF BEST PRACTICES IN ACQUISITIONS- The Director shall

encourage the heads of the executive agencies to develop and use the best

practices in the acquisition of information technology.

Section 5113. Performance-Based and Results-Based Management

(a) IN GENERAL- The Director shall encourage the use of performancebased and results-based management in fulfilling the responsibilities

assigned under section 3504(h), of title 44, United States Code.

(b) Evaluation of Agency Programs and Investments-

(1) REQUIREMENT- The Director shall evaluate the information

resources management practices of the executive agencies with respect to

the performance and results of the investments made by the executive

agencies in information technology.

(2) DIRECTION FOR EXECUTIVE AGENCY ACTION- The Director

shall issue to the head of each executive agency clear and concise

direction that the head of such agency shall-

(A) establish effective and efficient capital planning processes for

selecting, managing, and evaluating the results of all of its major

investments in information systems

2.2.6.2. Implementing Guidance:

2.2.6.2.1. Executive Order

Federal Information Technology. (Executive Order 13011).

1996.

Available at:

 < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1996_register&docid=fr19jy96-133.pdf>

Application to Federal Information Technology and Web

Management:

This Executive Order links the Clinger-Cohen Act, the PRA, and

the GPRA. It formalizes OMB’s oversight of IT management and

stresses the importance of performance-based planning and

implementation of Federal IT. Executive Order 13011 also creates

the Chief Information Officer Council, the Government

Information Technology Services Board, and the Information

Technology Resources Board. Also falling under the domain of

this executive order are the establishment of interagency support

structures to share IT ideas, minimize duplication, and increase

interoperability; and design technology procedures and standards.
McClure, Sprehe & Eschenfelder 2-22 Federal Website Performance Measures

The order does not formally define the support structure. Federal

web site design under this executive order should be interoperable

and standardized with other agencies. Website design must also

operate within the realm of the Clinger-Cohen Act, the PRA, and

the GPRA. The CIO would be in charge of overseeing the design

with ultimate control still resting with the OMB.

Excerpt of Original Text from Policy Instrument:

Section 1. Policy. It shall be the policy of the United States

Government that executive agencies shall: * * *

(b) refocus information technology management to support directly

their strategic missions, implement an investment review process

that drives budget formulation and execution for information

systems, and rethink and restructure the way they perform their

functions before investing in information technology to support

that work;

(c) establish clear accountability for information resources

management activities by creating agency Chief Information

Officers(CIOs) with the visibility and management responsibilities

necessary to advise the agency head on the design, development,

and implementation of those information systems. These

responsibilities include: (1) participating in the investment review

process for information systems; (2) monitoring and evaluating the

performance of those information systems on the basis of

applicable performance measures; and, (3) as necessary, advising

the agency head to modify or terminate those systems....

Section 2. Responsibilities of Agency Heads. The head of each

executive agency shall: (a) effectively use information technology

to improve mission performance and service to the public;

(b) strengthen the quality of decisions about the employment of

information resources to meet mission needs through integrated

analysis, planning, budgeting, and evaluation processes, including:

* * *

(2) establishing mission-based performance measures for

information systems investments, aligned with agency

performance plans prepared pursuant to the Government

Performance and Results Act of 1993 (Public Law 103-62)....

2.2.6.2.2. OMB Circular: Proposed Revision of OMB Circular No. A130, Management of Federal Information Resources, April 13,

2000

Available at:

http://www.whitehouse.gov/omb/federeg/reva130.pdg.
McClure, Sprehe & Eschenfelder 2-23 Federal Website Performance Measures

Application to Federal Information Technology and Web

Management:

In proposing to revise OMB Circular No. A-130, OMB announced

that its primary reason for doing so was to implement provisions of

the Clinger-Cohen Act. Agencies must develop annually an

Information Resources Management Plan. The plan must include

an annual performance plan as required by GPRA. This must

include an accountability report, comparing actual performance

with expected performance. Their benefit-cost analyses of

information systems must include performance measures. In its

capital planning process, each agency must institute performance

measures that monitor actual performance as compared with

expected results. They must also conduct evaluations of

information systems and of information resource management

processes.

2.2.7. SECURITY

2.2.7.1. Statute: Computer Security Act of 1987. (Public Law 100-235,

codified in part as amended at 15 U.S.C. §§ 278g-3, 278g-4). 1988.

Available at: <http://cio.doe.gov/ucsp/csa.htm>.

Application to Federal Information Technology and Web

Management:

The Computer Security Act of 1987establishes in law for information

technology the basic principle of security, namely, security commensurate

with the risk and magnitude of the harm resulting from the loss, misuse, or

unauthorized access to or modification of information. It also creates a

means, through the National Institute of Standards and Technology

(NIST), for establishing minimum acceptable security practices for

Federal computer systems without limiting the scope of security measures

already planned or in use. The act requires each agency with a Federal

computer system to establish a plan for the security and privacy of

sensitive information, and prescribes “cost effective” rather than

“absolute” computer security. The NIST is responsible for developing

standards and guidelines necessary for assuring the protection of sensitive

information. Federal websites offer one more entry point through which

security can be breached. Federal CIOs must utilize the most recent

proven effective hardware and software to protect sensitive departmental

information and computer systems.

Congress is currently considering H.R. 2413, the Computer Security

Enhancement Act of 1999, available at
McClure, Sprehe & Eschenfelder 2-24 Federal Website Performance Measures

<http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=106_cong_bills&docid=f:h2413ih.txt.pdf> This

bill would enhance the ability of NIST to improve computer security for

nonclassified information on Federal computer systems. It would also

promote the use of private sector security technology to protect Federal

computer systems. Rather than government driving security technology,

private industry or the market would be responsible for developing

security technology.

Excerpt of Original Text from Policy Instrument:

Section 3. Establishment of Computer Standards Program

The Act of March 3, 1901, (15 U.S.C. 271-278h), is amended—

* * *

(2) by redesignating section 20 as section 22, and by inserting after

section 19 the following new sections: “SEC. 20. (a) The National

Bureau of Standards shall—

* * * 

“(3) have responsibility within the Federal Government for developing

technical, management, physical, and administrative standards and

guidelines for the cost-effective security and privacy of sensitive

information in Federal computer systems except [as specified in the

law];

* * *

“(6) develop validation procedures for, and evaluate the effectiveness

of, standards and guidelines developed pursuant to paragraphs (1), (2),

and (3) of this subsection through research and liaison with other

government and private agencies.”

2.2.7.2. Implementing Guidance

2.2.7.2.1. OMB Circular No. A-130, Appendix III, Security of

Federal Automated Information Resources

Available at:

<http://www.whitehouse.gov/OMB/circulars/a130/a130.html>.

Application to Federal Information Technology and Web

Management

“This Appendix establishes a minimum set of controls to be

included in Federal automated information security programs;

assigns Federal agency responsibilities for the security of

automated information; and links agency automated information

security programs and agency management control systems

established in accordance with OMB Circular No. A-123. The
McClure, Sprehe & Eschenfelder 2-25 Federal Website Performance Measures

Appendix revises procedures formerly contained in Appendix III to

OMB Circular No. A-130 (50 FR 52730; December 24, 1985), and

incorporates requirements of the Computer Security Act of 1987

(P.L. 100-235) and responsibilities assigned in applicable national

security directives.”

2.2.7.1.2. Presidential Decision Directive 63, “Protecting America’s

Critical Infrastructures.”

Available at <http://www.pub.whitehouse.gov/urires/I2R?urn:pdi://oma.eop.gov.us/1998/5/26/1.text.1>.

Application to Federal Information Technology and Web

Management:

This Presidential Directive built on the recommendations of the

President's Commission on Critical Infrastructure Protection and

sets a goal of a reliable, interconnected, and secure information

system infrastructure by the year 2003, and significantly increased

security to government systems by the year 2000, by: Immediately

establishing a national center to warn of and respond to attacks.

Ensuring the capability to protect critical infrastructures from

intentional acts by 2003. PDD 63 addressed the cyber and physical

infrastructure vulnerabilities of the Federal government by

requiring each department and agency to work to reduce its

exposure to new threats. It set up a new structure to deal with this

important challenge: a National Coordinator; the National

Infrastructure Protection Center (NIPC) s; Information Sharing and

Analysis Centers (ISACs); a National Infrastructure Assurance

Council; and The Critical Infrastructure Assurance Office to

provide support to the National Coordinator’s work with

government agencies and the private sector in developing a

national plan. The applicability to agency websites is that PPD 63

is fundamental for website security and effectively requires that

agencies have performance measures of website security.

2.2.8. ELECTRONIC COLLECTION AND DIGITAL SIGNATURES

2.2.8.1. Statute: Government Paperwork Elimination Act. (Public

Law 105-277, Division C, Title XVII, amending 44 U.S.C.). 1998.

Available at:

 <http://thomas.loc.gov/cgibin/bdquery/z?d105:HR04328:|TOM:/bss/d105query.html|>.

Application to Federal Information Technology and Web

Management:
McClure, Sprehe & Eschenfelder 2-26 Federal Website Performance Measures

The Government Paperwork Elimination Act (GPEA) provides for Federal

agencies to give persons who are required to maintain, submit, or disclose

information the option of doing so electronically, when practicable, by

October 21, 2003. In addition, GPEA directs that electronic authentication

(electronic signature) methods be used to verify the identity of the sender

and the integrity of the electronic content. The Director of the Office of

Management and Budget is required to develop procedures for the use and

acceptance of electronic signatures by executive agencies and, in

cooperation with the National Telecommunications and Information

Administration, to conduct and report to Congress an ongoing study of the

use of electronic signatures on paperwork reduction and electronic

commerce, individual privacy, and the security and authenticity of

transactions.

Excerpt of Original Text from Policy Instrument:

Section 1. Policy

The Office of Management and Budget is directed to maintain

compatibility with standards and technology for electronic signatures

generally used in commerce and industry and by State governments and to

ensure that electronic signatures are as reliable as is appropriate for the

purpose in question ant that the electronic record keeping systems readily

preserve the information submitted.

Section 2. Procedures

An agency’s determination of which technology is appropriate for a given

transaction must include a risk assessment, and an evaluation of targeted

customer or user needs. Performing a risk assessment to evaluate

electronic signature alternatives should not be viewed as an isolated

activity or an end in itself. These agency risk assessments should draw

from and feed into the interrelated requirements of the Paperwork

Reduction Act, the Computer Security Act, the Government Performance

and Results Act, the Clinger-Cohen Act, the Federal Managers Financial

Integrity Act, and the Chief Financial Officers Act.

2.2.8.2. Implementing Guidance: Implementation of the Government

Paperwork Elimination Act. 1999.

Available at: < http://www.whitehouse.gov/omb/fedreg/gpea2.html >.

Application to Federal Information Technology and Web

Management:

The Government Paperwork Elimination Act provides for Federal

agencies, by October 21, 2003, to give persons who are required to

maintain, submit, or disclose information the option of doing so

electronically when practicable as a substitute for paper, and to use

electronic authentication (electronic signature) methods to verify the
McClure, Sprehe & Eschenfelder 2-27 Federal Website Performance Measures

identity of the sender and the integrity of electronic content. The Act

specifically provides that electronic records and their related electronic

signatures are not to be denied legal effect, validity, or enforceability

merely because they are in electronic form. OMB’s implementation

of the Act is in two parts. The first part sets forth the policies and

procedures for implementing the Act, and requesting certain specific

agencies to provide assistance in particular areas. The second part is

intended to provide Federal managers with practical implementation

guidance.

Excerpt of Original Text from Policy Instrument:

Section 2. What GPEA procedures should agencies follow?

a. Accordingly, agencies should develop and implement plans,

supported by an assessment of whether to use and accept documents in

electronic form and to engage in electronic transactions. The

assessment should weigh costs and benefits and involve an appropriate

risk analysis, recognizing that low-risk information processes may

need only minimal consideration, while high-risk processes may need

extensive analysis.

c. The assessment should develop strategies to mitigate risks and

maximize benefits in the context of available technologies, and the

relative total costs and effects of implementing those technologies on

the program being analyzed. The assessment also should be used to

develop baselines and verifiable performance measures that track the

agency's mission, strategic plans, and tactical goals, as required by the

Clinger-Cohen Act.

d. In addition to serving as a guide for selecting the most appropriate

technologies, the assessment of costs and benefits should be designed

so that it can be used to generate a business case and verifiable return

on investment to support agency decisions regarding overall

programmatic direction, investment decisions, and budgetary

priorities.

Section 3. How should agencies implement these policies and

procedures?

a. To ensure a smooth and cost-effective transition to an electronic

government that provides improved service to the public, each agency

must:

(1) Develop a plan (including a schedule) by October, 2000 that

provides for continued implementation, by the end of Fiscal Year

2003, of optional electronic maintenance, submission, or

transaction of information when practicable as a substitute for

paper, including through the use of electronic signatures when

practicable. A copy of the plan should be provided to OMB.
McClure, Sprehe & Eschenfelder 2-28 Federal Website Performance Measures

(7) Consider the record keeping functionality of any systems that

store electronic documents and electronic signatures, to ensure

users have appropriate access to the information and can meet the

agency's record keeping needs.

Section 2. Procedures

b. An agency’s determination of which technology is

appropriate for a given transaction must include a risk

assessment, and an evaluation of targeted customer or

user needs. Performing a risk assessment to evaluate

electronic signature alternatives should not be viewed as

an isolated activity or an end in itself. These agency risk

assessments should draw from and feed into the

interrelated requirements of the Paperwork Reduction

Act, the Computer Security Act, the Government

Performance and Results Act, the Clinger-Cohen Act,

the Federal Managers Financial Integrity Act, and the

Chief Financial Officers Act.

Section 3. Agency Responsibilities

a. In order to ensure a smooth and cost-effective

transition to a more electronic government providing

improved service to the public, each agency shall:

1. include in its strategic IT plans supporting program

responsibilities (required under OMB Circular A-11) a

summary of the agency’s schedule to implement

optional electronic maintenance, submission, or

disclosure of information when practicable as a

substitute for paper, including through the use of

electronic signatures when practicable, by the end of

Fiscal Year 2003 (note: agencies need not revise their

reports on Federal purchasing and payment already

required by OMB M-99-02, but should include the

automation of purchasing and payment functions in their

schedule)

2.2.9. FEDERAL RECORDS

2.2.9.1. Statute: Federal Records Act, Title 44 U.S. Code, Chapter 31,

Records Management by Federal Agencies

Available from < http://frwebgate.access.gpo.gov/cgibin/getdoc.cgi?dbname=1994_uscode_suppl_3&docid=44usc3101>
McClure, Sprehe & Eschenfelder 2-29 Federal Website Performance Measures

“The head of each Federal agency shall make and preserve records

containing adequate and proper documentation of the organization,

functions, policies, decisions, procedures, and essential transactions of the

agency and designed to furnish the information necessary to protect the

legal and financial rights of the Government and of persons directly

affected by the agency's activities. “

22.9.2. Implementing Guidance:

2.2.9.2.1. NARA Bulletin 98-02. 1998 [disposition of electronic

records].

Available at: <http://www.nara.gov/records/policy/b9802.html>.

Application to Federal Information Technology and Web

Management:

NARA Bulletin 98-02 reminds Federal agencies of their

obligations under Federal law, 36 CFR Part 1234, to provide adequate

documentation of agency activities, and provides guidance to Federal

agencies in accomplishing this obligation. Agency CIO's are obligated to

ensure that website records, web pages, are adequately documented and

preserved.

Excerpt of Original Text from Policy Instrument:

a. Agency heads are required by 44 U.S.C. 3101 to "make and

preserve records containing adequate and proper

documentation of the organization, functions, policies,

decisions, procedures, and essential transactions of the agency.

. . ." NARA regulations at 36 CFR Part 1222 specify agency

record keeping responsibilities, including standards for record

keeping requirements. NARA regulations at 36 CFR Part 1234

Subpart C specify standards for managing the creation, use,

preservation, and disposition of electronic records.

2.2.9.2.2. NARA Bulletin 99-05. 1999. [Disposition of Electronic

Records (NARA Bulletin 98-02)]

Available at: <http://www.nara.gov/records/policy/b9905.html>.

Application to Federal Information Technology and Web

Management:

NARA Bulletin 99-05 informs Federal agencies that they should

continue to follow NARA Bulletin 98-02, and reminds agency heads as to

their responsibilities to maintain adequate documentation of records.

Although not specifically stated, Web pages should be considered Federal

agency records, and subject to the same guidelines as print records.
McClure, Sprehe & Eschenfelder 2-30 Federal Website Performance Measures

2.2.9.2.3.General Record Schedule 20 (GRS 20). 1995 version.

[Electronic Records Workgroup]

Available at: < http://ardor.nara.gov/grs/grs20.html>.

Application to Federal Information Technology and Web

Management:

General Records Schedule 20 (GRS 20) was first published in

1972 as an attempt to address record management for electronic materials.

Since then GRS 20 has been rewritten five times with the latest version

being in 1995. The 1995 version of GRS 20 has been through several

actions in court (see page ?), and as of March 6, 2000, it still stands.

The 1995 version of GRS 20 addresses the topic of disposal for

certain electronic records and the paper or microform records associated

with them. If a department desires to dispose of electronic records, which

are not covered by GRS 20, the department must file for authorization

using SF 115s, Request for Records Disposition Authority. Web pages

and documents associated with Web pages may be Federal records, and if

so, their disposal should be regulated by NARA.

2.3. JUDICIAL OPINIONS

An enormous body of case law exists interpreting certain policy instruments affecting

aspects of public access to government information, while other policy instruments have

spawned no case law. The following materials illustrate some courts’ approaches to finetuning statutory language, with their resulting potential effects on Federal information

technology management. A survey of all relevant case law is beyond the scope of this

chapter.

2.3.1. Freedom of Information Act (FOIA)

The cases speak to courts’ willingness to apply a liberal interpretation to statutes

governing public access based on a perceived statutory intent to promote open

government. Courts check this impulse where the duty to provide this information seems

onerous for the agency or when the information bears slight resemblance to that

identified in the statute.

Cases pertaining to:

Freedom of Information Act, Public Law 89-487, codified as amended at 5 U.S.C.

§ 552. Electronic Freedom of Information Act Amendments of 1996, Public Law

104-231, amending 5 U.S.C. §552.
McClure, Sprehe & Eschenfelder 2-31 Federal Website Performance Measures

1. Regulation promulgated by National Archives and Records Administration

(GRS 20) authorizing disposal of electronic government records without

distinguishing significant records for different treatment violated the Records

Disposal Act and interfered with plaintiff’s right, under EFOIA, to request

electronic records. Public Citizen v. Carlin, 2 F. Supp 2d 1 (D.D.C. 1997);

reversed by Public Citizen v. Carlin, 184 F.3d 900 (D.C. Cir. 1997) (agencies

expected to develop their own website records management policies within

existing NARA guidelines subject to NARA approval), available at

<http://www.cadc.uscourts.gov/common/opinions/199908/97-5356a.txt>;

certiorari denied by Public Citizen v. Carlin, 68 U.S.L.W. 3565, 2000 U.S.

LEXIS 1744, 2000 WL 240442 (March 6, 2000).

2. Federal agencies are not required to provide records concerning software over

which they lack “sufficient control”; software and technical documentation is

protected by FOIA exemption for trade secrets and commercial or financial

information. Gilmore v. U.S. Department of Energy, 4 F. Supp. 2d 912, 153

A.L.R. Fed. 759 (N.D. Cal.1998).

3. Federal agencies requested to search “all records” must search relevant

computer databases to comply with FOIA. Mayock v. Immigration and

Naturalization Service, 714 F. Supp. 1558 (N.D. Cal. 1989).

4. Computer backup tapes of electronic messages are records within the meaning

of FOIA. NARA must change its guidance to Federal agencies on record

management. Armstrong v. Executive Office of the President, 877 F. Supp. 690

(D.D.C. 1995); reversed by Armstrong v. Executive Office of the President, 90

F.3d 553 (D.C. Cir. 1996); certiorari denied by Armstrong v. Executive Office of

the President, 520 U.S. 1239 (1997).

Privacy Act of 1974, Public Law 93-579, Sec. 3, codified as amended at 5 U.S.C.

§ 552a.

1. Exemptions to FOIA under the Privacy Act of 1974 pertain to electronic

records. Manna v. United States Department of Justice, 1994 WL 808070 (D.N.J.

1994).

2.3.2. The Copyright Act of 1976

Case law interpreting the Copyright Act of 1976 is similarly abundant and similarly

limited as regards Federal information technology and electronic media. Aside from

disallowing copyright protection for government work, 17 U.S.C. § 105, the statute does

not directly address the interests and duties of the Federal government in its provisions.

The Digital Millennium Copyright Act amended the Copyright Act in 1998 to increase

protection for authors of work in electronic media.

Individual Case Laws pertaining to:
McClure, Sprehe & Eschenfelder 2-32 Federal Website Performance Measures

Copyright law is governed by Title 17 of the U.S. code as amended by The

Digital Millennium Copyright Act, Pub. L. 105-304, 112 Stat. 2860.

1. Compilations of facts are not protected by copyright under the “sweat-of-thebrow” doctrine. Feist Publications v. Rural Telephone Service Co., 499 U.S. 340

(1991).

2. Holders of copyright may submit protected work to government agencies

without fear of losing protection of copyright. Practice Management Information

Corporation v. American Medical Association, 121 F.3d 516 (9th Cir. 1997);

Veeck v. Southern Building Code Congress International, Inc., 49 F. Supp 2d 885,

(E.D. Tex. 1999).

2.3.4. Electronic Records

Since the Iran-Contra scandal in the late 1980s, electronic records have been the subject

of a series of Federal court cases. The cases arose originally because the White House

announced its intent to destroy the computer backup tapes containing the electronic mail

that led to discovery of the Iran-Contra materials. Public interest groups sued to prohibit

destruction on the grounds that the materials in question were federal records. In

response to judicial opinions, the National Archives and Records Administration issued

General Records Schedule 20, Electronic Records (see above). GRS-20 itself then

became the subject of further litigation.

A seminal case in the series of litigation was Public Citizen et al. v John Carlin et al.,

U.S. District Court, October 22, 1997. In his opinion, Judge Paul Friedman voided GRS20, stating that the Archivist of the United States had overstepped his authority in

promulgating GRS-20 and arguing, among other things, that electronic records had

unique value that was not captured with paper printouts. The U.S. Court of Appeals later

overturned Friedman’s opinion on August 6, 1999, and the Supreme Court denied

certiorari on March 6, 2000.

The opinions cited above are available, together with additional legal resources, at <

http://www.nara.gov/records/grs20/>.

2.5. CONCLUSION

Table 2.1 is a summary presentation of the foregoing elaboration of federal policies

affecting agency websites. For each topic the table shows the relevant statute, executive

order, or other key document, plus implementing policy guidance, if any. The right hand

column summarizes the implications for websites.
McClure, Sprehe & Eschenfelder 2-33 Federal Website Performance Measures

Table 2.1 Summary of Federal Policies Pertaining to Agency Websites

 Topic Statute, Presidential

Directive, or Other

Document

Implementing

Guidance

Website Implications

A. Performance

and Results

Government Performance

and Results Act

OMB Circular A-11,

Part II

Performance plans, goals, and

measures for agency programs

B. Customer

Service

E.O. 12862, Setting

Customer Service Standards

- Identify customers, their needs,

and set standards and

benchmarks

C. Accessible

Information

Technology

Rehabilitation Act, section

508

- Information technology

accessible to persons with

disabilities

I. General

Government

Policy

D. Electronic

Government

Pres. Memo on Electronic

Government

- Standardized access to and ease

of finding government

information, plus privacy and

security

A. National

Information

Infrastructure

NII Agenda for Action - Make govt. information more

easily and equitably accessible

B. Privacy and the

NII

Principles for Providing and

Using Personal Information

- Guidelines to personal

information users and providers

C. Copyright Digital Millennium

Copyright Act

- Protecting copyright in

electronic media

Freedom of Information Act - State FOIA procedures on

websites

Electronic Freedom of

Information Act

- Establish electronic reading

room on websites

OMB Circular A130, Appendix IHandling of personal

information

Pres. Memo on

Privacy and Personal

Information in

Federal Records

Review privacy policies and

practices; update notices of

systems of records

M-99-18 on Privacy

Policies on Federal

Websites

Display privacy policies on

websites

Privacy Act

M-00-13 on Privacy

Policies and Data

Collection on Federal

Websites

Discouragement of and

restrictions on use of “cookies”

on websites; comply with

COPPA

D. Rights of

Access to

Information

Public Printing and

Documents: Depository

Library Program

- Publications provided to

depository library program.

E. Paperwork

Reduction Act

Paperwork Reduction Act OMB Circular A-130 Framework for agency

information management plan,

including information

dissemination

F. Clinger-Cohen

Act

Information Technology

Management Reform Act

E.O. 13011, Federal

Information

Technology

Websites to be interoperable and

standardized across government

Proposed Revision of

OMB Circular A130, April 13, 2000

Mission based performance

measures for information

systems

G. Security Computer Security Act

OMB Circular A130, Appendix IIISecurity controls for federal

information systems

PDD 63, Protecting

America’ Critical

Infrastructures

Performance measures for

website security

II. Federal

Information

Policy

H. Electronic

Collection and

Digital Signatures

Government Paperwork

Elimination Act

OMB Notice:

Implementation of

the Government

Paperwork

Elimination Act

Increase and encourage

electronic data collection and

implement digital signatures
McClure, Sprehe & Eschenfelder 2-34 Federal Website Performance Measures

I. Federal Records Federal Records Act General Records

Schedule 20,

Electronic Records,

and various NARA

Bulletins

Provide for management of

records created on websites
McClure, Sprehe & Eschenfelder 2-35 Federal Website Performance Measures

Reference List

Kent, Allen, ed. 1999. Encyclopedia of Library and Information Science, Vol. 65, Supp.

28. New York: Marcel Dekker p. 306.

National Telecommunications and Information Infrastructure (NTIA). 1999. Falling

Through the Net: Defining the Digital Divide. Available at:

http://www.ntia.doc.gov/ntiahome/fttn99/.
McClure, Sprehe & Eschenfelder 3-1 Federal Website Performance Measures

CHAPTER 3

REVIEW OF AGENCY PRACTICES

This chapter provides an overview of evaluation and performance measures practices at

the Defense Technical Information Center (DTIC), the Energy Information

Administration (EIA), and the Government Printing Office (GPO) current as of July

2000. On March 15 and 16, 2000, the investigators interviewed agency personnel from

each of the participating federal agencies in site visits to the agencies. Each of these

agencies supplied documents in answer to the investigators' queries. The agencies also

reviewed two drafts of the chapter, providing comments the study team incorporated, and

the investigators met with all three agencies together on July 19, 2000. This chapter is

the result of the meetings and a compilation and analysis of the documents supplied.

3.0 BACKGROUND

The study team completed an initial overview and assessment of the evaluation methods,

statistics, and performance measures currently in use by GPO Access, DTIC, and EIA.

From each agency, the study team requested the following:

VI. Set of documents by which the agency evaluates its website material.

VII.What types of statistics and performance measures is the agency currently

using for its website?

VIII. What overall evaluation methodologies are currently in place

and how are they conducted/operated?

IX. Can we determine the types of reports generated by the agency evaluation

efforts and the success or quality of these reports?

X. To what degree are information policy instruments/guidelines (as per the

compilation in I) being considered in the documents and reports?

To the above questions, the investigators added:

VIII. Preliminary review of website

IX. Summary comments

The objectives of the review and assessment were to:

• Identify the basic evaluation methodologies, statistics, and performance measures

currently in use at each of the agencies

• Provide a composite view of the three agencies that would allow each to compare its

practices and techniques with others

• Consider which activities might constitute “best practices” for federal website

performance measures

• Establish a first step toward developing statistics and measures for federal website

performance measures.
McClure, Sprehe & Eschenfelder 3-2 Federal Website Performance Measures

The review and assessment described in this chapter are based primarily on the materials

that were provided the study team. Additionally, agency staff from each of the

participating agencies reviewed and commented on two earlier drafts of this chapter.

3.1. DEFENSE TECHNICAL INFORMATION CENTER

DTIC not only publishes its own corporate site but is also responsible for developing and

implementing approximately 100 websites for external clients throughout the DoD. The

performance measures discussed in this study report apply only to the DTIC corporate

site (http://www.dtic.mil).

3.1.1. Submitted Document Inventory as of May 6, 2000

1. Weekly Server Usage Report

2. Usage statistics on bandwidth utilization

3. Email forwarded from Marcie Stone, DTIC, forwarded from Terry Hendrix. Email

Attachments:

a) Online service totals

b) “Metrics and Evaluation Proposal” identifying GPRA instrument and schedule for

planning evaluation of agency’s effectiveness in disseminating information

c) online services summary

d) CENDI Agency Survey Questions for 29 February 2000

e) Survey for assessing DTIC’s STINET (Science and Technology Information

Network) service

f) Chart of queries by month (or year) by type

g) Bibliography of websites to use for refining “metrics,” website evaluation.

4. Diskette containing preceding attachments

3.1.2. What specific types of statistics and performance measures is the

agency currently using for its website?

1. Access Watch - <http://www.dtic.mil/usage/20000312/DTICALL_index.html>

a) Number of hosts

b) Percent within DTIC and percent outside DTIC

c) Variety of file types requested (html, gif, jpg, etc)

d) Number of errors

e) Number of megabytes of information passed to users

f) Average number of html pages/hour

g) Average number of html pages/day

h) Total number of html pages requested

i) Number and percent of html pages requested from within DTIC vs. number and

percent from outside

j) Total number of all files requested

k) Frequency of particular page requests

l) Number of times website accessed by users from particular domains, hosts, and

using particular browsers
McClure, Sprehe & Eschenfelder 3-3 Federal Website Performance Measures

2. Multi Router Traffic Grapher (MRTG) “DTIC 7206 CISCO routers Usage Data”

a) Fractional DS3 to GTE (15MBit)

b) T1 to Uunet

c) GW2 10MBit Ft. Belvoir ITN/NiprNet via DASC

d) GW1 10MBit Ft. Belvoir ITN/NiprNet via INSCOM

e) T1 to Pentagon

f) DTIC 100 MBit LAN on GW

g) DTIC 100 MBit LAN on GW2

h) GW1 BBN Router Summation

i) GW2 UUNet Router Summation

j) 1st Single Line T1 to BBN - turned off Feb99

k) 2nd Single Line T1 to BBN - turned off Feb99

l) Fractional DS3 to BBN 15MBit (serial 4/0)

ß Daily graph - 5 minute average

ß Weekly graph - 30 minute average

ß Monthly graph - 2 hour average

ß Yearly graph - 1 day average

m) T1 to UUNet (serial 2/0)

ß Daily graph - 5 minute average

ß Weekly graph - 30 minute average

ß Monthly graph - 2 hour average

ß Yearly graph - 1 day average

3. Uncertain source

a) “onlinetotjanwebdata.txt,” a file comparing usage between DTIC’s online services

(FTP/Listserv/WWW/fulcrum) and between internal and external users (possibly

a report generated through Access Watch)

b) “onlinesumjanwebdata.txt,” a file summarizing who (which hosts?) used DTIC

services and how many megabytes of information they accessed (that month?)

(possibly a report generated through Access Watch)

3.1.3. What overall evaluation methodologies (usability, log analysis,

customer surveys, etc.) are in place and how are they conducted?

1. Log analysis (“Access Watch” software, modified for the agency, which records

usage statistics)

2. Bandwidth usage (“Multi Router Traffic Grapher”)

3. Customer survey being conducted currently on Secure STINET

4. “Keynote” software package, which tracks DTIC’s average response time and

compares it to 40 well-known websites in metropolitan areas

3.1.4. Can we determine the types of reports that are generated by the

evaluation efforts and the “success” or quality of these reports?

1. Customer survey results are still being compiled. Preliminary results showed

overall user satisfaction.

2. Weekly reports are generated by software display usage statistics, and bandwidth

utilization.
McClure, Sprehe & Eschenfelder 3-4 Federal Website Performance Measures

3.1.5. To what degree are information policy instruments and guidelines

being considered in these documents and reports?

1. “Metrics and Evaluation Proposal,” refers to requests relating to the Government

Performance and Results Act (GPRA).

2. The same proposal refers to “CENDI” agencies, which may wield some influence

within the participating agencies.

3. The privacy and security statement on DTIC’s website refers to the NARA

General Records Schedule 20, alluding to EFOIA and Privacy Act.

3.1.6. Preliminary Website Evaluation based on McClure 2000 criteria.

ß Although some DTIC holdings are available to the public, DTIC primarily serves

federal agency employees, government contractors, potential government

contractors, and participants in certain federal programs. Information unavailable

to the public (i.e., available only to those registered for DTIC services) is

available from NTIS (National Technical Information Service). DTIC provides

the information necessary to obtain such information to people ineligible to

register for DTIC services. This limitation is not immediately obvious upon

reaching the site but is available one layer below under “About DTIC.” DTIC

could make this plainer so that members of the public who are ineligible for

registration could find the information more quickly.

ß The language on the introductory page is very technical, and possibly confusing

for the general public. Alternatively, DTIC might explain the site contents in

simpler language so that the general public may more quickly decide whether to

use the site as a resource.

ß Fifteen buttons point to important areas of the site, but DTIC might consider

either an organizing structure or decreasing the number of buttons.

ß A privacy and security statement appears on the introductory page

ß General statistics are collected and the agency states it destroys the records it

collects in accord with the NARA General Records Schedule 20.

ß The text spans much of the screen and is not separated into smaller units, making

it difficult to scan for relevant information.

ß The tone is consistently professional and objective.

ß Most pages have not been updated within the last three months

ß All pages include a link to a contact person.

ß The site lacks an overarching structure to place the contents in logical relation.

The site includes a lot of information, some of which is redundant.
McClure, Sprehe & Eschenfelder 3-5 Federal Website Performance Measures

ß The “Find It” function defaults into an alphabetical listing, frequently by

acronym. A user unfamiliar with the acronyms and other agency nomenclature

might have a difficult time finding a desired topic and, although the list links to

another glossary of acronyms, the process is awkward.

ß Each page contains a heading. As noted before, however, the subject headings are

confusing. They do not suggest a controlled, logical hierarchy. The relationships

between the content areas are vague - it is unclear from the structure whether one

contains part of another or is preliminary to another.

ß From the materials it appears that statistics on website use are available; however,

the statistics and website evaluation pages are difficult to find.

3.1.7. Summary Comments

The DTIC website acts as a portal for the public to other sites designed for fuller public

use. Further, it provides fuller service to users who are technologically sophisticated and

familiar with the agency’s structure, context, and background.

In the past, DTIC has relied on technical self-evaluation sources such as log file analysis,

bandwidth usage and tracked web site response time. But recent efforts have also

included user based evaluation techniques such as the Secure STINET customer

satisfaction survey. DTIC is also involved in a larger CENDI Agency working group

effort to incorporate greater use of web evaluation tools.

Future DTIC evaluation efforts might include some of the following:

ß Continued and expanded use of user focused evaluation methods such as focus

group testing, usability testing, and collection and analysis of user comments.

User based evaluation techniques could be used to develop a more user-centric

organizing framework for DTIC site.

ß Design guidelines to control the use of technical language and acronyms without

accompanying explanations

ß Clearer and easier-to-find information about the restrictions faced by the general

public on the site, and possible alternative information sources. Currently this

information is available, but DTIC may consider making it more obvious to the

user.
McClure, Sprehe & Eschenfelder 3-6 Federal Website Performance Measures

3.2. ENERGY INFORMATION ADMINISTRATION

3.2.1. Submitted Document Inventory as of May 6, 2000

1. EIA help desk notes, customer comments, raw data

a) Nov99.htm

b) Dec99.htm

c) Jan00.htm

d) Feb00.htm

e) Feedback.wpd

2. Materials related to cognitive analysis of the EIA website

a) Interview protocol and questions for cognitive analysis

b) Paper providing background and details of evaluation project, methodology

used in evaluation.

c) Specific problems identified during Web evaluation, covers five groups of

websites. Bulleted list focuses on problems experienced.

d) Usability study conducted in the Bureau of Labor Statistics Lab. Analysis

of Results from EIA Web evaluation: Second Pre-test. Includes

demographics of test subjects, data, results and analysis.

e) Power point Presentation of Web Cognitive and Usability Evaluation

Status Report. Outlines goals and objectives of the site. Presents results,

conclusions and findings.

3. EIA Webtrends snapshot from March 20, 2000, user statistics, raw

4. Telephone surveys

a) 1998 Survey of telephone customers and printed publication subscribers

b) 2000 Survey of telephone customer satisfaction

5. Web products profile; number of visits and time of stay in minutes

6. Sample of Altavista queries to the EIA site, from March 20 2000, raw data

7. Sample of Google queries to the EIA site, raw data

8. Information on Channeling

a) Spreadsheet of web user session data and web file downloads.

b) Information channels handout: used for strategic planning purposes.

c) IPSC (Information Products and Services Committee) Report to Senior

Management – EIA Strategic Plan; provides implementation plan for

channels concept

9. Quarterly Report on EIA Performance Measures: deals with citations and data

integrity and quality issues. Lists results of performance measures that are

described in document 10, Draft ICSP (Interagency Council on Statistical Policy)

Statistical Agency Performance (See Performance Measures Section II, IV-9,

VIII-1)

10. Draft ICSP Statistical Agency Performance: Draft covers 12 agencies and EIA

cover sheet provides good reference point.

3.2.2. What specific types of statistics and performance measures is EIA

currently using for its website?
McClure, Sprehe & Eschenfelder 3-7 Federal Website Performance Measures

1. Complete Report: Log Analysis Statistics from WebTrends Software Package

General statistics:

a) No. of Hits per Home page

b) No. of User Sessions

c) No. of Hits per day

d) Average session length

e) Most requested pages-most popular website pages and how often they were

accessed

f) Top entry/exit pages-first/last page viewed when entering/leaving site

g) Most downloaded files-most popular file downloads of website

h) Activity level by day of week

i) Most downloaded file types-identifies the downloaded file types and total

kilobytes downloaded for each file type

j) Most accessed directories-analyzes accesses to the directories to the site

k) Top referring sites-identifies the domain names or number of IP addresses

w/links to site

l) Top search engines-which search engine referred visitors to site the most

often

m) Top search keywords-identifies, for each keyword, which search engines

led visitors to site

n) Most used browsers-identifies the most popular WWW browsers used by

visitors to site

o) Netscape browsers-breakdown of the various versions of Netscape

browsers that visitors use

p) Microsoft Explorer browsers-breakdown of the various versions of

Explorer browsers visitors use

q) Visiting spiders-identifies all robots, spiders, crawlers and search services

visiting site

r) Most used platforms -identifies operating systems most used by visitors to

site

s) Single access pages-pages that visitors access and exit without viewing any

other page

t) Most active organizations-companies or organizations that access website

the most often

u) Summary of activity by day-outlines general server activity, comparing

level of activity on weekdays and weekends

v) Activity level by day of week-shows the activity for each day of the week

by report period

w) Activity level by hour of day-shows the most/least active hour of the day

for the report period

x) Technical statistics and analysis-total number of hits for the site, how many

successful/failed, and calculates the percentage of hits that failed

y) Client errors-type of errors returned by the client accessing the server

z) Server errors-identifies by type the errors which occurred on the server

aa) Top referring URLs-provides the full URLs of the sites with links to

website
McClure, Sprehe & Eschenfelder 3-8 Federal Website Performance Measures

2. Complete Report: Log Analysis

Performance Measures from Quarterly Report (See Section IV)

a) Media Citations

b) Web use

c) Briefings to Congress and policymakers

d) Data accuracy

e) Surveys of subscribers

f) Telephone Customer Satisfaction Survey

3.2.3. What overall evaluation methodologies are in place and how are they

conducted/operated?

1. Web Cognitive and Usability Evaluation

a) Think Aloud Interviews

b) Customer feedback

c) Demographic Questionnaire

d) Timed exercises

e) Senior Staff Tested

2. Customer Comments/Surveys

a) Telephone customer satisfaction survey

ß Frequency of contact

ß Questions asked via telephone

ß Satisfaction w/customer service and information quality

ß Product users who used EIA Electronic Products or Services during

the Past Year

ß Preferences between paper vs. electronic copy of information

ß Reasons for Calling National Energy Information Center (NEIC)

ß Comments from Customers

° Timeliness

° Data access on the website

° Improving website

b) Feedback observations for February 2000

3. Emailed Customer Comments (Nov 1999 – Feb 2000)

a) Website Feedback

ß November 1999 – Received 4 comments/contents included

ß December 1999 – Received 1 comment/contents included

ß January 2000 – Received 2 comments/content

ß February 2000 – Received 2 comments/content

ß Webmaster mail – comments received by webmaster

ß November 1999 – Received 21 comments/content

ß December 1999 – Received 13 comments/content

ß January 2000 – Received 22 comments/content

ß February 2000 – Received 24 comments/content

b) Comments prepared by NEIC staff – Staff prepare summary reports based on

customer input. The summary reports include some comments verbatim.

ß November 1999 – Included 10 comments/content
McClure, Sprehe & Eschenfelder 3-9 Federal Website Performance Measures

ß December 1999 – Included 8 comments/content

ß January 2000 – Included 10 comments/content

ß February 2000 – Included 48 comments/content

3.2.4. Can we determine the types of reports that are generated by the evaluation

efforts and the “success or quality” of these reports?

1. Blessing, Colleen; Bradsher-Fredrick, Howard et.al. Cognitive Interviewing:

Applications to Evaluating the Energy Information Administration’s Web Site.

2. Finn, Pauline. Analysis of Results from EIA Web Evaluation: Second Pre-Test.

Usability testing results based on small scale, informal studies.

3. Blessing, Colleen, et.al. Web Cognitive and Usability Evaluation Status Report.

Power Point presentation of report No.4

4. 1998 Survey of Subscribers.

• 56 percent of respondents prefer paper publication to Web version,

38 percent prefer Web version

• Reasons Web version preferred: search capabilities; timely

information; easier to transfer data to others; don’t need bookshelves

for storage; cheaper than subscribing.

• Reasons Print version preferred: don’t like reading large documents

on the PC; use paper publications when away from the PC; like

having paper archives; takes too long to download; difficult to

download; difficult to navigate; Internet access unreliable.

5. 2000 EIA/NEIC Telephone Customer Satisfaction Survey: Findings concerning

electronic accessibility of information.

6. IPSC Report to Senior Management – EIA Strategic Plan

• Implementation of channel plan.

8. Performance Measures Quarterly Report

• Measures address citations, congressional briefings and data quality.

3.2.5. To what degree are information policy instruments and guidelines (as

per the compilation) considered in the documents and reports submitted by

EIA?

April 27, 2000, Draft Definitions Of Measures, A Supplement To Performance

Measures. On page 2, Paperwork Reduction Act of 1995 directs agencies to

achieve reductions in respondent burden of 5 percent per year.

3.2.6.Preliminary Website Evaluation based on McClure 2000 criteria

1. Performance measures determining the successful use of the search feature on the

website.

• Site uses commercial search engine – Google.com

• Searches lead to other pages on site; however, a button pointing to

the EIA home page (i.e. “return to EIA home page”) could be
McClure, Sprehe & Eschenfelder 3-10 Federal Website Performance Measures

included on those pages to insure the user easy navigation back to

the home page

• The “back” button is available but after numerous search strings, it

may be difficult to use “back” button to get back to home page

2. Does the website have evaluation information and if so what type and how much?

• There is no evaluation online available to the general public.

3. There is contact information for customer feedback:

a An email address, street address, and phone number is made available

to people who may have questions about the content of the website.

infoctr@eia.doe.gov

b An email address and phone number for the webmaster is made

available to people who have technical site problems.

wmaster@eia.doe.gov

c A feedback button on homepage for sending messages

d Contact list of personnel at EIA

e Home page has link to privacy policy

f Primary and Secondary pages have consistent format and design.

Pages deeper in the structure do not make it clear that pages are

official parts of EIA website or if user has left the site.

g Some of the deeper pages take too long to download

h Each page lists a modification date.

i Site contains no pricing information, as EIA does not sell anything.

j FAQ section

4. Privacy Statement

a Contains references to FOIA

b Explains security practices in place to protect the integrity of the site,

and detection of individuals trying to hack the site

c Explains the kinds of information collected and how the information is

used.

3.2.7. Summary Overview:

The task faced by EIA in putting its information on the Web is an organizational

conundrum. The EIA web committee has the unenviable assignment of posting highly

diverse technical information from many sources on one website suitable for use by the

layperson. To this end, and to their credit, the EIA Web committee has recognized the

need for continual evaluation and redesign of the site in reaction to a constantly changing

environment. They have conducted studies to aid in the creation of the next version of

their home page to be unveiled in the near future. The new design takes a channel

approach, strives for less jargon, and is designed with the layperson in mind.

To prepare for this new design, EIA chose to perform cognitive usability testing. The

results from this usability testing suggest that EIA’s current web site design meets key

usability requirements as most users could find the required information. Tests showed

no correlations among the following variables:
McClure, Sprehe & Eschenfelder 3-11 Federal Website Performance Measures

• The users’ level of experience using the Internet and their ability to find the

correct answer to a test question on the web site

• The users’ familiarity with energy terms and their ability to find the correct

answer

• The degree of frequency of EIA website use and ability to find the correct answer.

As part of the redesign, EIA may wish to develop basic website design guidelines

applicable to all EIA sites. EIA will doubtless want its design guidelines to avoid

stunting office or program creativity or limiting office and program ability to respond to

changing customer needs. How the guidelines are applied is a matter for agency

determination.

EIA already has performance measures in place, but these performance measures have

not been completely modified to include website services and products. For example,

EIA has output measures in place for the print environment, but not for the electronic

environment. Current website performance measures focus on number of downloads

from website and number of user sessions.

To expand its toolbox of performance measures the study team suggests the development

of performance measures based on the following:

• Identify the existing business processes on the Web, in order to develop

benchmarks for evaluating these processes.

• Documents supplied to the study team suggest that EIA could develop

performance measures for user satisfaction. Possible methodologies include an

online survey similar to GPO’s, or a survey to electronic newsletter participants.

• Testing by the study team also suggests the need for performance measures

related to bandwidth and page load times as some pages were slow to load at both

peak and non peak usage times.

• Further analysis of customer comments. EIA summarizes and distributes all

customer comments. Further analysis might include classifying comments as

positive or negative with the goal to reduce negative comments, or keeping track

of the number of comments that led to web site changes with the goal of

increasing customer suggested changes.

• Number of customer queries receiving responses within a given time period. The

support team receives about 30,000 customer contacts each year, including

telephone, web, mail, and walk-in. The team tries to respond to email contacts

within 24 hours and answer telephone inquiries immediately. EIA does not

currently keep data about whether all queries receive responses.

EXISTING REFERENCE MATERIALS LISTED BUT NOT IN STUDY TEAM’S POSSESSION

1. Bradsher-Fredrick, Howard, and Rutchik, Robert. Maximizing Feedback to

Increase Customer Use of the Energy Information Administration World Wide

Web Site. (Cited in EIA documents but not provided to study team)
McClure, Sprehe & Eschenfelder 3-12 Federal Website Performance Measures

2. Bradsher-Fredrick, Howard, and Rutchik, Robert. Customer Use of the Energy

Information Electronic Products During 1996 (With hints of what to expect in

1997). (Cited in EIA documents but not provided to study team)

Agency did preliminary focus group testing, but provided study team no results from

actual tests.

3.3. GOVERNMENT PRINTING OFFICE

3.3.1. Submitted Document Inventory as of April 8, 2000

1. GPO Access Web Committee hierarchy

2. Video Tape of GPO Access Focus Group

3. Training Course and Trade Show Evaluations and Evaluation Form

4. Sampling of Bandwidth Utilization Reports for 9.12 - 9.19, 12.26 - 1.02,

1.16 - 1.23, 1.23 - 1.30

5. Sample of user comments emailed to web committee

6. Draft website guidelines for GPO Access

7. Managing the FDLP Electronic Collection: A Policy Planning Document

8. Performance Measures: Talking Points

9. Update Report: The Indexing of GPO Access Web Pages

10. GPO Access Usage: Executive Summary: List of data showing number

of searches for databases and the number of retrievals for those searches: 1/2000,

2/2000

11. Web Trends: GPO Access usage, prepared by EIDS: 1/2000, 2/2000

12. Biennial Report to Congress on the Status of GPO Access 12/31/99

13. GPO Access User Survey Findings of April 18, 2000

14. GPO Access Open Forum Report, Spring Depository Library Council

Meeting Newport, RI, Tuesday April 11, 2000

15. GPO Summary of Search Engines Presentation, Spring Depository

Library Council Meeting Newport, RI, Tuesday April 12, 2000

16. GPO Access Open Forum Feedback, Federal Depository Library Spring

Conference, Wednesday April 14, 1999

3.3.2.What types of statistics and performance measures is GPO currently using for

its website?

1. Document Retrieval Statistics: In order to measure the units of content

retrieved by users from GPO Access, GPO decided to use documents

retrieval. This allows the agency to measure both the units of actual content,

or documents retrieved, as static html pages and those extracted from

databases made available on GPO Access. Since this eliminates the noncontent retrievals for items such as graphics on a web page, GPO believes it is

far more meaningful than the ubiquitous hit counts employed by many

websites.

2. Sampling bandwidth utilization: GPO keeps descriptors of server bandwidth

utilization.
McClure, Sprehe & Eschenfelder 3-13 Federal Website Performance Measures

3. Electronic User Comments: Electronic messages originate from the website

either through the website’s “Questions or Comments: gpoaccess@gpo.gov”

link. Further, GPO uses an Automatic Call Distribution (ACD) System and

Software, which provides real-time and historical data for statistical

measuring of performance of GPO Access User Support Team’s telephone

customer service performance.

• Electronic messages from comments indicate the page the request

originated from, the client’s browser; client’s name, client's phone

number; and the content of the comment.

• Electronic help or support comments only include name, email address,

date, and message.

Note, too, that GPO responds to all email and fax inquiries within 24 hours,

either with an answer (when possible), or with an acknowledgement of

message receipt and follow-up answer. GPO also plans to measures to track

user comments and to analyze their impact on changes made to the site.

4. Internet search engine retrieval statistics: GPO Access Usage - Executive

Summary: GPO staff believe that the ease with which users access the website

is important, and the ability to find the website using commercial search

engines is an important factor in accessibility. Therefore, GPO has done

studies using commercial search engines to see if GPO Access pages are

covered by commercial search engines, and what terms provide the best

results for retrieval of GPO Access pages through these search engines.

5. Biennial Report:

a) Online Tangible Titles Available through GPO Fiscal Year 1999

b) Cumulative Online Titles Available through GPO Access: April 1998-

November 1999

c) GPO Access user support team requests: March 1997 - October 1999

d) Expenditures and Operating Costs, Savings and Revenues, p.24.

6. Statistics generated by the Software Package Webtrends

a) Server Errors: internal server errors/server.

b) Page not found Errors: resident server 404 errors

c) Client Errors: error codes from servers accessing site

d) Dynamic Pages and Forms Errors

e) Technical Statistics and Analysis

• Total hits

• Successful hits

• Failed hits

• Failed hits as percent

• Cached hits

• Cached hits as percent

f) Activity level by hour of the day: bandwidth usage by hour of the day

g) Activity level by day of the week: bandwidth usage by day of the week

h) Summary of activity by time increment: bandwidth usage across all servers by

date.

i) Summary of activity by report period: allows comparison between weekday

and weekend activity.
McClure, Sprehe & Eschenfelder 3-14 Federal Website Performance Measures

j) Hits, and user sessions per organization tag; net, org, mil, com, edu, gov,

arpnet, international

k) Most active organizations: AOL, UUNET, att.net

l) Most active countries: identifies countries from which visitors originate and

which server they enter through.

m) Top users: Identifies the domain names from which most requests come from.

n) Most popular dynamic pages and forms.

o) Most downloaded file types: Most frequently downloaded type of file. Ex gif,

html, htm, shtml, pdf, jpg, txt, etc.

p) Most downloaded files: identifies most popular file downloads, does not count

if transfer was not completed and does not include files downloaded from a

database search. See Section II, number 1 for information on GPO document

download counts.

q) Top paths through sites: Identifies entry page and next six pages visited.

r) Single access pages: identifies the pages from which a user enters and exits

from without going to other pages in the site.

s) Top exit pages: the pages from which users exited the site.

t) Top entry requests: Identifies first page linked to from within the site.

u) Top entry pages: First page viewed when a user visits the site. Homepage and

links that connect with site.

v) Least requested pages: Least popular pages on website. If committee thinks

these pages are important, they need to increase their visibility.

w) Most requested pages: most popular websites based on access frequencies.

Also includes average time a user spends viewing the page.

x) Server cluster load balance: compares the performance of the individual web

servers (five servers) in a cluster based on kilobytes transferred (bandwidth

load balance)

y) General Statistics

• Number of successful hits for entire site

• Number of page views

• Number of user sessions

• User sessions from U.S.

• International user sessions

• User sessions of unknown origin

• Average number of hits per day

• Average number of page views per day

• Average number of user sessions per day

• Average user session length

• Number of unique users

• Number of users who visited once

• Number of users who visited more than once

Note that GPO does not use all of the measures produced by WebTrends reports. Instead

GPO focuses mainly on (1) activity level by hour of the day, (2) most requested pages,

and (3) server cluster load balance. GPO places greater emphasis on document download
McClure, Sprehe & Eschenfelder 3-15 Federal Website Performance Measures

statistics. They are more accurate measures of web site usage given the prominence of

the document databases on GPO Access.

3.3.3.What overall evaluation methodologies are in place and how are they

conducted/operated?

1. Focus Groups. GPO uses focus groups to gather the opinions of specific

communities about specific aspects of the GPO site. Following is a brief

outline of a group session assembled specifically for feedback from the

Federal Depository Library community, and therefore not necessarily

representative of all focus groups conducted by the GPO:

Participants: uncompensated librarians

Topics covered included, but were not limited to:

Initial impressions, what specifically do people like?

Ease of use

Navigation tools

2. GPO monitors user feedback through

• Telephone 4,350 per month (Biennial Report 1999 p.20)

• Email 2,150 per month (Biennial Report 1999 p.20)

• Faxes

• Letters

3. Online user surveys: GPO has conducted three user surveys, the latest of

October 1999. Each survey iteration has changed slightly as changes occurred to

the system. To better keep pace with these changes, GPO intends to conduct

annual surveys from this point on. The most recent online survey focused on the

following:

• Key characteristics of GPO Access users: affiliation with libraries,

schools, private industry, government etc.

• How users find GPO Access: internet search, following links

• Frequency of site usage: yearly, weekly etc.

• Reasons for using GPO Access: tracking legislation, legal research,

patron assistance, business decisions, homework, news, academic

research etc.

• Finding aids employed: MOCAT, Federal Agency Internet Sites,

Government Internet Sites by Topic etc.

• Ease of use of the site

• Content timeliness

• Technical specifications of the computers and internet connections

used to access GPO Access: platform, monitor size, connection speed

• The survey also asked users to rate GPO user support teams in terms

of: accessibility, courtesy, product knowledge, technical knowledge,

complaint resolution and response time

4. Open public forums:

• Open attendance (e.g. Search Engines Presentation and GPO

Access Open Forum at the Spring Depository Library Council

Meeting)
McClure, Sprehe & Eschenfelder 3-16 Federal Website Performance Measures

o Forums on several topic areas were held during this

meeting.

• Open, unguided discussion of GPO Access

• Forum announced in the Federal Register and on the GPO Access

Web site in advance of each session.

3.3.4.Can we determine the types of reports that are generated by the

evaluation efforts and the “success or quality” of these reports?

1. Document Retrieval reports “GPO Access Usage report.” As described in

Section II, number 1, GPO uses documents retrievals, as its primary measure of

web site usage. The reports show the number of documents, or units or content,

that users downloaded from GPO Access.

2. GPO Access Online User Survey reports. As described in the above section,

GPO has conducted three on-line user surveys, each resulting in a User Survey

Findings Report.

3. Update Report: The Indexing of GPO Access Web Pages

This GPO report analyzes the ability to retrieve GPO Access Web pages through

Internet search engines.

4. Biennial Report to Congress on the Status of GPO Access This is the fourth

of a series of reports produced every odd year. Most recent -12/31/99

 5. Performance Measures Talking Points. Provides a list of GPO Access

evaluation goals which GPO is currently working toward achieving

 6. Log Analysis Report uses Webtrends Software package.

3.3.5.To what degree are information policy instruments and guidelines (as

per the compilation) considered in the documents and reports

submitted by GPO?

1. Draft of website design guidelines for GPO Access,

a) “Guidelines do not apply directly to the World Wide Web Consortium

(W3C) Americans with Disabilities Act guidelines

b) “In accordance with Public Law 103-40, The Government Printing Office

Electronic Information Enhancement Act of 1993, GPO Access was

established as a means of enhancing public access to federal electronic

information. To achieve this objective, GPO has attempted to ensure

equitable access to public, including those persons with disabilities in

accordance with standards established by the Americans With Disabilities

Act (ADA) of 1990.” p.2

c) Whenever a significant change is made to the site, take and save a “snapshot” of GPO Access:

d) Mentions archive, permanent information, NARA. Does not mention

instruments per se, but implies NARA and EFOIA. P.4

2. Biennial Report

a) Public Law 103-40, the U.S. Government Printing Office Electronic

Information Enhancement Act of 1993, which charged the Superintendent
McClure, Sprehe & Eschenfelder 3-17 Federal Website Performance Measures

of Documents within the Government Printing Office to develop

mechanisms to enhance public access to a wide range of Federal electronic

information products. p.1

b) The Senate Report [103-27] incorporated the Federal Bulletin Board into

GPO Access p.5.

c) Title 44 of the U.S. Code (44 USC 1911) stipulates that public access to

official government information products disseminated through the FDLP

must be maintained permanently in regional depository libraries and

depository libraries not served by a regional depository library. Since

online products are not physically distributed to depository libraries for

retention, GPO has assumed responsibility for the provision of permanent

access to government information products residing on GPO Accessp.8.

d) Refers to meeting the privacy needs of users, but was instituted before

requirement by legislative action. p.15.

e) The Privacy and Security Notice on GPO Access was posted prior to

legislative actions that would require all Federal Web sites to provide such

notices.

f) Following standards set forth in OMB bulletin 95-01 and the National

Archives and Records Administration (NARA), in January 1996 GPO

implemented its Government Information Locator Service (GILS) online.

3. Managing the FDLP Electronic Collection: A Policy and Planning

Document

In several instances this document cites existing laws regarding the scope of

GPO and the Depository Library program as well as collection development

and ADA.

a) Criteria for inclusion of the material into the collection are determined by

44 U.S.C. §1901-1902 (p. 10, 13)”

b) “GPO will provide a text-only interface for government information

products made available through GPO servers whenever technologically

feasible and cost-effective. This will not only allow GPO to maintain

compliance with the Americans with Disabilities Act (ADA) but also will

ensure access to users with limited technological and communications

capability.” (p.13)

c) “...For such electronic products, GPO has assumed the role traditionally

taken by regional depository libraries as provided in 44 U.S.C. §1911.”

(p.9)

d) “except those determined by their issuing components to be required for

official use only or for strictly administrative or operational purposes

which have no public interest or educational value and publications

classified for reasons of national security.” (44 U.S.C. §1902) (p.9)

e) “ Based on the mandate of 44 U.S.C. chapters 19 and 41, GPO’s

responsibility for providing permanent public access is extended to

electronic government information products that meet the statutory

definition for government publications.” (p.7)
McClure, Sprehe & Eschenfelder 3-18 Federal Website Performance Measures

f) “Senate report 103-27 incorporated the Federal Bulletin Board into GPO

access as well.”(p.6)

g) “FDLP is codified by 44 U.S.C. Chapter 19. Additional detail concerning

the Cataloging and Indexing is codified at 44 U.S.C. §1710-1711.” (p.5)

h) “The Government Printing Office Electronic Information Access

Enhancement Act of 1993 (Public Law 103-40) charged the

Superintendent of Documents....” (p.5)

3. Electronic Transition Plan.

GPO considers Permanent Public Access to be an important performance

measure for Federal Agency Websites. While GPO Access offers the

immediacy that the public demands from the Internet, it differs from most

online services in that it also maintains historical archives of information

previously available on its servers. Permanent public access to a collection

of electronic government information products is thus ensured by the

system's storage facilities, allowing the same historical research that is

possible through printed media.

3.3.6. Preliminary Review of website

• Official pages in the first two layers of the site carry the GPO Access seal.

Third level pages carry the words “GPO Access” from the seal. Currently the

only section that does not have the seal is the children’s area.

• When leaving the official GPO site, the user is notified

• Few images on site, pages download relatively quickly

• Text only version available

• Format consistent from page to page

• Children’s area clearly marked, downloads a little slower due to more

graphics

• Has a “What’s New” section

• Privacy policy link posted on each page

• Navigation tools provided, quick links, site map

• Last update date provided

• No user statistics posted on the site

• No access fees

• Several ways to reach support services

a) Link at top of page called comments: this provides form to fill in

b) Help link at top of page, takes user to page with contact information;

phone, fax, email - gpoaccess@gpo.gov

c) gpoaccess@gpo.gov also at bottom of each page

• Provides information about this site link

3.3.7.Summary Comments

Initially, GPO established the GPO Access website to fulfill the mandate required by PL

103-40. Under the provisions of this law it has subsequently evolved to provide public

access to many key government documents. “As of November 1999, the website
McClure, Sprehe & Eschenfelder 3-19 Federal Website Performance Measures

provided access to more than 104,000 titles on Government Printing Office servers and

over 62,000 additional titles through links to other federal agency websites.” (Biennial

Report 1999 p. 1). GPO considers Permanent Public Access to be an important

performance measure for federal agency websites. GPO Access maintains historical

archives of information previously available on its servers, ensuring permanent public

access to its collection of electronic government information products, and facilitating the

same historical research that is possible through printed media.

GPO Access is geared toward meeting the changing needs of the general user

community. As a content oriented web site, GPO Access primarily measures the number

of documents retrieved by users as a selection of the units of content being distributed by

the site. Standard page hit statistics do not capture information about the number of files

downloaded through a search of the GPO Access databases, which serve as the main

focus on GPO Access and users’ main source of information. Document retrieval

statistics more accurately show how many of those documents users accessed and to what

extent users are making use of the resources on GPO Access. In addition to its use of

document retrieval statistics, staff constantly evaluate other aspects of the website

through focus groups, online user surveys, user comments, log analysis, trade shows and

conferences. Further, GPO conducts user-training sessions.

GPO has ensured that the information on its website is listed in commercial search

engines by analyzing search results from commercial search engines, and conducting

search and retrieval studies using commercial search engines. GPO has contacted the

commercial search engines to eliminate problems with missing pages, and changed their

hardware configuration to allow spiders - indexing tools used by commercial search

engines to add sites to their searchable database - to work better. Also, GPO has added

keywords and descriptive meta-tags to improve their pages' rankings in retrieval lists for

engines that take advantage of these tags.

In addition, GPO has conducted bandwidth usage studies to determine peak user times

and peak usage pages. They have also changed their hardware to allow equivalent

distribution of bandwidth among their servers.

GPO access has a privacy statement posted on its homepage and on its children’s area,

Ben’s Guide to Government. The GPO access website maintains design and format

consistency throughout, except for search pages and search result pages.

The GPO website has addressed Universal Accessibility in its Website Design

Guidelines. These guidelines enable GPO Access to comply with section 508 of the

Rehabilitation Act Amendments of 1998 (36 CFR Part 1194). They include providing

the information in ASCII text format, using only HTML code, minimizing the use of

images, provide text alternatives for graphics, and test all pages using a range of

browsers.

Future steps to enhance current evaluation activities and web site design might include

the following:
McClure, Sprehe & Eschenfelder 3-20 Federal Website Performance Measures

• GPO could enhance evaluation by keeping more exact, categorized figures of user

comments and customer service responses. GPO could set benchmarks for

adequate response time, and information usefulness to customers. That is, how

often are questions answered promptly, adequately and correctly?

• GPO could enhance the value of training surveys by adding questions pertaining

to the website to the training questionnaire.

GPO has a well developed management approach for its website and has committed

significant effort and thought to evaluating it.

A recent presentation given by T.C. Evans offered the revised strategy for assessing GPO

Access (See Appendix B). This strategy could serve as a guideline for other agencies in

establishing or amending their website development strategies. GPO’s strategy is based

on nine areas for performance measurements:

• Legal Mandates: Information Policy Instruments

• Agency Goals and Objectives

• Technological Infrastructure

• Site Content

• Site Design: Consistency

• User Support

• Awareness

• Usage Metrics

• Customer Satisfaction.

The outline referenced in Appendix B is an excellent approach and these nine

measurement areas should be considered in developing a website evaluation and

measurement strategy. Although all nine areas may not be of equal importance for

different agencies in ensuring the citizen access to federal government information, they

are an excellent starting point.

3.4. DIVERSITY AND CHALLENGES

The overview provided in this chapter of the evaluation and measurement efforts at

DTIC, EIA, and GPO demonstrates the wide range of strategies for measurement. The

efforts also suggest that different strategies can be useful in different agency settings. For

example, some agencies may be better able to employ regular and automatic log analysis

techniques than to implement full-scale usability studies. Others may find that the

establishment of formal usability labs to “test” the effectiveness of selected pages and

content may be more effective. Ultimately, the “best” strategies may be those that best

accommodate agency mission and available resources.

The review of agency practices described here may also be useful for other federal

agencies to compare themselves against. As suggested above, it is not that one agency is

better or worse than another; rather, that these agencies have taken different approaches
McClure, Sprehe & Eschenfelder 3-21 Federal Website Performance Measures

to developing evaluation and measurement techniques. A range of issues and situational

factors (see Chapter 5) affect the success with which any agency can engage in an

ongoing program of evaluation and measurement.

Each of these three agencies is in the process of developing and evolving their assessment

techniques. Within this process each has struggled with a comprehensive planning

approach to formally establish responsibilities, tasking, schedules, statistics, and

measures to be employed to assess the quality of agency websites and their usefulness to

users of that website.

A common comment heard throughout the federal government is the lack of time,

resources, and personnel to engage in ongoing assessment of websites. While this

certainly has been true based on the investigator’s experience, agencies will still need to

develop “quick and clean” assessment techniques – utilizing not only the techniques and

measures identified in this chapter but also those proposed in the next.
McClure, Sprehe & Eschenfelder 4-1 Federal Website Performance Measures

CHAPTER 4

PROPOSED PERFORMANCE MEASURES

4.0 INTRODUCTION

This chapter describes and elaborates on the findings from the assessment of the

materials collected and reported in Chapter 3 from the three sponsoring agencies

concerning the types of statistics and performance measures the agencies were using.

Appendix A, Criteria for the Evaluation of Federal Websites, and Appendix B, Website

Performance Measurement and Evaluation for GPO Access, supplement the information

in this chapter about performance measures.

In interviews and meetings with agency representatives involved in this study, the

investigators received a wealth of information and suggestions. These materials pertain

to how the agencies are currently measuring their websites’ performance as well as

management steps the agencies are taking – or plan to take – to better manage their sites.

In addition, the policy analysis report in Chapter 2 provides a basis for specific

recommendations about policy based assessment that federal agency staff should consider

when assessing their websites.

The chapter presents a way of organizing and understanding website performance

measures. As the investigators sifted through the data collected from the agencies and

from their own analyses, they realized that the term “performance measure” can be

understood in several ways. Many things lumped under the general heading of

performance measures might be considered conditions prerequisite to performance

measures. In other cases, the materials pertained to management or operational

considerations rather than measurement of the website’s performance. Performance

measures necessarily imply performance goals and the measures yield indicators of

progress toward achieving the goals. Websites can be the subject of many measures and

management activities that are not goal achievement indicators. This point should

become clearer from the exposition below.

4.1 LEGAL AND POLICY CONDITIONS AFFECTING AGENCY WEBSITES

Table 4.1 presents a set of questions or possible measures based on Chapter 2 that the

investigators consider as prerequisites for performance measures. These are questions

that all federal webmasters should ask of their websites, irrespective of the specific

mission served by the agency. That is, they apply to all websites simply because they are

federal websites and not specifically because they are the site of Agency X or Agency Y.

In effect, the list in Table 4.1 constitutes a set of performance measures of the “Yes-No”

variety. The “Yes-No” questions are only a beginning point for an agency in considering

legal and regulatory conditions. They represent a level of minimal compliance, but do

not necessarily represent “best practice.” For best practices, agencies need to ask
McClure, Sprehe & Eschenfelder 4-2 Federal Website Performance Measures

themselves about the quality of their response to the conditions. For example, by now

every federal agency is likely to have some sort of privacy policy notice on its websites.

Not every agency, however, has a privacy policy notice that complies with the OMB

guidance and conformss with the model language OMB provided. Again, most federal

websites have at least token compliance with the statutory requirement and OMB

guidance for the Government Information Locator Service. Relatively few have

thoroughly integrated GILS into the overall design of their websites. Hence, Table 4.1

should only be thought of as a starting point for performance measures to satisfy legal

and regulatory conditions.

Table 4.1. Checklist of Legal and Policy Conditions

Pertaining to Federal Agency Websites

Statute/Policy Checklist Question

A. Does the website contain a privacy notice that complies with the OMB guidance

and model language for federal websites?

B. Does the website avoid the use of “cookies” or observe OMB-stipulated

restrictions?

1. Privacy

C. Does the website comply with the Children’s Online Privacy Protection Act,

particularly with regard to collecting personal information from children? (Same as

9-B below)

A. Conventional FOIA: Does the website contain clear procedures for requesting

agency records under FOIA?

2. Freedom of

Information Act

B. Electronic FOIA: Does the website contain an Electronic FOIA Reading Room?

A. Does the website management include provisions to ensure that copyrighted

materials are not posted without permission from copyright holders?

3. Copyright

B. Reuse Restrictions: Can the site content be freely reused without restriction?

4. Accessibility Does the website make provision for accessibility for persons with disabilities in

accordance with §508 of Rehabilitation Act?

5. Security Does the website management include adequate provisions for protecting the

security of the website and other agency information systems?

A. Do information collections undertaken via the website have appropriate OMB

clearances?

6. Paperwork

Reduction Act

B. Does the website comply with provisions for the Government Information

Locator Service?

7. Depository

Library Program

Have publications posted to the website been made available to the Federal

Depository Library Program?

8. Government A. Does the website permit and encourage electronic information collection?

Paperwork

Elimination Act

B. Does the website permit use of digital signatures?

9. Federal Records

Act

Does the website management include adequate provision for identifying website

records and transferring records to agency record keeping systems?

A. Does the website comply with the President’s April 1997 guidance on

expanding Internet access for children, parents, and teachers?

10. Access for

Children

B. Does the website comply with the Children’s Online Privacy Protection Act,

particularly with regard to collecting personal information from children? (Same as

1-C above)
McClure, Sprehe & Eschenfelder 4-3 Federal Website Performance Measures

4.2 MANAGEMENT AND INFRASTRUCTURE FACTORS

Table 4.2 lists some management and operational factors that affect websites. As with

legal and regulatory conditions, the list can be considered as performance measures or as

prerequisites to performance measures. And as with the preceding section, Table 4.2 lists

factors in a more or less “Yes-No” fashion that represents a minimal level of adequacy.

The list is once again a starting point for agencies to ask themselves how their websites

stack up on the factor criteria, and then to go beyond the starting point. Going beyond

entails asking questions about the quality of agency practices and procedures.

Table 4.2. Management and Infrastructure Factors

Pertaining to Federal Agency Websites

Type of Factor Factor

A. Does the website have sufficient technology infrastructure (bandwidth,

server controller array, etc.) to ensure adequate response time?

B. Does the website have the appropriate software to achieve its purposes?

C. Does the website have the appropriate level of availability (e.g., 24 hours, 7

days a week)?

D. Does the website have adequate steps for emergency preparedness and

disaster recovery?

E. Does the website have adequate operating resources (funding and personnel)

to achieve its purposes?

1. Infrastructure

F. Are server errors and 404 errors (pages not found errors) at an acceptably

minimum level?

A. Does the website comport with industry best practices, such as the World

Wide Web Federal Consortium’s Home Page Guidelines, as regards content

design, navigation, organization, style, and markup.

B. Access control: Does the agency exert management control over who may

post to the website?

C. Quality control: Does the agency exert management control over the quality,

design, and style of website postings?

D. Awareness and Visibility: Has the agency ensured the site is easily reachable

by registering with search engines and portals and periodically searching for its

sites on these engines?

E. User Support: Has the agency provided an adequate user support staff for

technical and content queries?

F. Security: Is the agency providing adequate security to prevent intrusion into

the website?

G. Evaluation: Does the agency regularly evaluate the website through log file

analysis, focus groups, online user surveys, open public forums, and other

appropriate methodologies?

2. Management

H. Records Management: Has the agency made adequate provision for records

management on the website and integrated the website into the agency records

management program?

As an example, Table 4.2 lists records management and suggests taking and saving of

website snapshots. It does not ask whether the snapshots are transferred to an approved

agency recordkeeping system and indeed whether the agency has made any efforts to
McClure, Sprehe & Eschenfelder 4-4 Federal Website Performance Measures

seek records scheduling approval from the National Archives and Records

Administration for its website records.

The quality control questions asked in Table 4.2 are also just a beginning. The questions

pertain to management control, design, and style of the website. They do not pose

queries about the quality of content found on the site. The table mentions evaluation

methodologies but only scratches the surface of this complex and extensive topic.

Agencies may wish to go beyond the query in Table 4.2 and ask how website evaluation,

if it exists, dovetails with evaluation of other aspects of evaluating the agency’s

information dissemination products.

A website is only one dissemination medium among a range of media the agency uses

including CD-ROMs, printed products, telephonic information services, fax-on-demand,

in-person presentations at conferences and workshops, and so forth. A fully integrated

evaluation system would consider all media as an integrated set so that various products

and services, offered in several media, supplement and complement one another. The

foregoing examples serve as hints of the depth and complexity of institutional selfquestioning that the Government Performance and Results Act envisioned and that

website performance measures entail.

4.3 PERFORMANCE MEASURES

4.3.1. Performance Goals. Broadly speaking, all federal agencies share some

performance goals, while other goals are specific to individual agencies. The divergence

of specific goals was apparent even within the three agencies participating in this study.

GPO is working with interested agencies to facilitate permanent public access

partnerships for electronic agency information products. Permanent public access means

that government information products within the scope of the Federal Depository Library

Program remain available for continuous, no-fee public access. GPO also works with

partner agencies to provide a full range of development and website hosting services to

agencies without the necessary resources.

DTIC, because of its particular mission, aims primarily to serve the defense community,

meaning the Department of Defense and its contractors; serving the general public is a

secondary goal, albeit quite important. DTIC also is a center for development,

implementation and hosting of websites for other DoD organizations, and also provides

consulting services when requested to do so.

EIA’s website serves only EIA’s mission as a federal statistical agency and is not

intended to service other agencies. Whereas dissemination to the general public is a

secondary goal for DTIC, public access and dissemination of energy information are the

primary goals of EIA, goals aggressively and purposefully pursued.
McClure, Sprehe & Eschenfelder 4-5 Federal Website Performance Measures

Table 4.3. Performance Measures for Federal Agency Websites

Performance Goal Basic Measures Other Agency Specific Measures

Extensiveness:

Amount or extent

to which services

are used

-Information on content

unit retrievals; e.g., no. of

document downloads

-No. of user sessions per

time period (not no. of hits)

-No of User contact sessions

-Activity levels by time periods

-Ratio of unique to repeat (2 or more) user sessions

per time period.

Efficiency: Use of

resources in

providing services

-Cost of providing website

session per user

-Percent of operational

time when website is not

available

-No. of FTE hours or days devoted to website

creation/ maintenance by size of site in pages

-Cost per user help session

-Relation to diminishing costs of other publications

media (e.g., printing) as indicator that website may

be replacing other media.

Effectiveness: How

well the website

meets the general

governmental

objectives and

specific agency

objectives

-Completeness of coverage

of agency publications,

press releases, etc.

-Degree to which website

is increasing the timeliness

of access to agency pubs

-Permanent public access to agency publications

-Degree to which GILS is integrated into website

design/operations

-Degree to which website shows agency reaching

new constituent audiences

Service Quality:

How well the

website functions

-User success rate in

finding specific

information in a given time

period

-Average time between

user contact request and

agency response

-No. of customer

complaints/suggestions and

whether agency action

results.

-Whether agency has Help Desk dedicated to its

website

-24/7 availability measures

-Minimal 404 errors

-Courtesy, helpfulness of user support staff

-Increase in no. of repeat users per time period

Usefulness: How

well the website

meets the needs of

users

-Customer comments plus

surveys and focus groups

-Degree to which website

information increases user

productivity

-Degree to which website

information is incorporated

into other tasks inside and

outside the agency

-Cognitive and Usability Evaluation

-Measured user satisfaction with:

 *Clarity of homepage; organization of site

 *Timeliness of website information

 *Links to other useful information

-No. of referrals from other websites and sources of

referrals

4.3.2. Performance Measures. Table 4.3 provides selected proposed

performance measures from the many available in Chapter 3 and the appendices to this

report. The lists are nonexhaustive and only indicative of a range of measures that can be

listed.

Performance goals, like performance measures, shade into one another. For example, the

goal of effectiveness is how well the website meets the objectives of the agency. One

objective of the agency is doubtless to serve the needs of its information users in the
McClure, Sprehe & Eschenfelder 4-6 Federal Website Performance Measures

public, so that effectiveness blurs into the goal of usefulness. Hence, the same measure

may serve as indicator for two different goals.

Table 4.3 for the most part leaves out performance measures arising from log server

statistics and products such as WebTrends. Chapter 3 indicates that the participating

agencies are already employing these measures to various degrees. The numbers and

kinds of statistical measures from web log servers are extensive and have been

adequately treated. Additional detail about log analysis techniques and issues can be

found in Rubin (2000).5

Table 4.3 proposes sets of basic and additional performance measures and should be read

in conjunction with Tables 4.1 and 4.2. The table suggests a set of basic measures and a

set of other measures that agencies may choose to adopt for their own purposes.

In using the measures suggested in Table 4.3, agencies will need to carefullly define the

measures, operationalize them such that valid and reliable data can be collected, and

develop standardized procedures for collecting and analyzing the data needed to produce

the measures. It may be desirable for an interagency working group to develop agreed

upon definitions and procedures so that the statistics and performance measures can be

compared across agencies.

4.4. A BEGINNING APPROACH FOR WEBSITE ASSESSMENT

This chapter presents a view of website goals and performance from three different

viewpoints:

• Legal and regulatory conditions affecting federal websites, summarized in Table

4.1.

• Management and infrastructure factors pertaining to federal websites, summarized

in Table 4.2.

• Performance measures, in the conventional sense of the term, summarized in

Table 4.3.

In conclusion, the investigators offer the following recommendations:

4. Federal agencies should address the questions in Table 4.1 regarding agency

websites. These are basic questions about whether the practices of the agencies’

sites comply with federal law and policy.

5. Agencies should consider the questions in Table 4.2 as a useful but

nonexhaustive checklist of infrastructure and management questions they should

pose with respect to their websites.

6. Agencies should ask themselves whether their website performance measures

programs address the six performance goals listed in Table 4.3 and their

 

5

 Rubin, Jeffrey. Website Assessment Using Log Analysis: Issues and Techniques, in Charles R. McClure

and John Carlo Bertot, eds., Evaluating Networked Information Services. Medford, NH: Information

Today, 2000 (in press).
McClure, Sprehe & Eschenfelder 4-7 Federal Website Performance Measures

associated basic measures. The investigators do not necessarily recommend that

websites use the exact measures listed in the table, but they do recommend that

all agencies employ measures at least similar to these. The Other Agency

Specific Measures are a list of metrics agencies may choose to adapt to their

particular goals and purposes for their websites.

The routine use of measures described in Tables 4.1, 4.2, and 4.3 can assist agencies to

improve their websites as well as meet requirements for performance assessment as

stipulated in the Government Performance and Results Act. In using these or other

measures, agency staff should also consider the issues raised in Chapter 5, following.
McClure, Sprehe & Eschenfelder A-1 Federal Website Performance Measures

CHAPTER 5

ISSUES, RECOMMENDATIONS, AND NEXT STEPS

5.0 INTRODUCTION

As suggested in earlier chapters, a number of factors affect the successful management,

operation, and assessment of federal websites. These factors raise various issues to be

considered during the evaluation and assessment process of websites. Although site

visits occurred only at EIA, GPO, and DTIC for this study, the investigators have visited

and worked with other agencies regarding assessment of their websites.

Based on this knowledge and experience, this chapter identifies and discusses a number

of key issues and recommendations that agency officials should consider when assessing

their websites. The chapter also suggests some possible next steps that can be taken to

continue the work reported in this study.

5.1 ISSUES AND RECOMMENDATIONS

The issues discussed here are not intended to be comprehensive; rather, they suggest

important concerns that will need to be reviewed, discussed, and resolved at individual

agencies. The solution to these issues is likely to vary depending on a range of

situational factors in each agency.

5.1.1. Top Level Administrative Support for Website Assessment

While some ongoing assessment of selected and specific aspects of the website can be

done by individuals who simply have an interest in the website, a serious effort, and one

that is ongoing, requires top level administrative support. Top level administrative

support means that senior agency officials commit resources, staff, and moral support to

those involved in the evaluation assessment. Top level administrative support also

includes administrative interest in the evaluation effort and the desire to obtain

information and data that might be used to improve the website. Finally, top

administrative support means seeing to it that someone takes as a result of the evaluation

information as a means for continual improvement for that website.

5.1.2. Ongoing Program of Website Evaluation

Given the increased dependence of many agencies on disseminating information and

conducting business via their website, ongoing assessment of the website is essential. An

ongoing program of website assessment is one that minimally:

• Has assessment goals and objectives

• Assigns responsibilities to specific individuals or teams
McClure, Sprehe & Eschenfelder A-2 Federal Website Performance Measures

• Operates on a schedule in which specific aspects of the website are regularly

assessed

• Uses carefully selected performance measures that yield reliable and valid data

• Relates website assessment to larger agency performance assessment as

prescribed in the Government Performance and Results Act

• Regularly reports assessment findings to agency decision makers and users.

One-shot assessment of selected components of a website is not a program of assessment.

Agencies need to establish a regular ongoing program of website assessment that is

supported by top administrative levels.

5.1.3. Organizing for Website Assessment

Agencies have a number of options for organizing and managing the website assessment

process. In some instances this responsibility has been lodged in the office of the CIO; in

others, made a part of the Webmaster activities; in still others, accomplished by a team or

interoffice committee. There are strengths and weaknesses to each of these approaches

depending on the agency, but the key point is some clearly defined management structure

and one person who has responsibility for website assessment. In addition, agency staff

should know who has specific responsibilities in the assessment process, and other people

in the organization should know who is responsible for what tasks regarding website

assessment.

The experience of the investigators is that oftentimes there are no clear lines of

responsibility for websites assessment, and those assessments carried out are not made

known to other agency staff. Clear reporting lines and responsibilities for web

assessment are essential if successful evaluation is to occur.

5.1.4. Funding Website Assessment

The experience of the investigators is that very few agencies provide a regular budget or

adequate resources to support website assessment and the development of performance

measures appropriate for ongoing assessment of the website. Often, assessment costs are

buried in other budget categories or are not formally made available. In other instances,

evaluation occurs only because one or two staff (typically the webmaster) recognize the

importance of ongoing assessment and simply “do it” despite the lack of a budget.

Agencies need to recognize that a formal program of ongoing assessment of the website

will not occur, or at least will not be very successful, without resources and budgeting

that supports assessment work. Website assessment requires resources and agencies must

budget the resources so that those doing the evaluation can plan the approach to be used

in the assessment. The size of the budget will vary from agency to agency, but a clear

statement of resources needed and committed to website assessment is essential.
McClure, Sprehe & Eschenfelder A-3 Federal Website Performance Measures

5.1.5. Agree on Approach and Measures

Multiple methods and approaches (log file analysis, usability, surveys, focus groups, etc.)

can be employed to assess federal websites and multiple measures can be developed to

describe website activities and quality. Readers should consider Chapter 4 as a starting

point only. It may be less important which approach and which measures are used than it

is to have agreement on a program that employs some approach and some measures to

use. Thus, those responsible for website assessment should agree initially how the

assessment will be done and which measures will be used.

One problem observed by the investigators in this area is that there are so many possible

measures that either the staff are stymied by determining which approach should be used,

or they collect an endless amount of data in hopes of producing some useful measures

later. A general rule is to collect only that data that produces measures specifically

related to mission critical operations and services of the website and collect only that data

that will be analyzed and reported. There simply is not enough time and resources to

conduct exhaustively comprehensive website assessments. Thus, consider carefully

which approaches and which measures will have the greatest payback for (1) determining

the degree to which mission critical goals and objectives are being met, and (2)

improving the quality of services to agency users (both within and outside the

government).

5.1.6. Training

To some degree, seat-of-the-pants training can develop some basic skills necessary for

website development. But increasingly, staff charged with the responsibility of website

assessment and performance measures development will need training in areas such as:

• Data collection techniques, e.g., survey development, focus groups, interviews,

etc.

• Research methods and measurement scales and techniques

• Statistical analysis

• Report generation and use of graphics

• Use of software and technology such as log file analysis to assist in the

assessments.

Other skills and knowledge are likely to be needed in the future as web technology

evolves. Agencies must realize that an ongoing training process in assessment and

measurement will be needed. Needs assessment of staff skills and development of a

training program is an excellent starting point here. Agencies can acquire much of this

training by having staff participate in activities of organizations such as the World Wide

Web Federal Consortium and attend the annual FedWeb conferences.

5.1.7. Recognizing the Policy Context
McClure, Sprehe & Eschenfelder A-4 Federal Website Performance Measures

A host of policy guidelines, directives, and requirements affect the manner in which

federal agencies design, develop, administer, and evaluate their websites. Chapter 2

provides an overview and assessment of these policies. Many agencies simply are not

aware of these policies and routinely fail to consider them as they develop and operate

their websites.

At one level, failure to recognize these policies and guidelines may cause agencies to

reinvent the wheel related to website evaluation. Some of these policies and guidelines

(e.g., privacy protection) offer very useful information on how to design and operate the

website. At another level, agencies may put themselves at risk by ignoring these policies

and guidelines. Increasingly oversight efforts from Congress, the General Accounting

Office, and OMB are being focused on website performance and operations at the

agencies.

5.1.8. Sharing Knowledge

The investigators have observed the efforts of a number of agencies as they experiment

with web assessment methods, measures, and organization to assess their website. As

agencies move through this process much is learned, but much stays within the agency

and is not available for others who are confronting the same issues to learn from and

incorporate in their assessment efforts. Indeed, the three agencies that participated in this

study were able to learn much from each other.

Agencies need to do a better job of sharing their knowledge of website assessment,

letting other agencies know what works and what does not work, and offering ideas and

suggestions for new approaches. As part of this study the investigators identified similar

efforts to develop measures and conduct website assessments at CENDI (an interagency

science and technology information group that includes Commerce, Energy, National

Technical Information Service, National Library of Medicine, National Agricultural

Library, Defense, Interior, and others) and from the ICSP (Interagency Council on

Statistical Policy). While there are groups such as the Federal Webmasters Forum and

WWW Federal Consortium that might serve as a mechanism to support such sharing of

information regarding website assessment, no clearinghouse of information about

methods, techniques, measures, and strategies for assessing federal websites now exists.

One of the existing groups might profitably undertake establishing such a clearinghouse.

5.1.9. A Changing Web Evaluation Context

The context in which agency officials will collect, analyze, and report measures related to

websites is considerably different than in the past:

• Agency staff often do not control access to and use of a range of data that describe

vendor-supplied information services and resources. Some agencies contract with

vendors (or other governmental units) that are unwilling or unable to provide the

types of statistics and use data that officials request. Statistics and measures for

database use and services, nonetheless, are essential.
McClure, Sprehe & Eschenfelder A-5 Federal Website Performance Measures

• The rapidly changing technical nature of the networked environment also affects

the types of services and resources that can be provided by agency websites. As

the networked services change, new types of evaluation approaches and measures

may be needed.

• The level of effort needed to collect, analyze, and report data to produce

performance measures for web-based services may be greater than that needed to

produce more traditional statistics.

• Software such as web server log analysis provides “automatic” counts and

statistics of selected services and network activities. One can expect that these

automatic or software-based approaches will become further refined and

developed.

• Networked services, costs, and use may be difficult to “unbundle” if the agency

obtains these services through a consortium of other governmental units. Costs

can either be hidden or be extremely difficult to allocate to individual operations

or agencies.

Agency officials may be entering a period of time where measures for website operations,

use, and services may be useful for two to four years, perhaps less, and then will have to

be redeveloped or discarded. Such an environment is quite different than the statistics

collecting environment in which many agencies previously existed. Despite these

concerns and factors, agencies can move forward and learn how best to produce and use

such measures in this new environment.

5.2. NEXT STEPS

Federal agency website evaluation and development of performance measures is in its

infancy and much work lies ahead in designing, testing, and implementing evaluation

methods and measures. The investigators recommend two broad areas for next steps that

can build upon the work described in this report.

5.2.1. Follow-on Study

The first area is to continue study on a number of topics that have been identified in this

report. These topics include, but are not limited to:

• Design and test additional performance measures for agency websites

• Determine the degree to which website performance measures (as suggested in

Chapter 4) can be used in agencies beyond EIA, GPO, and DTIC

• Assess organizational structure and managerial strategies as they affect website

assessment

• Identify, describe, and assess costs associated with different levels of web-based

services

• Integrate and coordinate various data collection methods and approaches into

models of evaluation that could be used by agencies at different stages in website

evaluation
McClure, Sprehe & Eschenfelder A-6 Federal Website Performance Measures

• Assess the way in which agencies have complied with policies affecting website

development and operation

Appendix C provides an example proposal for studying the degree to which agencies are

complying with website policies. The investigators can also develop proposals that

address other topics identified above.

An important follow-on to the study reported here would be to determine agency

compliance with the various policies affecting websites as summarized in Chapter 2.

Such a study would have a number of benefits for an agency, not the least of which

would be overall improvement and coordination of its website procedures with the

ultimate goal of better meeting user needs.

5.2.2. Disseminate Findings Widely and Increase Awareness

A second key area for next steps is to disseminate study findings widely and increase the

awareness and knowledge base of agencies related to website assessment. The findings

presented in this report might be best distributed by holding a government-wide

conference for federal webmasters, IT systems experts, policymakers, records managers,

and others. At this conference agency officials could learn about possible performance

measures, assessment techniques, and new developments related to agency best practice

in this area. They could also share their knowledge of assessment as done in their

agencies. Such a conference would also provide an excellent opportunity for agency

officials to interact and exchange views on assessment.

The report presented here might also be formally published by the GPO or another

federal agency and distributed widely. As suggested earlier in this chapter, there is a

dearth of information about federal website evaluation and performance measure

assessment. Wide dissemination of a published report should increase attention to the

topic and provide others with information that may assist them in their efforts to evaluate

websites.

Finally, there needs to be better organization and sharing of knowledge related to website

assessment and performance measures. An important next step would be to formally

charter an agency or interagency group (e.g., Federal Webmasters Forum) to serve as a

clearinghouse on information, reports, best practices, etc. related to assessing federal

websites. Such an effort could have significant impact for agencies by saving them time

and effort to develop and refine methods and measures that may already be known and

tested.

5.2.3. Continual Improvement of Federal Websites

In the near term, it is clear that federal agencies will continue to develop and expand their

web presence. Not only will a significant increase in resources and information be added

to these websites, but also interactive services and e-commerce will increasingly be
McClure, Sprehe & Eschenfelder A-7 Federal Website Performance Measures

conducted with both citizens and with other agencies. Given the government’s ongoing

efforts to reduce costs and improve services, one can also expect increased policy

emphasis that encourages agencies to conduct business electronically and via their

websites.

Evaluation and performance measures to describe federal websites and resources in the

networked environment are important tools. But an ongoing program of evaluation that

regularly produces these performance measures is a tool that is essential for the long-term

success of the agency. These measures provide agency decision makers with a means to:

• Describe the degree to which web-based services and resources are accomplishing

agency goals and objectives;

• Assess the degree to which resource allocations for various web-based services

and resources are appropriate;

• Monitor the status and development of these services so that it is possible to make

quick corrective actions;

• Identify the impact, benefits, importance, and problems with web-based services

and resources;

• Assist decision-makers to determine whether to continue or modify existing webbased services or develop new services;

• Provide objective information for justification of the website or otherwise

demonstrate accountability for the web-based programs; and

• Educate decision-makers, staff, policymakers, users, and others as to the

importance and benefits from web-based activities and services.

Perhaps most importantly, an ongoing program of evaluation contributes to the process of

constant improvement – looking for ways to improve the usefulness, impact, and benefits

that can result from web-based resources and services.

At issue is the degree to which these web-based resources and services are cost effective,

deliver high-quality services, meet the needs of users, comply with existing policy,

reduce agency costs, and help accomplish agency mission and objectives – to name a few

criteria. If agencies are to operate successfully in this web-based information

environment they must assess their success in addressing these criteria in a regular and

ongoing program of evaluation and measurement. Developing and implementing such a

program is both doable and essential if the long-term health of the website and the

agency’s programs are to be insured.
McClure, Sprehe & Eschenfelder A-8 Federal Website Performance Measures

APPENDIX A

CRITERIA FOR THE EVALUATION OF FEDERAL WEBSITES

NOTE: The materials in Appendix A were developed by Charles

R. McClure for a class recently taught at Florida State University.

This appendix represents a slightly different approach to many of the

subjects addressed in this report. Nonetheless, the criteria listed here

can be a useful tool to assess Federal websites. Hence, the authors have

included the appendix for its potential usefulness to readers of the

report.
McClure, Sprehe & Eschenfelder A-9 Federal Website Performance Measures

APPENDIX A

CRITERIA FOR THE EVALUATION OF FEDERAL WEBSITES

Introduction: Listed below are a number of possible criteria that can be used to assess

websites, and to some degree, other types of electronic services provision in a networked

environment. All of these criteria may not be appropriate for specific types of websites.

The purpose of the website (e.g., educational, entertainment, commercial, services

provision, etc.) shapes the types of crtieria that may be appropriate for a particular

website.

I. INFORMATION CONTENT CRITERIA

This section evaluates the substantive aspects of the website

Orientation to website

• A website overview is provided (is it clear if this is a commercial, educational,

advocacy, informational, or other type of website?)

• The scope of website is clearly stated.

• The services and information provided at the website are described.

• Instructions on the use of the website are provided.

• The purpose/mission should be appropriate to the targeted audience(s).

• A liability statement warning the user of information provided through the links is

provided

(e.g., access by children).

• Copyright statements are provided if necessary.

Content

• The content of the homepage should match the purpose/mission.

• The content and links match the needs of the expected audience.

• The website includes only necessary and useful information. content coverage does

not overlap.

• The amount of information is significant and not overwhelming, is balanced

throughout the website.

• There are full text references or other resources available.

• The content has rich and unique quality that inspires users to visit regularly for

information.

• The content is written in a clear and consistent language style that matches the

expected audience.

• Avoids jargon, humor, condescension, accusation, and chit chat.

• Uses a positive and professional tone.

• Language does not show bias.
McClure, Sprehe & Eschenfelder A-10 Federal Website Performance Measures

• Content is organized logically throughout the website and by the user's needs.

Currency

• Address of contact person and last update information appears at the bottom of pages

with substantive content.

• Pages should be kept up to date.

• The content provided by the website creators is up-to-date.

Bibliographic Control

• Headings are clearly phrased, descriptive and understandable.

• Each screen is titled clearly.

• If the headings cannot be completely descriptive, coherent and concise, descriptions

follow.

• The information that is provided through the link matches the headings and

descriptions.

• Terminology and layout are consistent within the headings throughout the website.

• Navigational tools standardized and on each page of site.

Services

• Provision of services is different than provision of information resources,

(e.g.,leaving a question to be answered by the reference staff is a service).

• Are the services open to everyone on the Internet or do parts require fees?

• Services meet the needs of the user.

• Services are fully operational and if they cost money are the fees clearly presented?

Accuracy

• Statement of document/website status is provided (e.g., if in progress will note,"under

construction").

• Reference or sources of information cited are accurate.

• Typing, spelling, and grammar errors and other inconsistencies are absent.

Privacy

• Degree to which the site provides users with explicit policy on how users’ privacy

rights are protected.

• Degree to which the site provides users with information about making public site-use

information, repackaging, or selling such information to others.

• Can the user exchange encrypted information with the site?

Security
McClure, Sprehe & Eschenfelder A-11 Federal Website Performance Measures

• Is the user assured that his/her use of the website is scure?

• Is encrypted use of the website possible?

• Are passwords needed to access certain parts of the website?

Availability of Log File Information

• Do users have access to basic statistics describing use of the website?

• What information does the website make publicly available regarding the use and

assessment of the website?

Retrieval/Search Engine

• Does the website have its own search engine?

• Are their instructions to help the user better understand how to do searching on the

website?

• What is the overall effectiveness, ease of use, and power of the search engine?

Policy Issues

• Written policies and procedures that govern the development, use, and access to the

website (who is responsible for what, marketing, commercial use of the site, access

by children, etc.?

• Use of Electronic Records Management policies that guide the management of the

website (length of time material is on the website, approval for placing material on

the website, legal liability, acceptable as evidence ina court of law. etc.?

• Preservation, archival, and scheduling of web content for future access and

preservation?

• Privacy, FOIA, and other federal guidelines and laws?

• Degree to which organizational policies support broader institutional policies?

II. EASE OF USE CRITERIA

This section evaluates the physical movement through the website.

Quality of Links

• There are no dead-end links.

• Temporary forwarding addresses do not qualify as good links.

• "What's new" section provided for new links - (good for frequent users).

• Shortcut links are possible for frequent users.

• Warning statements are provided, if link will lead to large document or image.

• Indication of restricted access for a link is provided.

• Links are provided to mention documents.
McClure, Sprehe & Eschenfelder A-12 Federal Website Performance Measures

Speed

• Minimal use of large graphics and bright colors so that document loading is relatively

fast at slow connection speeds?

• Efficient programming of links for quick movement through the website?

Feedback Mechanisms

• On the homepage and other significant pages there is a contact person: name, address

and email address of the responsible person or entity for example,

"webadmin@library.org."

• Feedback links are fully operational.

Accessibility

• There is an awareness of the existence of this website through search engines or other

publicity.

• Full name of website organization is provided in the title, heading, document address,

graphical link and/or URL in order that the source can be recoverable.

• It is usually possible to reach the site. It is not frequently overloaded.

• The URL is not likely to be confused or mistyped.

• Consistent metadata embedded in pages to improve search engine indexing.

Design

• Format is appropriate to subject matter and functionality. A good design directs users

toward information rather than away from it.

• The screens are uncluttered.

• The format is consistent throughout the website.

• The homepage for the website is short and simple.

• The website is written in standard HTML language. The site is consistent when

accessed via different browsers (text and graphic).

• Graphics and color are used to lead the user through the information appropriately.

• Monster graphics are not used.

• Many little graphics are avoided.

• Use of bold, italics, blinking and other attention getting devices is appropriate.

• Icons are understandable and make sense as what they actually do.

• User has the option of turning off the automatic loading of graphics.

Navigability

• Essential instructions appear before links requiring user interaction (e.g., email).

• Navigation options are distinct and spelled out.

• Minimal user skills are required.

• All the parts work.
McClure, Sprehe & Eschenfelder A-13 Federal Website Performance Measures

• Interactions are secured if they involve private information.

• Links are provided to return to website homepage on all supporting pages.

• Links are provided to assist navigation (e.g., "return to top," "return to previous

page").

Video and Audio

• Applications on the site are appropriate.

• Applications work as intended and instructions to operate are clear and straightforward

• Audio and video uses provide links to sites where the necessary “plug and play”

software can be obtained and downloaded for immediate use.

• Enhances site performance and services and is not “glitz.”
McClure, Sprehe & Eschenfelder B-1 Federal Website Performance Measures

APPENDIX B

WEB SITE PERFORMANCE MEASUREMENT

AND EVALUATION FOR

GPO ACCESS

A PowerPoint presentation by

T.C. Evans

Given at FedWeb 2000
McClure, Sprehe & Eschenfelder B-2 Federal Website Performance Measures

Slide 1 _____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 2 Purpose of Presentation

l To enumerate the key areas of performance

measurement, as we see them

l To give specific examples of how we evaluate

these factors for GPO Access

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 3 Performance Measurement Elements

l Legal Mandates l Goals and Objectives l Infrastructure l Site Content l Site Design

l User Support l Awareness l Usage Metrics l Customer Satisfaction

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-3 Federal Website Performance Measures

Slide 4 Performance Measure:

Legal Mandates

l OMB Directives l Federal Regulations l Mandates passed into law

l Executive Orders l Court orders

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 5 Legal Mandate for GPO Access:

Public Law 103-40

l Online Federal Register and Congressional Record; other databases determined by

Superintendent of Documents l A Federal information locator l Electronic Storage Facility for Federal

Information

l Federal Bulletin Board (included by

Senate Report)

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 6 Performance Measure:

Goals and Objectives

l Based on agency/program mission

l Based on intended audience l Based on legal mandates

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-4 Federal Website Performance Measures

Slide 7 GPO Access Goals and Objectives

l Improve access to Federal Government

information in electronic formats l Provide well-designed, easy to use, publicly

available online services l Provide official and complete information

l Provide Finding Aids l Ensure access to broadest audience possible l Electronic information a byproduct of

print process l Ensure Permanent Public Access

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 8 Performance Measure:

Infrastructure

l Sufficient bandwidth

l Appropriate hardware/software l System performance l Site availability

l Geographic Separation/Mirror Site l Disaster recovery

l Partnerships

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 9 GPO Access Infrastructure

l Server Controller Array

l Server farm

l Distributed storage l Permanent data backup

l Bandwidth

l Partnerships with Depts. of Energy, State,

and Commerce, LSU, others

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-5 Federal Website Performance Measures

Slide 10 Performance Measure:

Site Content

l Fulfillment of potential l Permanent Public Access l Authentic, official information

l Timeliness

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 11 Content of GPO Access Today

l Almost 1,500 databases; over 80 applications l Over 176,000 titles l Suite of Finding Aids l Hosted sites for outside agencies l Entered into partnerships l Secure Online Bookstore l Ben’s Guide to U.S. Government for Kids l Commitment to Permanent Public Access

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 12 Performance Measure:

Site Design

l ADA compliance and universal accessibility

l Privacy and security

l Consistent design

l Consistent linking structure l Kids’ pages

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-6 Federal Website Performance Measures

Slide 13 GPO Access Site Design

l Level 2 of W3C Web Design Guidelines l Simplified URLs l Consistent look and feel l Consistent navigational tools l Site designed to accommodate “lowest

common denominator” technology

l dial-up/telnet access

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 14 Performance Measure:

User Support

l Availability of support service l High-quality responses l Timeliness l Accessibility

l Trained intermediaries l Effective self-support tools l Scalability

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 15 GPO Access User Support

l Help available via phone, fax, and e-mail l Live phone help available on weekdays l Timely response l User contacts monitored for quality

l Variety of online self-help materials

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-7 Federal Website Performance Measures

Slide 16 Performance Measure:

Awareness

l Reaching intended audience l Educating audience l Expanding audience l Facilitating visibility through

search engines/portals

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 17 GPO Access Awareness Efforts

l Promotional efforts l Training classes l Trade shows and conference exhibits l Speeches and demonstrations l Ongoing search engine ranking

improvement project

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 18 Performance Measure:

Usage Metrics

l Useful statistical measurements l Appropriate usage measurements l Consistently recorded and reported

l Consistent coverage

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-8 Federal Website Performance Measures

Slide 19 GPO Access Usage Metrics

l Measure content retrieval l Measure retrieval for all GPO Access components l Identify and understand gaps in usage data l Content retrieval statistics reported on a

monthly basis l Provide retrieval statistics to hosted site

agencies

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 20 Performance Measure:

Customer Satisfaction

l Understanding of user expectations – ease of use – timeliness – responsiveness – intuitive site organization l Meeting of users’ expectations l Problem resolution

l Courtesy

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________

Slide 21 Customer Satisfaction and GPO Access

l Open forums and focus groups l E-mailed suggestions l Training sessions l Trade shows l Online surveys

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder B-9 Federal Website Performance Measures

Slide 22 Evaluation of Performance Measures

l Reexamine site in light of gained

knowledge; begin cycle again

l Performance measures will evolve with

the Web site l Ongoing monitoring of performance

measures needed to ensure mandates,

goals and objectives are met

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________________________

_____________
McClure, Sprehe & Eschenfelder C-1 Federal Website Performance Measures

APPENDIX C

EXAMPLE: PROPOSAL TO STUDY AGENCY COMPLIANCE

WITH FEDERAL WEBSITE POLICIES
McClure, Sprehe & Eschenfelder C-2 Federal Website Performance Measures

APPENDIX C

Example: Proposal to Study Agency Compliance

with Federal Website Policies

Study Objectives

The overall purpose of the proposed project is to better understand the degree to which

Federal agencies are aware of and have responded to current information policies that

affect the management and operation of Federal websites. More specifically, study

objectives include the following:

• Determine the degree to which a selection of agency web managers and related staff

are aware of the various information policies affecting the operation and management

of these websites.

• Determine the degree to which a selection of agency websites have responded to and

complied with these various information policies in the management and operation of

their websites.

• Assess the degree to which the various information policies actually assist agencies in

operating successful websites and propose possible policies that should be considered

for improved management and operation of their websites.

• Make specific recommendations to the agency as to how its websites might better

respond to and address the various information policies currently in place.

Ultimately, the study will assist agencies to determine how well they are responding to

the existing information policies that affect the operation and management of Federal

websites and improve the usefulness, impact, and management of those websites.

Timeline and Tasking

The study would be a five to seven month effort depending on the depth of investigation

require by an agency or group of agencies. It would have the following general phases

and tasking:

Phase I: Project Organization (one month). In this first phase the investigators will

develop detailed project tasking. They will meet with appropriate agency staff to

obtain a better understanding and knowledge of their websites. Agency staff will

provide the investigators with a range of internal documents and reports related to

their website management. The investigators will (with the assistance of agency

staff) identify a sample of agency websites to be used for the assessment. And the

agencies each will establish a single point of contact for the study as well as a small

advisory committee that can assist in study activities and logistics.

Phase II: Data Collection (two-four months). During this phase the investigators

will use a range of methods to obtain data that will answer the study objectives
McClure, Sprehe & Eschenfelder C-3 Federal Website Performance Measures

outlined above. These methods may include interviews with key informants at

participating agencies; log analysis; policy analysis; surveys; focus groups; usability

assessments; and content analysis of websites. The investigators will provide a draft

report of preliminary findings at the conclusion of this phase.

Phase III: Final Report, Recommendations, and Dissemination (two months).

Based on the findings from the data collection, the investigators will produce a final

report that will summarize the degree to which the websites successfully respond to

the information policies as outlined in Chapter 2. The report will also offer specific

recommendations for (1) how these policies might be improved, and (2) how the

agency can better utilize these policies for improved management and operation of

their websites. The investigators will also, with the assistance of participating

agencies, disseminate project findings and recommendations within the agency and

the larger Federal community via an Executive Briefing session.

These proposed Phases and general tasking are likely to be modified and refined to best

meet the needs of the participating agencies during Phase I of the study.

Budget

The investigators propose a project budget in the range of $60,000 - $75,000 for this fiveseven month study. McClure and Sprehe, as Co-Principal Investigators would contribute

a predetermined number of days per month on the project. They would hire additional

staff, 2-3 graduate research assistants at 20 hours per week. Other resources would be

used for travel, communications, supplies, copying, etc. This budget assumes that agency

liaisons will be able to work with the investigators and can draw upon some assistance

from agency staff in the completion of the study.

Study Products

A number of study products will result from this effort, including:

• A final report that offers specific recommendations to the participating agencies on

how they might better respond to and comply with various information policies for

improved management of and services from their websites.

• The report will also provide a set of procedures that the agencies can use in the future

to develop an ongoing program to insure adequate awareness of and compliance with

new information policies affecting website management and operation.

• A revised and expanded Chapter 2 (Information Policies Affecting Federal Websites)

based on the findings from the study.

• An Executive Briefing session for selected agency officials that would increase

agency knowledge of policies affecting the management and operation of Federal

websites and how those websites might be improved.

These products provide an important set of tools and strategies by which agencies can
McClure, Sprehe & Eschenfelder C-4 Federal Website Performance Measures

continue to improve and develop their websites.

Importance of the Study

The investigators’ work in Chapter 2, Information Policies Affecting Federal Websites

will provide an important tool to agencies that can assist them to better manage and

operate their websites – and be in compliance with the existing policies affecting website

management. The procedures and findings from the study are also likely to benefit other

agencies that are learning how to deal with the various information policies affecting

website development.

In addition, the study will significantly raise the visibility of the importance and use of

agency websites. The project will also demonstrate how agencies are taking a leadership

position in addressing the range of Federal information policies that affect website

management and operations. Finally, and perhaps most importantly, the study will result

in agency websites that are more successful in meeting agency mission and goals, more

successful in meeting user information needs, and more successful in how they are

managed and operated.