Lessons Learned About Designing,

Developing, and Disseminating

Environmental Information Products

U.S. Environmental Protection Agency

Office of Environmental Information

Environmental Analysis Division
Abstract

The Environmental Protection Agency (EPA) is committed to expanding access

and enhancing the usability of environmental information under Goal 7 of the

Agency’s Strategic Plan. One way to expand access to environmental information is for the Agency to enhance existing data and to develop integrated information products derived from data compiled to support existing environmental

management and regulatory programs. This is sometimes referred to as the

secondary use of environmental data. The Office of Environmental Information

(OEI) recently conducted a series of interviews with senior information managers to determine and articulate best practices for designing, developing, and

disseminating information products, especially those based upon data collected

for other purposes.
November 17, 2000 i

CONTENTS

EXECUTIVE SUMMARY ................................................................. 1

1. INTRODUCTION ..................................................................... 5

1.1. Background ......................................................................................... 5

1.2. Approach ............................................................................................ 6

1.3. Organization of this Document ........................................................... 7

2. KEY FINDINGS ........................................................................ 9

2.1. Design ................................................................................................. 9

2.2. Development and Review ................................................................. 16

2.3. Dissemination, Maintenance, and Feedback .................................... 17

3. IMPLICATIONS FOR OEI ........................................................ 21

APPENDIX A – Information Products Discussed ........................................ 24

APPENDIX B – Advisory Committee Members .......................................... 26

APPENDIX C – Interview Questions........................................................... 27

LIST OF EXHIBITS

Exhibit 1. Secondary Use of Environmental Data:

How Data Become Information Products ...................................... 8

Exhibit 2. The Six Principle Aspects of the

Information Product Life-Cycle .................................................... 9

Exhibit 3. Stakeholders, Customers, and the Public .................... 10

Exhibit 4. Phases of Stakeholder Input ....................................... 12

Exhibit 5. Illustrative Information Partnership ............................. 15

Exhibit 6. Affected Parties by Product Types ................................ 18

Exhibit 7. Elements of a Strategic Information Product

Planning Process ....................................................................... 23

November 17, 2000 Page 1

Lessons Learned About Designing,

Developing, and Disseminating

Environmental Information Products

EXECUTIVE SUMMARY

The Environmental Protection Agency is committed to expanding access and

enhancing the usability of environmental information under Goal 7 of the

Agency’s Strategic Plan. One way to expand access to environmental information

is for the Agency to enhance existing data and develop integrated information

products based on data inventories that support environmental management and

regulatory programs. It is important to evaluate information products as part of an

overall effort to assure customer satisfaction and continuous improvement.

This Lessons Learned report focuses entirely on one environmental information

gateway to the public: government information products. Environmental information products produced by EPA and states vary greatly in terms of management

and implementation. As a result of their varying design and development protocols, the Office of Environmental Information sought to examine past efforts to

develop information products in order to delineate lessons learned that may be

helpful to future endeavors. Information contained in this report was derived

primarily from interviews conducted with agency staff involved with the development and use of a variety of information products in order to understand and build

upon their experience developing and managing information products. The

ultimate goal of this project was to determine if there were some common lessons

learned across various types of products and product developers.

An effort was made to incorporate a range of different product types. These

include those driven by statute or regulation (e.g., the Air Trends and TRI reports); compliance-based products (e.g., Pennsylvania Compliance Reporting

System and the Sector Facilities Indexing Project); those designed to enhance

data management and compliance (e.g., Region 3’s Enviroviz, and the Office of

Water’s Index of Watershed Indicators); and those involving numerous external

stakeholders (e.g., the South Baltimore Pilot Project).1

As noted above, we focused on identifying overarching issues rather than

limiting the investigation to any one information product or category of products. However, our analysis did recognize that the lessons learned will vary

according to specific product type and category. For example, compliance

products can be controversial due to the response of specific facilities and

1 Please see appendices for a complete list of the products evaluated, brief descriptions of each

project, and a list of the individuals interviewed.
Page 2 November 17, 2000

individuals to Agency programs. In this report, we have identified a set of

issues and concerns that are generally applicable to the design and management

of EPA and state information products. Where appropriate, we have referenced

specific information products when discussing lessons learned.

Key lessons drawn from the interview process include the following:

• Almost all interviewees agree that it is critical to articulate a clear sense

of the product’s purpose early in the design process. As obvious as this

may seem, interviewees stress that this step is frequently overlooked,

resulting in redundancy among products, suboptimal user functionality,

and other problems.

• It is a mistake to develop a generic information product. While the reality

that information products circulate beyond their intended audience

should be considered during planning, it should not be used as a rationale

for avoiding the need to specify a target audience and its unique needs for

environmental information.

• Stakeholder and product audience input must be sought at all points in

the information product life-cycle. Early stakeholder involvement is

especially important as it can help clarify goals, reduce skepticism, build

consensus, and ensure the dissemination of information that meets

audience needs.

• Data ownership is directly related to data quality concerns. While some

users of secondary information products take the initiative and communicate effectively with originators of data, EPA lacks an organized framework to guide this kind of technical interaction. Many interviewees stress

the importance of developing a mechanism to facilitate communication

among producers and users of EPA data.

• To address concerns over data accuracy, some developers and managers

interviewed note the importance of a “data feedback loop,” through

which facilities and other data providers are requested to verify data used

in a particular information product. Beyond sending the data back to

providers, some interviewees feel that the Agency should also consider

initiating an internal comment period via an intranet-accessible form,

allowing adequate time to make corrections prior to public release.

• It is necessary to consider all aspects of information usability. Misunderstanding and misapplication of information products can be avoided by

providing users with comprehensive and carefully articulated metadata

and contextual information.
November 17, 2000 Page 3

As a result of these findings, the OEI, Office of Information Analysis and

Access (OIAA) has embarked on developing a Best Practices series for information product development. Building on the feedback and recommendations

from this report, OIAA will lead the Agency effort to develop and release

guidance documents on each important phase of an information product’s lifecycle. Each guide will consist of EPA examples of best practices, case study

applications of these practices, and reference pertinent policy and guidance

already in existence for each topic area. The overarching objective of the series

is to disseminate helpful practices to EPA information product developers and

managers. In doing so, OIAA and partners strive to heighten Agency information sharing and best practice implementation for environmental information

products, thereby enhancing the credibility, usability and defensibility of EPA

products.
Page 4 November 17, 2000

What is an Information Product?

A joint EPA/State Action Team was recently convened to design

and publish a periodic bulletin to inform stakeholders and the

public of development and release of significant information

products. The joint team defines a significant information product

as: “A product under development or modification by EPA which

derives from federal, state, local, tribal, or other organizations’

data, and a state product that is regional or national in scope and

aggregates data from more than one state. Such products often

generate considerable attention when they use [pre-existing] data

to describe environmental conditions, trends, potential risks, and/

or portray compliance or performance.”

Significant information products include the following:

• Products that analyze and/or compare data from various

agencies and organizations, including industry, as well as

various federal, state, tribal and local agencies;

• Significant raw unanalyzed data collected by EPA from

various agencies and organizations;

• Products that describe or assess environmental conditions,

trends, or risks;

• Products that apply to a large segment of the population or

large geographic area;

• Models used by the public to perform environmental analyses

based upon data from various agencies and organizations; and

• Annual reports and other products released on a regular basis

that describe environmental conditions, trends, risks, and/or

portray compliance or performance.
November 17, 2000 Page 5

1. INTRODUCTION

1.1. Background

Information about the environment – environmental characteristics; physical,

chemical, and biological processes; and chemical and other pollutants - underlies all environmental management decisions. Access to quality information and

analytical tools is essential for public decision making, measuring environmental improvement, assessing and managing risks, and evaluating progress. Access to high-quality, user-focused information products will enable the public to

understand environmental conditions and make informed decisions about their

communities.

The Environmental Protection Agency articulates its commitment to expanding

and enhancing access to environmental information in its Strategic Plan. Goal 7,

Expanding American’s Right to Know About Their Environment, stresses the

importance of developing and disseminating information products that will

enable the public, governmental agencies, non-governmental organizations and

private sector parties to easily access relevant, usable, and quality-assured data

and information products.

The vision of the EPA Office of Environmental Information (OEI) is to “advance the creation and use of data as a strategic resource to enhance public

health and environmental protection, inform decision making, and improve the

public’s access to information about environmental conditions and trends.” Key

goals of the Office of Environmental Information include the following:

• Promote the public’s right to know,

• Improve information infrastructure,

• Reduce burden,

• Foster information partnerships,

• Improve data quality, and

• Foster data integration.

The Agency’s approach to information product development and dissemination

is relevant to all of OEI’s goals. It is therefore critical to understand what makes

information products effective, widely used, and technically credible. It is

especially important to assure the quality and user applicability of information

products derived from existing data.

It is...critical to

understand

what makes

information

products

effective,

widely used,

and technically

credible.
Page 6 November 17, 2000

To this end, the Environmental Analysis Division (EAD) within the Office of

Information Analysis and Access interviewed senior staff involved with the

development of major environmental information products within and outside

the Agency. The lessons learned from this activity serve as a resource for

formulating information product guidance and will support a variety of efforts

to foster continuous improvement in the development and dissemination of EPA

information products. The themes and implications outlined in this document

are part of an overall effort to characterize best practices and ensure continuous

improvement in EPA information products, services, and policies.

1.2. Approach

As indicated above, the

findings and recommendations articulated in this report

were derived through a series

of interviews with senior

EPA and state staff involved

in the development and/or

administration of environmental information products.

Working with an Advisory

Committee of senior EPA

staff and managers, OEI

selected interviewees closely

involved with projects resulting in a variety of information products, including

paper-based reports, webbased products and information applications, and multi-media products with

geographical scopes of interest ranging from local to national perspectives.

Many of these information applications utilize data collected for other purposes.

All interviewees responded to the same questionnaire, and the research team

focussed on common issues rather than investigation and critique of individual

products. The products evaluated through the interview process can be grouped

in terms of four broad categories:

• Information products that are driven by statutory or regulatory reporting

requirements, such as the annual Toxics Release Inventory (TRI) and Air

Quality Trends reports.

• Information products developed or used to facilitate regulatory compliance, such as the Pennsylvania Compliance Reporting System (PA CRS)

and the Sector Facility Indexing Project (SFIP).

Information Projects Reviewed

✓ Environmental Monitoring for Public

Access and Community Tracking

(EMPACT)

✓ Index of Watershed Indicators (IWI)

✓ National Air Quality and Emission

Trends Report

✓ Pennsylvania Compliance Reporting

System (PA CRS)

✓ Enviroviz

✓ Sector Facility Indexing Project (SFIP)

✓ South Baltimore Pilot Project

✓ Toxics Release Inventory (TRI)
November 17, 2000 Page 7

• Information products designed to enhance data management or facilitate

the integration of diverse data sets, such as Enviroviz and the Index of

Watershed Indicators (IWI).

• Information products that involve extensive stakeholder input and/or

review, such as the South Baltimore Pilot Project.

While diverse in many respects, the information products discussed during the

interviews are similar because they are all based upon numeric, statistical, or

graphical data. Most of these products are developed for secondary uses, a

category that includes most information products from OEI. As illustrated in

Exhibit 1, information products for secondary use are frequently based upon

data extracted from EPA program systems. EPA data systems are used to support or administer regulatory programs, whereas secondary information products are designed to address other purposes. The information products reviewed

during the interview process are all “data rich,” as opposed to those intended

only to disseminate a message, broadcast the occurrence of an event, express an

opinion, or describe an activity or program. Although all public communications are important and require thoughtful formulation, data-based information

products present a special set of challenges. For instance, data-based information products require careful attention to factors such as accuracy, quality and

precision, and the ability to support integration across multiple systems. These

and other attributes of the primary data must be carefully considered in the

design, development, and dissemination of information products intended to

serve secondary purposes.

1.3. Organization of this Document

The remainder of this document is organized around two principal sections:

(1) a summary of key findings organized in terms of major phases of the information product life-cycle: Design; Development and Review; and Dissemination, Maintenance, and Feedback; and (2) a concluding section that highlights

implications for the Office of Environmental Information and recommendations

for the success of EPA information products.

The document also includes three appendices: (A) Summary of Information

Products Considered, (B) Advisory Committee Members, and (C) Interview

Questions.

Although all

public communications are

important and

require

thoughtful

formulation,

data-based

information

products

present a

special set of

challenges.
Page 8 November 17, 2000

Exhibit 1. Secondary Use of Environmental Data:

How Data Become Information Products

Most EPA data are initially collected or generated by first-order data

providers, usually industry or other regulated entities and states. Data are

then transferred from the original provider to EPA/state data systems,

where they are used to support various regulatory and program administration activities. The data collected by EPA/state programs can also be used

or adapted for use in secondary applications. Secondary applications

include integration, consolidation, analysis, and context-specific uses of the

original program data. The offices within the Agency, particularly the

Office of Environmental Information (OEI), undertake various information

development and enhancement initiatives, resulting in “secondary information products,” which are used by a wide range of decision-makers at the

local, state and national level. Use of these information products lead to

environmental and/or right to know outcomes. As suggested by the schematic (above), feedback from information users and decision-makers helps

to improve information products. The process of moving from data to

effective information products is inherently iterative and requires coordination with data providers and stakeholders.
November 17, 2000 Page 9

2. KEY FINDINGS

As illustrated in Exhibit 2, the

information product life-cycle can

be characterized in terms of six

aspects: design, development,

review, dissemination, maintenance, and feedback. Findings

from the interviews suggest

important overlaps among these

phases. For this reason, findings

have been combined and presented in three basic groups:

design; development and review;

and dissemination, maintenance

and feedback.

Placement of findings within the life-cycle framework will help to guide future

product developers in a logical and progressive manner. As previewed below,

the lessons derived through this study highlight the importance of a life-cycle

approach to information product management.

• Design. The design stage of an information product includes defining the

purpose and audience, aligning product scope and user needs, involving

stakeholders, coordinating with other relevant efforts, defining data

suitability criteria, addressing resource issues, and developing a product

plan.

• Development and Review. The development phase requires input from

customers, stakeholders, and data providers; consultation with data

providers to assure suitability and accuracy; and configuration of products that are consistent with metadata.

• Dissemination, Maintenance, and Feedback. The dissemination stage

goes beyond merely providing access to the product to ensuring timeliness, providing metadata, providing contextual information, and seeking

and utilizing continuous feedback from users and stakeholders.

2.1. Design

The majority of interviewees strongly recommend adoption of a strategic

approach in the information product design phase. Accordingly, information

project managers should carefully consider the goal of the project, articulate

intended audience and user needs, determine criteria to ascertain whether data

collected for one purpose are suitable for other purposes, and identify possible

Exhibit 2. The Six Principle

Aspects of the Information

Product Life-Cycle

1. Design

2. Development

3. Review

4. Dissemination

5. Maintenance

6. Feedback
Page 10 November 17, 2000

linkages with Agency partners with whom to collaborate on product development. Interviewees explain that early strategic planning allows project managers

to better anticipate and respond to potential changes in a project’s budget, goals,

and target audience.

Identify Product’s Purpose and Audience

Almost all interviewees agree that it is critical to articulate a clear sense of

the product’s purpose early in the design process. As obvious as this may

seem, interviewees stress that this step is frequently overlooked, resulting in

redundancy among products, suboptimal user functionality, and other problems. As part of this process, it is important to identify primary audiences for

the product, key stakeholders, and potential sponsors within and beyond the

Agency (see Exhibit 3). Clear understanding of the purpose and the audience

will help determine the methods used to compile and present the information

for dissemination. Consequently, front-end goal articulation and audience

identification are critical to the success and usability of an information

product.

Align User Needs and Product Scope and Character

Most interviewees recognize the importance of delivering a product that

meets the specific needs of an intended audience. Customer needs vary

depending on the users’ levels of proficiency in data use or analysis and their

ultimate plans for the information. Interviewees caution that the actual

Exhibit 3. Stakeholders, Customers, and the Public

All of EPA’s information products are

developed and disseminated to serve

the public’s right to know about

environmental conditions and trends.

However, it is helpful to distinguish

between “general public,” “stakeholders,” and “customers” for an

information product. Customers are

individuals or groups who will use

the product. Information products should be developed to address the

needs and capabilities of customers. Stakeholders are individuals or

groups who will likely be affected by the use of the product. Stakeholders

are frequently the original providers of data used in developing an information product. Although it is important to distinguish between stakeholders and customers, it is also important to recognize that stakeholders

and customers sometimes overlap.
November 17, 2000 Page 11

audience frequently extends beyond the intended audience. This requires

managers to consider all potential users when defining the scope of a project.

For instance, IWI was originally targeted at states as a tool to spark placebased discussion of water quality issues. Project managers recognized its

potential value to the public, and expanded the scope of the project accordingly. While the reality that information products circulate beyond their

intended audience should be considered during planning, it should not be

used as a rationale for avoiding the need to specify a target audience and its

unique needs for environmental information.

Interviewees emphasize that product developers should take care to distinguish between a user’s need for raw data, interpreted results, or a summary

report. They stress that product managers should adjust the degree of data

interpretation according to the level of expertise of the expected audience.

Interviewees also note that information products should avoid certain types

of interpretation. Specifically, the Agency has often run into difficulty when

attempting to rank facilities or areas based on available data. Interviewees

stress that information products should not rank aspects of their information

(e.g., facility compliance, relative stream quality, air quality variance, etc.).

Instead, they should provide information that enables users to conduct such

analyses on their own.

During the interviews, information product managers and designers reported

on ways they address the issue of data interpretation when required by their

products. Examples include:

✓ Informing stakeholders of the expected level of interpretation in the

early phases of a project (PA CRS);

✓ Enabling users to adjust the algorithms that calculate ranking, risk, or

similar assessments (IWI); and

✓ Targeting projects to local communities interested in specific sets of

interpreted data (EMPACT).

Early Stakeholder Identification and Involvement

In addition to identifying and involving a product’s audience, interviewees

indicate that product teams should identify essential stakeholders and involve

them early in the design phase. This distinction between users and data providers will vary and even overlap depending on the product, but interviewees felt

it was essential for product developers to consider the needs of both the external user and the identified stakeholder group(s). This frequent overlap can be

easily understood if adequate time is devoted to identifying the various parties

(see Exhibit 3). Oftentimes, product developers consult only stakeholder

groups, or “user groups”, as a means of soliciting feedback. Some interviewees
Page 12 November 17, 2000

suggest that such limited external input is not only inadequate, but that feedback is often misleading and perhaps even related to the perceived failure of an

information product by the user community.

Early stakeholder involvement is especially important as it can help clarify

goals, reduce skepticism, build consensus, and ensure the dissemination of

information that meets audience needs. Exhibit 4 demonstrates how the scope

and nature of stakeholder input can vary across the product life-cycle.

Interviewees stress that while the initial cost of engaging stakeholders may

exceed perceived benefits, in the end actual benefits will be higher than the

initial cost. Resources may be utilized for product development and implementation rather than to justify and defend the information product to stakeholders.

The PA CRS provides a good example of the importance of involving stakeholders, especially when the product is perceived as controversial and faces

criticism or resistance. PA CRS managers met frequently with industry

representatives clearly indicating their intentions regarding the final product

while being receptive to industry’s concerns. Despite initial doubts, the

affected industries participated in the design of the final product.

While interviewees advocate early involvement of stakeholders, they also

believe there are limits to helpful stakeholder input. One respondent indicates that Agency experts need to spend more time determining what information they want to collect and why, before inviting stakeholders into the

process. He believes that getting divergent opinions before having a clear

Exhibit 4. Phases of Stakeholder Input:

Diameter of circles suggests comparative degree of input
November 17, 2000 Page 13

strategic plan results in confusion. The majority of interviewees agree that

asking structured questions about a product results in the most manageable

and useful form of feedback.

Determine Criteria to Ascertain Data Suitability

As discussed previously, many environmental information products tend to be

data rich. It is therefore important to ensure that data utilized for these applications are suitable and appropriate for the anticipated use. The suitability of data

may be determined by addressing the following questions:

What does the database cover? It is important to identify the kinds of

information contained in the database (e.g. types of pollutants, facilities,

permits, and acquisition methods).

Can the database be used for spatial analysis? Product developers should

determine whether the data are available at appropriate geographical scales

such as ZIP code, latitude and longitude, county code, state code, etc.

Can the database be used for temporal analysis? Product developers should

determine whether the data are collected on a fixed schedule (daily,

monthly, yearly, etc.) and if the data are sufficiently consistent over time to

allow period-to-period comparisons.

How consistent are the variables over space and time? It is crucial to assess

the degree of internal consistency, allowing comparisons across space

(facilities, Regions, etc.) and over time (monthly, yearly, etc.)

Can data be linked with information from other databases? Product developers should determine the degree to which information can be linked with

other databases based on common characteristics such as facility identification numbers, latitude and longitude, geographical codes, etc.

How accurate are the data? Product developers should seek and consult

information from data quality checks performed by the Program Office as

well as from statistical analysis performed by other organizations.

What are the limitations? Each of EPA’s databases has a primary purpose

for which it was developed and is maintained. As the databases are assessed for suitability for alternate uses, it is important to understand the

constraints and limitations of the database.

How can I get information? It is crucial to identify formats in which the

database is available such as printed form, diskettes, CD-ROM, online

access, etc., along with names, addresses, and phone numbers to contact for

detailed information.
Page 14 November 17, 2000

Is there documentation? A quality database requires documentation to

support it, such as information on data collection methods, quality assurance mechanisms, data management, users’ guides, and information dissemination. Details on the availability of such documentation should be

provided.

Coordinate with Other Relevant Efforts

Interviewees feel it is important to ensure that information product development is coordinated among similar efforts underway elsewhere at EPA, other

federal agencies, or states. Adequate coordination will limit duplication of

efforts, increase the utility of EPA data, and help ensure that Agency goals

are being addressed. Several interviewees suggest that establishing Agencywide procedures will result in a stronger relationship between information

products and EPA’s goals, providing managers with a meaningful context

from which to measure product performance.

One interviewee emphasizes the importance of identifying external partners

who are committed to the project and have the expertise to provide constructive assistance, especially in the early design phases of an information

product. Effective partnerships between similar efforts can also help overcome

budget constraints, promote data sharing, and increase data quality. Exhibit 5

portrays an illustrative partnership between EPA and a regulated industry.

EPA’s Environmental Monitoring for Public Access and Community Tracking (EMPACT) Program, for example, provides funding to initiate community-level projects and encourages local partners to assume financial

responsibility after EPA funds are exhausted. Data quality is enhanced

through reviewing methods and sharing data by experts involved in similar

efforts. Sharing information and coordinating efforts should go beyond the

Agency to other federal agencies, as developers of IWI indicate.

Address Resource Issues

Resource considerations play a significant role in determining project success. This fact is especially true for information products with relatively less

up-front Agency funding or support. Products mandated by legislation or

regulation are often better funded than products that stem from user feedback, internal collaborations, or community-based partnerships. While a

clearly defined budget should be developed in the early stages of the product

life-cycle, it is necessary to maintain flexibility and revise the budget on

encountering unexpected contingencies. Perhaps most importantly,

interviewees stress that information product development requires a long-term

commitment, suitable to support a product as it evolves through the design,

development, stakeholder review, and dissemination and feedback phases.
November 17, 2000 Page 15

Budget planning should clarify both resource needs and availability. The

definition of resource needs should extend beyond the early stages of product

development and include all aspects of product design, development, and

dissemination and feedback. While specifying resource availability, product

planners should earmark initially available funds from the Agency as well as

identify potential funding sources, internal and external to the Agency, to

ensure successful completion of the project. As an example, developers of

the South Baltimore Pilot Project note that they established their goals based

on the size of the initial budget in order not to take on too much at one time.

Even so, the project exceeded its budget and funds have not been made

available to publish all findings.

Develop a Product Plan

Several product developers interviewed for this study emphasized the need

for a product plan, which would formally contain strategic planning considerations relating to information product development and management. The

product plan would contain three key elements:

• A summary of current circumstances, including explicit user-focused

rationale for the product;

• A schedule and process description dealing with how the project team

will approach each aspect of the product life-cycle (see Exhibit 2); and

• A clear statement of the outcomes expected to result from the product.

The plan should also address key tactical issues associated with the use of

existing data, such as data appropriateness and accuracy, methodological

issues, peer review procedures, data display and access issues, and general

management of product content.

In some cases, EPA obtains data from regulated industries, often through

other government agencies (frequently at the state level). Under such a

configuration, EPA and its state and/or industry partners can collaborate on

efforts to improve public access and usability.

Exhibit 5. Illustrative Information Partnership
Page 16 November 17, 2000

2.2. Development and Review

Traditionally, resources and activities tend to be concentrated on activities associated with the actual development of an information product. The focus is on

methodological issues, testing, and review of the product. Interviewees express

concern about this limited field of activity and articulate several issues requiring

increased attention during the development phase of the product life-cycle.

Obtain Stakeholder Input During Product Development

As with product design, interviewees stress the importance of stakeholder

input during product development. However, participants are divided on

whether to seek open-ended or structured stakeholder input. Some

interviewees indicate a preference for soliciting feedback on developed

concepts rather than utilizing an entirely open-ended, unstructured approach.

Representatives from TRI and the SFIP note the importance of thoroughly

developing concepts and product drafts prior to soliciting stakeholder feedback, arguing that such an approach leads to more constructive feedback.

Other interviewees contend that stakeholder input should be gathered to

affect the product revision and redesign that tend to occur during development of product components. In one example, IWI managers worked with

states and other stakeholders to complete a review of the overall program,

the rankings, and the underlying methodology – a process that resulted in

valuable contributions to the final product.

Recognize the Relationship between Data Acquisition, Use, and Quality

Data ownership is directly related to data quality concerns. While some users

of secondary information products take the initiative and communicate

effectively with originators of data, EPA lacks an organized framework to

guide this kind of technical interaction. Many interviewees stress the importance of developing a mechanism to facilitate communication among producers and users of EPA data. This is especially important when data are being

used for purposes other than those of their original collection. Respondents

also express a desire for Agency-wide guidelines on conducting quality

control protocols on data prior to using it for a secondary purpose.

An important question for interviewees is, “who should be responsible for

ensuring data quality?” In many cases the concept of data quality is used

casually by information product developers and designers as a way to refer to

data accuracy and error correction procedures. This interpretation provides

only a partial understanding of data quality, and does not sufficiently emphasize the fact that quality is a function of intended use. What one user considers

data of high quality, another might deem wholly inadequate. A user’s perspective on data quality often extends beyond data accuracy into other data characteristics. For instance, is the data generalizable (for example, was it collected
November 17, 2000 Page 17

using a probabilistic sampling model? Is this method for collection similar to

other data sets?), is the data current (annual, monthly or periodic updates), or

are there reproducible results (how thorough is the metadata and documentation information)? This is particularly relevant in cases where users must work

with data they did not collect. In these situations, it can be presumed that users

are not aware of the purpose behind the original data collection. Interviewees

emphasize that information product developers must obtain and examine all

available documentation, guidelines, and contextual information to assure that

data are suitable for use in new, possibly unintended applications.

Improve Data Accuracy with Input from Data Providers

To address concerns over data accuracy, developers and managers of SFIP,

IWI, and the South Baltimore Pilot Project note the importance of a “data

feedback loop,” through which facilities and other data providers are requested to verify data used in a particular information product. In addition to

assuring accuracy, such a procedure helps address concerns that information

providers might have regarding the projects. Beyond sending the data back to

providers, interviewees feel that the Agency should also consider initiating

internal review procedures or a comment period via an intranet-accessible

form, allowing data to mature internally prior to public release. For example,

one respondent explained that his managers wanted an assessment of the

quality of the data used in Enviroviz. Lacking established Agency standards,

he developed a 3-level set of standards that he could then apply to data used

in Enviroviz.

Continuously Involve and Utilize Feedback from the User and Information Provider

Feedback from product users and information providers is essential to develop useful products and improve data quality. Most interviewees stress the

importance of eliciting feedback about data quality, usability, and clarity

from product users and information providers. As illustrated in Exhibit 6,

interviewees recognize that the need for input from users and data providers

will vary depending on product type and affected parties.

2.3. Dissemination, Maintenance, and Feedback

The ultimate success of an information product is reflected by the reaction of

its audience. Interviewees emphasized issues relating to user satisfaction

such as timely release, data quality, metadata, contextual information, and

continued solicitation of feedback.

Remain Committed to Information Availability in a Timely Manner

Interviewees stress the importance of making environmental information

available in a timely manner. When the research team began this project, a
Page 18 November 17, 2000

primary concern was the degree to which Agency data managers might be

hesitant to release information for fear of its inappropriate use. For example,

some non-government organizations may use data from TRI or the Air Trends

Report and include it in area rankings in a manner that EPA would not support.

The research team raised this issue with each interviewee. In almost all

cases, they indicate that they recognize the problem, but view it as beyond

the Agency’s control. Some respondents suggest that managers should make

an effort to anticipate secondary uses and do their best to consider them as

they complete product development, while recognizing the impossibility of

foreseeing all possible uses for Agency data. Above all, interviewees said

that the Agency cannot let this concern interfere with the timely dissemination of information.

Rather, the Agency should work to ensure that products are derived from

suitable data and are packaged with sufficient metadata and other descriptors

of appropriate use to minimize the misuse of the primary data. All data-based

information products should include relevant metadata (i.e., data about the

data) and clearly identify the intended use of the product to minimize, if not

eliminate, possible misuse of data. Agreed-upon data quality and suitability

goals can be integrated into a product’s design and facilitate timely release.

Approach Dissemination as an Opportunity to Improve Data Quality

Some interviewees argue that data quality will improve through widespread

dissemination and use. This group argues that users will offer feedback, and

in turn, Agency commitments to the public’s right to know will not be

compromised with lengthy internal reviews. The impetus for this view is

based on the following logic: improved access and display will lead to

Exhibit 6. Affected Parties by Product Types

Affected Parties

Regulated community

Regulated community,

state and local governments

Regulated community,

state and local governments,

interested members of the

general public

Sample Product Types

Models to estimate emissions,

calculate compliance costs, and

identify substitute inputs

Software for inputting data or

transferring data to EPA

Status and trends tool

Role

Users

Data

providers

Data

providers

and

users
November 17, 2000 Page 19

increased use, increased use leads to increased scrutiny, increased scrutiny

will identify discrepancies, and the Agency will respond by correcting errors

and/or clarifying misperceptions.

Provide Customers with Easy-to-Understand Metadata

The term “metadata” refers to information about a data-based product’s

content, technical configuration, and the methods used for analysis and/or

data collection. Metadata is needed so that users can assess data quality and

appropriate use in relation to their own needs. The development and placement of adequate metadata within information products is a particularly

important issue for EPA because secondary users constitute a significant

audience for the Agency’s information products. While EPA is making

widespread public access to environmental information a reality, it is critical

that all information products provide sufficient documentation for their

intended audience.

Furthermore, by defining data suitability objectives and making them available as part of the metadata, product designers can avoid potential misunderstandings and clarify the known quality of the data. For example, SFIP

managers received several complaints about displaying out-of-date data;

however, since SFIP is only updated once every quarter to half-year, users

should expect some time lags. Therefore, it is important to indicate how

often the data are refreshed or updated using simple, non-technical language.

Provide Sufficient Context when Releasing Data

Beyond the provision of metadata, several interviewees suggest that, where

appropriate, products need to include an overarching statement of context as

another way to orient users and avoid potential misinterpretation. For example, several interviewees express concern that releasing monitoring data

may lead to the wrong impression that states with more comprehensive

monitoring or enforcement programs have more serious environmental

problems than other states that collect and/or maintain comparatively less

data. Interviewees suggest that a solution might be to include information

describing the extent and nature of a state’s monitoring network(s). This

concern extends beyond a previous finding which calls for adequate

metadata with information products.

A similar problem arises whenever new or enhanced monitoring is implemented or displayed. As one interviewee explains, methodological changes

for estimating emission trends in the Air Trends Report resulted in data and

graphs that did not look the same as they had in the past. This caused some

readers to mistakenly infer changes in environmental outcomes. He suggested that increased attention to contextual information would help to avoid

such misinterpretation.

Improved

access and

display will

lead to increased use,

increased use

leads to

increased

scrutiny,

increased

scrutiny will

identify discrepancies,

and the

Agency will

respond by

correcting

errors and/or

clarifying

misperceptions.
Page 20 November 17, 2000

Seek and Facilitate Feedback and Continued Communication with Data

Users and Providers

Communication should extend past the design and developmental phases of

an information product, and continue after the product’s release. Maintaining

communication with data providers will enable product designers to solicit

and receive corrections and updates. Several interviewees note that the

realization that information will be analyzed and distributed is a motivating

force behind efforts to improve data quality. For example, IWI currently

includes a designation noting cases when insufficient data exist to draw a

conclusion about water quality. When states or other users see the designation applied to their information, they often seek to increase reporting and

provide additional data, therefore improving data quality.
November 17, 2000 Page 21

3. IMPLICATIONS FOR OEI

The EPA Office of Environmental Information acts as the steward of the

Agency’s environmental information, ensuring that EPA and other concerned

organizations collect, develop, and disseminate high-quality and useful information about environmental conditions. Although focused on serving the

public’s right to know about environmental issues, OEI aims to move beyond

the provision of existing data and generate new information products that are

responsive to identified user needs. Increasingly, environmental information

products are derived from data originally collected to support state or federal

regulatory programs or management activities. Assuring the suitability of these

data for new applications is therefore of paramount importance.

After completing the interviews with information product designers and managers, the research team focused on identifying a set of overarching lessons

learned and their implications for the Office of Environmental Information.

Summarized below, these lessons will serve as the basis for a series of “Best

Practices” reports. The best practices will help EPA product developers and

managers assure that information products are useful, credible, and fully applicable to user needs and access preferences. Organized around the themes

outlined below, the best practices series will compile existing knowledge and

documentation, and when necessary, introduce new methods and approaches to

assure that EPA information products are always “on the mark” with respect to

user needs and environmental decision making.

Avoid Defining Your Audience as “The Public”

EPA is currently striving to expand and enhance its role as a communicator

of environmental information. OEI should develop guidelines to help analysts integrate product performance goals with the needs of specific target

audiences. Such guidelines would help the Agency anticipate how the utility

of a particular product might vary from group-to-group. Depending on the

audience, developers can incorporate various types of software formats,

technical language, embedded caveats, collection context, and helpful hints.

Beyond the presentation of information, the rate of error correction or product updates could also depend on the intended use and corresponding audience. Issues such as these are easy to overlook when the external audience is

classified as “the general public.”

Data Quality is a Function of Data Use

Data quality is not a generic concept. It is critical that data quality be considered in light of its intended use. The quality of data is a function of its characteristics and limitations in the context of a particular application. OEI

should develop guidelines and resources to help product planners and

Agency analysts judge the suitability of data for particular user applications.

Without best practices concerning the suitability of existing data for anticipated applications, the credibility of EPA information products may suffer.
Page 22 November 17, 2000

Assure Credibility with Methods Review and Documentation

A standardized approach to assure that information products are planned and

developed using appropriate analytical tools, documentation, and review

procedures will help to increase the credibility and usability of information

products. Although all EPA products, prior to dissemination, are subject to

review by the originating office’s designated communications officer, OEI

should develop Agency-wide guidelines for methods review and documentation of the development of an information product. The latter is especially

important as it assures replicability of results.

Develop Guidelines for Stakeholder Involvement

Depending on the nature of the information product and its intended use, the

appropriate type, scope, and timing of stakeholder participation can vary

significantly. Input can range from comments on both the products themselves

and/or the raw data contained within a product. OEI could help develop guidelines for stakeholder involvement, focussing on how input might vary with

level of product maturity, phase of the product life-cycle, and other key factors.

Information Customers Need Metadata

OEI has a lead responsibility in developing Agency-wide guidelines for

metadata to be included with information products. Increased attention to

preparing metadata should be a key component of the Agency’s public access

efforts as it is critical to ensuring the development of useful and credible

information products and assists secondary uses of Agency information.

Facilitate Life-Cycle Information Product Planning

The factors discussed above must be approached in an integrated manner. As

a final step, OEI should develop comprehensive information product planning procedures to assist program offices in evaluating the long-term goals

for designing and maintaining information products. The guidelines should

help managers consider target audiences, appropriate stakeholder involvement, quality assurance procedures, budget requirements, and other key

product development and maintenance factors. Resources developed by OEI

should be formulated and couched to help planners address the full life-cycle

of an information product (see Exhibit 2).

The Upshot: A Need for “Best Practices”

The Environmental Protection Agency strives to gather, utilize, and provide

reliable and timely data to make decisions, improve information management,

document and measure performance, reduce reporting burdens, and enhance

public access. OEI’s OIAA sought to reexamine past information product

efforts in order to delineate lessons learned that may be helpful to future endeavors of information product design, development, and dissemination as they

feed into these overarching themes. The interview process did not result in any
November 17, 2000 Page 23

“magic bullets” to assure perfect information products. Interviewees did, however, recommend a variety of practices that tend to contribute to the effective

design, development, and dissemination of information products. The individuals interviewed for this project share decades of experience working with

environmental information products. They shared both successes and failures,

what to do as well as what to avoid. Overall, the findings demonstrate that EPA

has a wealth of information product experience that is not being adequately

communicated to information product developers who could learn from it.

However, beyond information sharing and technology transfer, a perceived gap

exists for guidance on sound analytical approaches to the reoccurring data

concerns, specifically for information product applications.

As a result, OEI will now embark on the Best Practices Series, a series of Agency

guides on information product development. The purpose of this series is to help

information product developers create products of known quality and usability for

an external audience. In doing so, OEI and partners will strive to heighten Agency

information-sharing and technology transfer and reduce internal burden, thereby

enhancing the credibility, usability and availability of EPA products and ultimately providing more information of higher quality to the user. Analysis of the

information captured through this interview process suggests that best practices

tend to reinforce one another. Not only is it important to articulate and heed best

practices, it is important to do so in an integrated manner. Depicted in Exhibit 7,

below, adoption of an integrated approach to information product management

would help to ensure continuous improvement, enhance the public’s right to

know, and support the mission of the Office of Environmental Information.

The best practices series

will also help EPA and state

program managers as they

oversee regulatory and

administrative data systems,

helping them to anticipate

new uses for their data. In a

similar vein, the best

practices will be useful to

grantees and other external

stakeholders who collect

environmental data, assuring the ever increasing

suitability of data for

unforeseen applications and

the overall usability of EPA

information.

Not only is it

important to

articulate and

heed best

practices, it is

important to

do so in an

integrated

manner.

Exhibit 7. Elements of a Strategic

Information Product Planning Process
Page 24 November 17, 2000

Project and Category

Environmental Monitoring

for Public Access and

Community Tracking

(EMPACT)

statute or regulation

(Executive Order)

Index of Watershed

Indicators (IWI)

data management and

information dissemination

National Air Quality and

Emission Trends Report

statute or regulation

Pennsylvania Compliance

Reporting System (PA CRS)

compliance focus

Contact(s)

Denice Shaw

Charlotte

Cottrill

Charles

Spooner

David Mintz

Tom Curran

Kim Nelson

Information Product Description

EMPACT sponsors and assists with local

projects that collect real-time and timerelevant information about ambient

environmental conditions such as water

quality and ozone levels. At present, there

are EMPACT projects in 86 metropolitan

areas. See also http://www.epa.gov/empact/.

IWI is a web-based product that displays

water quality data on regional and national

levels. The data displayed on the maps

reflect the current conditions and

vulnerability of water resources. IWI is

targeted at both states and the public. The

data come from a variety of sources

including OW partnerships with USGS and

NOAA. See also http://www.epa.gov/iwi/.

The Air Quality Trends Report assesses

national and regional trends in air quality

and monitored emissions. OAR uses the

report to gauge air quality, pollution

reduction, and pollutant levels as a part of

the Clean Air Act (CAA). Sections include

discussion on air quality trends for criteria

pollutants, air quality on local levels, the

status of areas that do not meet NAAQS

standards, air toxics, visibility trends, and

acid deposition. The report is available in

hard copy or as a PDF file from the OAR

web site. See also http://www.epa.gov/oar/

oarpubs.html.

Pennsylvania designed PA CRS to collect and

display compliance information for

Department of Environmental Protection

(DEP) employees. As part of a shift in DEP

policy away from enforcement and towards

compliance assistance, PA CRS measures and

reports on ongoing compliance efforts. While

the reporting system itself is for internal use

only, results are reported on the web for the

public. See also http://falcon.state.pa.us/crs/

web/crs_interface.frames_start.

APPENDIX A  Information Products

Discussed
November 17, 2000 Page 25

Information Product Description

Enviroviz displays compliance, environmental quality, budget, and GPRA information

on maps and graphs using two and threedimensional graphics. Users can “brush”

certain features to retrieve metadata about

the monitoring station, data collection

method, numerical data, specific pollutants,

and other details. Users can view information by region, state, congressional district,

county, or watershed. Enviroviz is not yet

available beyond the pilot project.

SFIP is a web-based pilot project that

integrates environmental compliance data

from five different industrial sectors including automobile assembly, iron and steel

production, pulp manufacturing, petroleum

refining and nonferrous metals smelting and

refining. It reports on approximately 650

facilities. Its primary purpose is to centralize

and increase accessibility to facility information from a range of EPA databases.

See also http://es.epa.gov/oeca/sfi/.

This project represents a community-based

approach to environmental protection and

economic development. When complete, it

will provide South Baltimore communities

with risk-based environmental information

on a variety of areas including cancer, parks,

abandoned housing and trash, economic

development, and air and water quality.

Volunteer teams of citizens and environmental, industry, and government representatives

coordinate different project components. See

also http://www.epa.gov/opptintr/cbep/

baltodex.htm.

Mandated by the Emergency Planning and

Community Right to Know Act (EPCRA) of

1986, TRI displays information about

chemicals designated as potentially hazardous. Facilities that meet TRI requirements

must report on their estimated chemical

releases, transfers, waste treatment, pollution

prevention, and recycling activities. TRI data

are available in web and CD-ROM format as

well as in various information packets and

reports. See also http://www.epa.gov/tri/.

Contact(s)

Alvin Morris

Rich Paiste

Mike Barrette

Henry Topper

Steven

NewburgRinn

Odelia Funke

John Melone

Project and Category

Region III Enviroviz

data management and

information dissemination

Sector Facility Indexing

Project (SFIP)

compliance focus

South Baltimore Pilot Project

stakeholder involvement

Toxics Release Inventory

(TRI)

statute or regulation
Page 26 November 17, 2000

Date Interviewed

August 30, 1999

August 12, 1999

August 18, 1999

September 28, 1999

August 24, 1999

September 3, 1999

September 8, 1999

August 26, 1999

September 13, 1999

Name

George Bonina

Office of Environmental Information (OEI)

Tom Curran

Office of Air and Radiation (OAR)

David Davis

Office of Water (OW)

Steve Goranson

Region V

Barnes Johnson

Office of Solid Waste and Emergency Response

(OSWER)

Alvin Morris

Region III

Charles Spooner

Office of Water (OW)

Frederick Stiehl

Office of Enforcement and Compliance

Assurance (OECA)

Henry Topper

Office of Prevention, Pesticides, and Toxic

Substances (OPPTS)

APPENDIX B  Advisory Committee

Members
November 17, 2000 Page 27

Product Identification

• What was the original impetus for developing your project?

• What information need is the product designed to address?

• Who is your primary audience? What methods, if any, did you use to

target your audience or get feedback during the development process?

and

• Have you included stakeholder input in the development of your product?

Product Development

• How much raw data are included in the project? How much interpretation? What kind of metadata do you make available to product users?

• Are you concerned about the respectful use or misrepresentation of the

information included in your product? If so, what do you do to address

these concerns?

• Do you solicit customer feedback? If so, how? Do you target a specific

subset of your users, or do you attempt to reach all users?

• Do you make any effort to coordinate your product with others offered by

EPA through linkages, shared data, or another mechanism? Have you

tried to integrate your product with other databases or products?

• Do you have a process for identifying or soliciting alternative reporting

methods for data? Do you accept data from external sources? and

• Are there Agency guidelines that dictate how you collect or present data?

Do you submit your data or product to peer review?

Product Management and Future Commitment

• What resources do you currently have available for the production of

your information product?

• Do you have a product evaluation program in place? If so, who are the

evaluators? Has your product changed over time because of such input?

Do you think an evaluation process adds credibility to a product?

APPENDIX C  Interview Questions
Page 28 November 17, 2000

• What kind of political commitment is there to sustaining your product in

your office/in the Agency? Do you think that the current level of commitment will be sufficient to support the continued growth and development

of your product?

• Do you expect any dramatic changes in your future mode of operation or

in the maintenance of your product; and

• What would you do differently if you could redesign this product?