COMPANION USER'S GUIDE:

[A Modular Approach to Analyzing Environmental Management
System]{.smallcaps}

[Projects and Programs]{.smallcaps}

**INTRODUCTION**

Projects involving Environmental Management Systems (EMSs) may be
process or substance-oriented, project or program-based, on a facility
or geographic scale, community-based or nation-wide, or produce systemic
organizational change. In all cases, it is important to determine
whether the project is achieving its intended outcomes, whether it
provides greater process efficiencies or superior environmental
performance compared to standard practice, and to identify ways in which
to improve the project. Program managers must be able to measure and
describe to others the impacts of their programs.

The process of analyzing EMS projects will help practitioners answer
these questions by agreeing on clear goals, developing performance
indicators to track progress, establishing baseline data, setting
targets for future performance and measuring progress toward such
targets. A well-planned and thoughtfully conducted analysis can help
determine if program activities are providing the outcomes needed to
achieve the stated goals. With sound analysis, managers can understand
barriers to an EMS project and modify the program as needed to
accomplish objectives, or modify program goals to set more appropriate
or realistic expectations. Knowledge and insights obtained from analysis
can serve as performance feedback and can play a pivotal role in
mainstreaming EMS projects and policies into everyday work.

The National Center for Environmental Innovation (NCEI) has developed a
series of analysis "modules" to assist staff and program managers to
answer these questions and others throughout the life cycle of an EMS
project -- from pilot testing to broad-scale application of a successful
project. The purpose of this document is to assist users with the
application of the modules. [Each chapter corresponds to the actual
module, each of which may be used independently or in combination with
other modules, and is intended to serve three purposes:]{.underline}

(1) To inform the evaluative process.

(2) To help a manager plan for evaluation at the beginning or
    intermediate stages of an EMS project.

(3) To help serve as a project management/development tool.

The modules can be tailored to meet each of the purposes mentioned above
and the needs of the user. Each module presents an assessment framework,
which can be answered with varying degrees of rigor in order to answer
the questions for his/her purposes. The main goal of these tools is to
allow the user to collect data in an organized manner, conduct an
ongoing assessment of the EMS project, and provide the information
necessary to conduct a full evaluation.

**ENVIRONMENTAL MANAGEMENT SYSTEMS**

EMSs are being adopted by a rapidly growing number of organizations in
the United States. An EMS is a practical tool that provides a systematic
approach to environmental management, based on a Plan-Do-Check-Act
cycle. An EMS is a set of management processes and procedures that
allows an organization to integrate environmental considerations into
daily decisions and practices. EMSs include processes for developing and
continuously improving an organization's environmental policy and goals
under all media, and reducing negative regulated and unregulated
environmental impacts,.

Many of the module questions are designed to help evaluate the outcomes
that result when EMSs are used innovatively in a regulatory program. In
April 2004, the Environmental Protection Agency (EPA) issued a *Strategy
for Determining the Role of EMSs in Regulatory Programs*, which can be
found at:
(<http://www.epa.gov/ems/policy/EMS_and_the_Reg_Structure_41204F.pdf>).
This strategy provides guiding principles and a framework to assess what
role, if any, EMSs should play in regulatory programs. Evaluation of the
EMS projects is critical to the strategy and helps EPA weigh future
policy decisions on the use of EMSs in regulatory programs. EPA and
states are interested in whether there are potential benefits to be
gained from incorporating EMS options into the permitting and regulatory
structure. States are already linking rule development and permitting to
EMS. EPA has been increasingly confronting questions about whether EMSs
can be used to improve the efficiency and effectiveness of regulatory
tools and what roles, if any, regulatory tools should play in promoting
EMSs. Continuing research and experimentation by EPA and its regulatory
partners will assist in determining the effectiveness of EMSs in the
regulatory structure and will ensure that EMSs remain a voluntary choice
made by organizations that can lead to improved environmental
performance. This companion user's guide provides additional assistance
with evaluating EMS projects.

**WHAT ARE THE EMS ANALYSIS MODULES?**

The EMS analysis modules are a suite or series of evaluative questions
that provide a framework to evaluate, understand, and share information
on EMS projects. The module questions are a compilation of research
questions, some of which are presented in the aforementioned EPA
strategy and other published work in the field.

Because of the diversity of EMS projects, NCEI designed the evaluation
modules to provide guidance, direction, and flexibility. The questions
are designed to encourage critical thinking and assessment of
environmental data, successes, obstacles, and lessons learned in order
to help the user improve the EMS project. The user is encouraged to
consider the core questions contained in each module, determine their
applicability to the project, make appropriate modifications to the
questions, and gather the data---quantitative, qualitative, or
anecdotal---available to assess progress. **The quantity and quality of
data that the user has and the rigor with which the modules are applied
will determine the quality of the analysis.** Although all of the
modules do not have to be completed, the information collected from all
the modules could offer a more complete picture of whether an EMS
project is working well and highlight areas that need improvement. Each
module is described briefly below. Detailed information on each module
follows in the subsequent sections.

-   **Mapping the EMS Project --** This module provides a systematic way
    to map the logic behind the EMS project by asking the user to list
    the goals, resources, activities, partners/customers, outputs, and
    intended outcomes of the EMS project. This module also gathers
    background information to describe the project, its scope, goals,
    purpose, regulatory and programmatic issues, participants, and
    stakeholders.

```{=html}
<!-- -->
```
-   **Assessing the Environmental Results of the EMS Project** -- This
    module is intended to assist the practitioner in measuring the
    environmental results of the EMS project. Questions regarding the
    establishment of baseline data, environmental indicators, and
    performance measures are included.

```{=html}
<!-- -->
```
-   **Assessing the Costs and Cost Savings of an EMS Project --**
    Questions regarding the economic impact of the project are asked in
    this module. The practitioner begins to gather information necessary
    to conduct a cost-effectiveness assessment.

```{=html}
<!-- -->
```
-   **Enforcement and Compliance Assurance** -- This module is designed
    to assess the practical enforceability of the project. This module
    may require the active participation of federal and state
    enforcement and compliance staff.

```{=html}
<!-- -->
```
-   **Public Involvement** -- The questions contained in this module
    gather information regarding stakeholder/public participation in the
    process.

-   **Assessing the Potential Transferability of the EMS Project** --
    This module sets forth questions that rank the project on a
    five-part transferability scale, with the objective of determining
    whether the EMS project is ready for broad-scale application.

**WHO SHOULD USE THE ANALYSIS MODULES?**

Program managers, designers, and staff participating in the EMS project
should use the analysis modules to focus their thinking about the
process, assess how well the EMS project is working, and assist in
project management and development. Different members of the project
team may be responsible for different modules or different components of
the modules. For example, one member of the team may be in charge of
data collection and management, whereas another member may be the
coordinator of public participation. The modules have been designed so
that EPA or other federal government agencies, state agencies, local and
tribal governments, regulated entities, and the public at large, can use
them for evaluation purposes. The questions in each module may be more
or less relevant depending on who the user is and how the modules are
applied. Another example of a potential user may be a project manager
who is designing an EMS project and decides to use the modules to help
build evaluation into the design of the project.

**WHO IS THE AUDIENCE FOR THE RESULTS OF THE ANALYSIS MODULES?**

Information gathered by using the analysis modules will be a valuable
resource for team members, decision makers within an environmental
agency, participants in the project, the public, and others interested
in the progress of the project. The quality and results of the
information obtained will likely influence decisions on the need for
modifications to the project, or whether the project is ready for
broad-scale application.

**HOW CAN THE MODULES HELP?**

The modules are flexible tools intended to guide and shape decisions and
discussions around important EMS analytical questions. The analysis
modules provide a systematic way to conduct a full evaluation of the EMS
project, collect and analyze data, make adjustments to improve
performance, and organize, track, and monitor the progress of the EMS
project. The analysis modules can help determine whether a project is
working as intended and whether it has the potential for broad-scale
application. As a project management tool, they can be used to raise
important questions and make methodological decisions explicit. Finally,
they can, and should, be modified to adapt to the project and the needs
of the project team.

**AT WHAT PHASE OF THE EMS PROJECT SHOULD THE ANALYSIS MODULES BE
USED?**

The analysis modules are intended for use throughout the project -- to
help design a project for analysis at the planning phase, during
implementation, when the pilot experimentation is complete, and to
inform a formal evaluation. For example, at the planning phase of an EMS
project, a team member may describe the monitoring information that will
be used to assess performance, the parties responsible for collecting
and verifying the data, and the frequency and mode of collection. At a
more advanced phase in the project, there will be monitoring data to
analyze and normalize. Table 1: Uses of the Analyses Modules provides
examples of how each of the modules can be used during the three primary
phases of a project -- design and planning, implementation, and maturity
of the project (or completion of a phase of the project).

**Designing and Planning an EMS Project --** In order to design an EMS
project, it is necessary to clearly define the problem that the project
is addressing and then outline how the project will address the stated
goals. The modules provide a series of questions that identify the
following: 1) project goals and desired environmental and behavioral
outcomes; 2) baseline data; 3) performance measures; 4) enforcement and
compliance aspects; 5) public involvement requirements; and 6) how to
plan for the transferability of the project. **It is often difficult to
assess how well the project is working or to conduct an evaluation
without planning for data collection early on in the project design.**

**Implementation of an EMS Project --** An EMS project can be assessed
at different levels. At the very basic level, every project should have
a tracking and monitoring component during project implementation. The
modules provide questions to obtain this data and to help set up an
adequate data collection system. Project tracking and monitoring does
mean that data and information on how the project is working is being
collected according to schedule or protocol, and that the project is
being tracked for progress. The goal of using the modules at this phase
is to be able to pinpoint and address successful elements of the project
and barriers to success. If the project is not meeting expectations in
any one area of importance, the modules can be used to conduct a deeper
level of analysis.

# 

**Assessing the EMS Project at the Termination Point** -- If the
practitioner is looking back at the project to see how it performed, the
modules can be used to conduct a more in-depth analysis. Robust
qualitative and quantitative information gained from each module can
provide a complete picture of how well the EMS project performed in
meeting its stated goals. Each module asks the practitioner to assess
how well the innovation performed relative to the traditional approach.
Determining the relative advantage of the innovation over the
traditional approach is vital to innovation transferability and overall
success.

**Designing a Formal Evaluation** **--** The modules can be used to help
the practitioner design a formal evaluation study of the EMS project.
Evaluation looks at [how]{.underline} well an innovation is working to
achieve its stated outcomes and [why]{.underline} it is working the way
it is. The modules ask the practitioner to construct a logic model
(Module 1), which is an integral first step to doing an evaluation and
to formulating the right evaluation questions. In order to construct the
rest of the evaluation study, the practitioner uses the completed
modules to help address the following questions: 1) to what extent have
the stated outcomes been achieved and why; 2) what aspects of the
innovation lead to those outcomes; and 3) what is the context in which
the outcomes were achieved.

# 

**WHAT KIND OF DATA IS NEEDED TO COMPLETE THE ANALYSIS MODULES?**

Depending on user needs, the module questions may be answered with
anecdotal, qualitative, and/or quantitative data. Generally, more robust
data results in greater certainty in the analysis, or the ability to
characterize the uncertainties or successes of an EMS project in greater
detail. For example, in some cases, an anecdotal reporting of the costs
required to pursue an EMS project may be sufficient, whereas a financial
accounting of capital costs may be necessary for another type of
project. The modules encourage the user to collect anecdotal,
qualitative, and/or quantitative data as important sources of analytic
information. Data collection, however, will depend largely on the needs
of the project and available resources. The modules together can be
powerful evaluative tools if the user ensures that it is supported by
thorough analyses and quality data.

# HOW TO USE THE SIX EVALUATION MODULES

In the sections that follow, each module is described in greater detail,
with guidance provided on its application, the organizational structure
of the questions, useful sources of information, and a methodology for
addressing the content of each module. Each module is attached in
Appendix A. *It is recommended that the user answer the questions in
each of the corresponding modules rather than within the companion
user's guide.* In addition, the practitioner should collect information
from a variety of sources and in an iterative fashion throughout the
life of the EMS project. By returning to the key questions contained in
the modules at different phases of the innovative process to assess
progress, the practitioner will be better informed and can make
adjustments to the EMS project, if necessary. **Since the modules were
designed with a variety of EMS projects in mind, the key to using the
modules is to answer the questions that are most relevant to the
project.**

# HOW TO USE THE SIX EVALUATION MODULES FOR EMSs

Evaluation of EMSs and EMS-type projects do follow the basic
fundamentals of evaluation. It is important to know if the project
works, how well it works, what are its shortcomings, and if the EMS
project provides lessons for other projects. This user's guide and set
of modules provides an additional level of questions to factor in the
characteristics of EMSs and the types of EMS projects that are being
implemented.

The typical characteristics of EMSs provide a basis for identifying
potential objectives (desired outcomes) for EMS-based innovative
projects and potential indicators to measure and assess performance in
those objectives with respect to pre-project baseline conditions. Each
EMS based project reflects one or more of the following areas of
inquiry, which are covered by the six evaluation modules:

-   **Environmental performance:** (*for regulated entities and/or third
    parties such as suppliers, customers, and environmental quality
    trading partners*) Assess whether an EMS-based project reduces
    significant environmental impacts (as identified in EMS aspect and
    impact analyses) relative to baseline conditions for regulated
    entities and/or third parties.

-   **Cross-media environmental performance:** Determine whether a
    project leads to significant improvements in overall environmental
    quality due to decisions to redirect resources from non- or
    less-significant impact areas to high impact areas, based on EMS
    cross-media analysis.

-   **Regulatory compliance:** Evaluate whether an EMS-based project
    lowers the frequency and/or severity of compliance problems and/or
    improves the ease of determine compliance, relative to existing
    regulatory programs.

-   **Efficiency:** (*for regulatory agencies and/or regulated
    entities*) Assess whether an EMS-based project decreases the time
    and costs of permit development and implementation for regulators
    and regulated entities.

-   **Effectiveness:** (*specifically, the ability of regulatory
    agencies to shift regulatory oversight from lower to higher priority
    areas*) Evaluate whether an EMS-based project allowed a regulatory
    agency to redirect time and resources for regulatory oversight from
    lower to higher priority activities.

-   **Public involvement:** Assess whether an EMS-based project improved
    the quality of information available to the public, the transparency
    of decision-making, and the leverage stakeholders have over
    decisions through changes in the type of public involvement
    mechanism, and the scope, content, timing, and amount of information
    accessible to the public.

+--------+---------+---------+---------+---------+---------+---------+
| # Ta   |         |         |         |         |         |         |
| ble 1: |         |         |         |         |         |         |
|        |         |         |         |         |         |         |
| #      |         |         |         |         |         |         |
|  USES  |         |         |         |         |         |         |
| OF THE |         |         |         |         |         |         |
|  ANALY |         |         |         |         |         |         |
| SIS MO |         |         |         |         |         |         |
| DULES  |         |         |         |         |         |         |
+--------+---------+---------+---------+---------+---------+---------+
| **     | *       | *       | *       | *       | *       | *       |
| Phases | *Module | *Module | *Module | *Module | *Module | *Module |
| of the | 1:**    | 2:**    | 3:**    | 4:      | 5:**    | 6:**    |
| EMS    |         |         |         | Enfo    |         |         |
| Pro    | **      | **As    | **As    | rcement | *       | **As    |
| ject** | Mapping | sessing | sessing | and     | *Public | sessing |
|        | the EMS | the     | the     | Com     | Invo    | the     |
|        | Pr      | Enviro  | Costs   | pliance | lvement | Po      |
|        | oject** | nmental | and     | Assu    | and     | tential |
|        |         | Results | Cost    | rance** | Stak    | T       |
|        |         | of the  | Savings |         | eholder | ransfer |
|        |         | EMS     | of the  |         | Fee     | ability |
|        |         | Pr      | EMS     |         | dback** | of the  |
|        |         | oject** | Pr      |         |         | EMS     |
|        |         |         | oject** |         |         | Pr      |
|        |         |         |         |         |         | oject** |
+--------+---------+---------+---------+---------+---------+---------+
| **     | -   I   | -   I   | -   I   | -   I   | -   I   | -   I   |
| Design | dentify | dentify | dentify | dentify | dentify | dentify |
| and    |         |         |         |     the |     key |         |
| Plan   |   goals |  enviro |   types |         |         |    data |
| ning** |         | nmental |     of  |    moni |   parti |     ne  |
|        | -   I   |         |         | toring, | cipants | cessary |
|        | dentify |   goals | savings |     rep |         |     to  |
|        |     p   |         |     and | orting, | -   I   |     de  |
|        | artners | -   I   |         |     and | dentify | termine |
|        |     and | dentify |   costs |         |     a   |     r   |
|        |     cu  |     f   |     ass |  record | pproach | elative |
|        | stomers | easible | ociated | keeping |     for |     ad  |
|        |         |         |         |         |     e   | vantage |
|        | -   I   |    meas |    with |   requi | ngaging |         |
|        | dentify | urement |         | rements |         | -   I   |
|        |         |     a   | project |         |   stake | dentify |
|        |   tools | pproach |         | -   I   | holders |     a   |
|        |     to  |         | -   I   | dentify |         |         |
|        |         | -       | dentify |     the | -   De  |    path |
|        |  assist |   Chara |     who |         | termine |     to  |
|        |         | cterize |     is  |  method |     re  |         |
|        | project |     b   |     in  |     of  | sources |    diss |
|        |         | aseline | curring |         |     av  | eminate |
|        | -   I   |         |     the |    dete | ailable |         |
|        | dentify | -   I   |         | rmining |     for |    info |
|        |         | dentify | savings |     com |     add | rmation |
|        |    prel |         |     and | pliance | ressing |         |
|        | iminary |    anti |         |         |         | ```     |
|        |         | cipated |   costs |  (i.e., |    stak | {=html} |
|        | drivers |         |         |         | eholder | <       |
|        |     and |  medium |  (i.e., |  record |         | !-- --> |
|        |     b   |     and |     fa  |         |  issues | ```     |
|        | arriers |     lo  | cility, | review, |         | -       |
|        |         | ng-term |         |         | ```     |  Define |
|        | -       |     beh |    gove |    insp | {=html} |     and |
|        |  Create | avioral | rnment) | ection) | <       |         |
|        |     a   |     and |         |         | !-- --> |   focus |
|        |         |         | -       | -   I   | ```     |         |
|        |   logic |  enviro |   Chara | dentify | -   I   | targets |
|        |         | nmental | cterize |         | dentify |     of  |
|        |   model |     o   |     b   |    resp |     po  |     di  |
|        |     for | utcomes | aseline | onsible | tential | ffusion |
|        |     the |         |         |         |         |         |
|        |         | -   I   | -   I   | parties |    stak | efforts |
|        | project | dentify | dentify |     for | eholder |         |
|        |         |         |         |     ve  |         |         |
|        |         |    data |    data | rifying |  issues |         |
|        |         |         |         |         |         |         |
|        |         | sources | sources |    data |  (e.g., |         |
|        |         |     and |     for |     and |         |         |
|        |         |         |         |         |  Enviro |         |
|        |         |    coll | savings |    info | nmental |         |
|        |         | ection/ |     and | rmation |     J   |         |
|        |         |     mon |         |         | ustice) |         |
|        |         | itoring |    cost |         |     u   |         |
|        |         |     pr  |         |         | p-front |         |
|        |         | otocols |    info |         |         |         |
|        |         |     to  | rmation |         |         |         |
|        |         |         |         |         |         |         |
|        |         |  obtain |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         | outcome |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |    data |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         | -       |         |         |         |         |
|        |         |  Set-up |         |         |         |         |
|        |         |     s   |         |         |         |         |
|        |         | chedule |         |         |         |         |
|        |         |     to  |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |  update |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |    info |         |         |         |         |
|        |         | rmation |         |         |         |         |
+--------+---------+---------+---------+---------+---------+---------+
| **Impl | -       | -       | -       | -       | -       | -       |
| ementa |  Modify |  Review |  Review |  Review |   Check | Provide |
| tion** |     to  |         |         |     and |     in  |         |
|        |         |    data |    cost |         |         |  opport |
|        |    acco |     col |         |   track |    with | unities |
|        | mmodate | lection |    info |         |         |     for |
|        |         |     and | rmation |    info |   stake |     po  |
|        | changes |     mon |     for | rmation | holders | tential |
|        |     in  | itoring |         |     to  |     to  |         |
|        |         |         |   compl |         |         |   early |
|        | project | results | eteness | monitor |  assess |     a   |
|        |         |     to  |     and |     com |         | dopters |
|        |    conc |         |     a   | pliance | whether |     of  |
|        | eption, |  verify | ccuracy |     and |         |     the |
|        |         |     ad  |         |     i   |   there |         |
|        |  tools, | herence | -   No  | dentify |     are | project |
|        |         |     to  | rmalize |     p   |         |     to  |
|        |    etc. |     pr  |     and | roblems |    stak |         |
|        |         | otocols |         |     or  | eholder |    part |
|        | -       |         | compare |         |     c   | icipate |
|        |  Review | -   No  |     mid |  trends | oncerns |     in  |
|        |         | rmalize | -course |         |     and |         |
|        |  goals, |     and |         |    that |     the | impleme |
|        |     pa  |         |    data |         |         | ntation |
|        | rtners, | compare |     to  | require |   level |         |
|        |     cus |     mid |     b   |     mid |     of  | -       |
|        | tomers, | -course | aseline | -course |         |    Comm |
|        |     d   |         |     to  |         |  partic | unicate |
|        | rivers, |    data |     de  |    corr | ipation |         |
|        |     b   |     to  | termine | ections |         |   early |
|        | arriers |     b   |         |         | -       |         |
|        |         | aseline |    need | -       |  Assess | results |
|        |    etc. |     to  |     for |  Review |         |     of  |
|        |     ide |     de  |     mid |         |   avail |     the |
|        | ntified | termine | -course |   requi | ability |         |
|        |     in  |         |         | rements |     of  | project |
|        |     the |    need |    corr |         |         |         |
|        |     p   |     for | ections |    esta |    info | ```     |
|        | lanning |     mid |         | blished | rmation | {=html} |
|        |         | -course | -       |         |     to  | <       |
|        |   stage |         |  Review |  during |     the | !-- --> |
|        |         |    corr |         |         |         | ```     |
|        | -       | ections |    proj | impleme |  public | -       |
|        |  Review |         | ections | ntation |         | Analyze |
|        |         | -       |     ide |         | ```     |         |
|        |   logic |  Review | ntified |         | {=html} | project |
|        |         |         |     in  |         | <       |     for |
|        |   model |    data |     the |         | !-- --> |     its |
|        |     for |     col |     p   |         | ```     |     r   |
|        |         | lection | lanning |         | -       | elative |
|        |   compl |     to  |         |         |  Assess |     ad  |
|        | eteness |         |   stage |         |         | vantage |
|        |     and |  ensure |     for |         |    stak |         |
|        |     a   |         |     com |         | eholder |         |
|        | ccuracy |    data | parison |         |         |         |
|        |         |         |         |         |  partic |         |
|        |         |    will | between |         | ipation |         |
|        |         |         |     pe  |         |     and |         |
|        |         | provide | rceived |         |         |         |
|        |         |         |     and |         |  partic |         |
|        |         |    info |         |         | ipation |         |
|        |         | rmation |  actual |         |         |         |
|        |         |     on  |         |         |   plans |         |
|        |         |         | results |         |         |         |
|        |         |  enviro |         |         |         |         |
|        |         | nmental |         |         |         |         |
|        |         |     and |         |         |         |         |
|        |         |     beh |         |         |         |         |
|        |         | avioral |         |         |         |         |
|        |         |     o   |         |         |         |         |
|        |         | utcomes |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         | -       |         |         |         |         |
|        |         |  Review |         |         |         |         |
|        |         |     mon |         |         |         |         |
|        |         | itoring |         |         |         |         |
|        |         |     and |         |         |         |         |
|        |         |     me  |         |         |         |         |
|        |         | asuring |         |         |         |         |
|        |         |     ap  |         |         |         |         |
|        |         | proach, |         |         |         |         |
|        |         |     b   |         |         |         |         |
|        |         | aseline |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |   data, |         |         |         |         |
|        |         |     and |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |    anti |         |         |         |         |
|        |         | cipated |         |         |         |         |
|        |         |     o   |         |         |         |         |
|        |         | utcomes |         |         |         |         |
|        |         |     ide |         |         |         |         |
|        |         | ntified |         |         |         |         |
|        |         |     in  |         |         |         |         |
|        |         |     the |         |         |         |         |
|        |         |     p   |         |         |         |         |
|        |         | lanning |         |         |         |         |
|        |         |         |         |         |         |         |
|        |         |   stage |         |         |         |         |
+--------+---------+---------+---------+---------+---------+---------+
| **End  | -       | -   No  | -   No  | -       | -       | -       |
| of the |  Verify | rmalize | rmalize |  Verify | Request | Develop |
| EMS    |     a   |         |         |     and |         |     and |
| Pro    | ccuracy |    data |   cost/ |     e   |    that |     fac |
| ject** |     of  |     to  | savings | valuate |         | ilitate |
|        |     o   |         |         |         |   stake |     wo  |
|        | riginal | account |    data |   final | holders | rkshops |
|        |         |     for |     to  |         |         |     and |
|        |    info |         |         |  record | provide |     net |
|        | rmation | changes | account |     r   |     f   | working |
|        |         |     in  |     for | eviews, | eedback |         |
|        | -       |     pro |         |         |     re  |  opport |
|        |  Adjust | duction | changes |   inspe | garding | unities |
|        |     for |         |     in  | ctions, |     the |     to  |
|        |         | -       |     pro |     or  |         |         |
|        |  unanti | Compare | duction |         | quality | promote |
|        | cipated |         |         |   other |     of  |     l   |
|        |     o   |    pre- | -       |         |         | earning |
|        | utcomes | project | Compare |   means |   their |         |
|        |     or  |     b   |         |     of  |     exp | -       |
|        |         | aseline |    pre- |     com | erience | Develop |
|        | changes |     to  | project | pliance |     in  |         |
|        |     in  |         |     b   |     as  |     the |  users' |
|        |     a   |   post- | aseline | surance |         |         |
|        | pproach | project |     to  |         | project |  guides |
|        |         |         |         | -   De  |         |     and |
|        | -       | results |   post- | termine | ```     |     We  |
|        | Compare |     to  | project |     if  | {=html} | b-based |
|        |         |     de  |         |         | <       |         |
|        |   goals | termine |  costs/ | project | !-- --> |   tools |
|        |     and |     net | savings |     is  | ```     |     to  |
|        |         |         |     to  |         | -   I   |     fac |
|        |   items |  change |     de  |    prac | dentify | ilitate |
|        |     ide |         | termine | tically |         |     s   |
|        | ntified | -   I   |     net |         |   areas | cale-up |
|        |     in  | dentify |         |    enfo |     of  |         |
|        |     the |         |  change | rceable |         | ```     |
|        |     p   |   areas |         |         | success | {=html} |
|        | lanning |     of  | ```     | ```     |     and | <       |
|        |         |         | {=html} | {=html} |         | !-- --> |
|        |   stage | success | <       | <       |   short | ```     |
|        |         |     and | !-- --> | !-- --> | comings | -   I   |
|        |    with |         | ```     | ```     |         | dentify |
|        |         |   short | -   I   | -   I   | ```     |         |
|        |    what | comings | dentify | dentify | {=html} |   areas |
|        |     h   |         |         |         | <       |     of  |
|        | appened | -   De  |   areas |   areas | !-- --> |         |
|        |         | termine |     of  |     of  | ```     | success |
|        |  during |     if  |         |         | -   De  |     and |
|        |         |         | success | success | termine |         |
|        | impleme |    more |     and |     and |     if  |   short |
|        | ntation |     i   |         |         |         | comings |
|        |         | n-depth |   short |    shor |    more |         |
|        | -       |     eva | comings | tcoming |     i   | ```     |
|        |  Verify | luation |         |         | n-depth | {=html} |
|        |     a   |     is  | -   Is  | -   Is  |     eva | <       |
|        | ccuracy |     ne  |         |         | luation | !-- --> |
|        |     of  | cessary |   there |   there |     is  | ```     |
|        |         |         |     a   |     a   |     ne  | -   De  |
|        |   logic | -       |     co  |     r   | cessary | termine |
|        |         |  Assess | st/cost | elative |         |     if  |
|        |   model |         |         |     ad  |         |         |
|        |         |  enviro | savings | vantage |         |    more |
|        |         | nmental |     r   |     to  |         |     i   |
|        |         | /public | elative |     the |         | n-depth |
|        |         |         |     adv |     p   |         |     eva |
|        |         |  health | antage? | roject? |         | luation |
|        |         |     r   |         |         |         |     is  |
|        |         | elative | ```     | ```     |         |     ne  |
|        |         |     ad  | {=html} | {=html} |         | cessary |
|        |         | vantage | <       | <       |         |         |
|        |         |     of  | !-- --> | !-- --> |         |         |
|        |         |     the | ```     | ```     |         |         |
|        |         |         | -   De  | -   De  |         |         |
|        |         | project | termine | termine |         |         |
|        |         |         |     if  |     if  |         |         |
|        |         |         |         |         |         |         |
|        |         |         |    more |    more |         |         |
|        |         |         |     i   |     i   |         |         |
|        |         |         | n-depth | n-depth |         |         |
|        |         |         |     eva |     eva |         |         |
|        |         |         | luation | luation |         |         |
|        |         |         |     is  |     is  |         |         |
|        |         |         |     ne  |     ne  |         |         |
|        |         |         | cessary | cessary |         |         |
+--------+---------+---------+---------+---------+---------+---------+
| **     | -       | -   De  | -   De  | -   De  | -   De  | -   De  |
| Formal | Compare | termine | termine | termine | termine | termine |
| Evalua |         |     why |     why |     if  |     why |     how |
| tion** |   goals |         |         |         |         |     the |
|        |     and |   there |   there | project |   there |         |
|        |         |     is  |     is  |     is  |     are | project |
|        |   items |     a   |     a   |         |         |         |
|        |     ide |     dif |     dif |    prac |   areas |   would |
|        | ntified | ference | ference | ticably |     of  |         |
|        |     in  |         |         |         |         |    fare |
|        |     the | between | between |    enfo | success |     if  |
|        |     p   |         |         | rceable |     and |         |
|        | lanning |    pre- |    pre- |     and |         | applied |
|        |         | project | project |         |   short |         |
|        |   stage |     b   |     b   |    what | comings |    more |
|        |         | aseline | aseline |     it  |         |         |
|        |    with |     and |     to  |         | ```     | broadly |
|        |         |         |         |   means | {=html} |         |
|        |    what |   post- |   post- |         | <       | ```     |
|        |     h   | project | project | ```     | !-- --> | {=html} |
|        | appened |         |         | {=html} | ```     | <       |
|        |         | results |  costs/ | <       | -   Why | !-- --> |
|        |  during |         | savings | !-- --> |     is  | ```     |
|        |         | -   De  |         | ```     |     or  | -   De  |
|        | impleme | termine | -   Why | -   De  |         | termine |
|        | ntation |     why |     or  | termine |   isn't |         |
|        |         |         |     why |     why |         |    what |
|        | -       |   there |     not |         |   there |         |
|        |  Verify |     is  |     is  |   there |     a   | aspects |
|        |     a   |     or  |         |     is  |     r   |     of  |
|        | ccuracy |     is  |   there |     a   | elative |     the |
|        |     of  |     not |     a   |     r   |     ad  |         |
|        |         |     an  |     co  | elative | vantage | project |
|        |   logic |         | st/cost |     ad  |     to  |     are |
|        |         | environ |         | vantage |     the |         |
|        |   model | mental/ | savings |     in  |         | working |
|        |         |         |     r   |     com | project |         |
|        | -   Use |  public | elative | pliance |     in  |    well |
|        |         |         |     adv |     and |         |     and |
|        |   logic |  health | antage? |         |   terms |         |
|        |         |     r   |         |    enfo |     of  |   those |
|        |   model | elative | -   De  | rcement |         |     key |
|        |     to  |     ad  | termine |     to  |  public |         |
|        |         | vantage |     if  |     the |         | aspects |
|        |    look |     of  |         |     p   |   invol |         |
|        |     for |     the |    more | roject? | vement- |    that |
|        |         |         |         |         | --i.e., |         |
|        |    gaps | project |   cost- | -   D   |     did |    need |
|        |     and |         | benefit | escribe |     the |     to  |
|        |     una | -   D   |     a   |         |         |     be  |
|        | nswered | escribe | nalysis |    enfo |  public |     m   |
|        |     qu  |         |     is  | rcement |         | odified |
|        | estions | environ |     n   |     and |    have |     in  |
|        |         | mental/ | ecessar |     com |         |         |
|        | -   Use |         | y---why | pliance | greater |   order |
|        |         |  public |     or  |     as  |         |     for |
|        |   logic |         |     why | surance |  access |     the |
|        |         |  health |         |     in  |     to  |         |
|        |   model |         |    not? |         |         | project |
|        |     to  | results |         |   terms |    info |     to  |
|        |         |     in  | ```     |     of  | rmation |     be  |
|        |    help |         | {=html} |     cu  |     or  |         |
|        |     i   |   terms | <       | stomer, |         |    more |
|        | dentify |     of  | !-- --> |         | greater |         |
|        |     key |     cu  | ```     | partner |         | broadly |
|        |     eva | stomer, | -   D   |     and |   means |         |
|        | luation |         | escribe |         |     to  | applied |
|        |     qu  | partner |     cos |    stak |     par |         |
|        | estions |     and | ts/cost | eholder | ticipat |         |
|        |         |         |         |         | e---why |         |
|        |         |    stak | savings |   satis |     or  |         |
|        |         | eholder |     in  | faction |     why |         |
|        |         |         |         |     and |         |         |
|        |         |   satis |   terms |         |    not? |         |
|        |         | faction |     of  | discuss |         |         |
|        |         |     and |     cu  |     why | -       |         |
|        |         |         | stomer, |     the | Analyze |         |
|        |         | discuss |         |         |         |         |
|        |         |     why | partner | results |  public |         |
|        |         |     the |     and |         |         |         |
|        |         |         |         |    have |    invo |         |
|        |         | results |    stak |         | lvement |         |
|        |         |         | eholder | meaning |     in  |         |
|        |         |    have |         |         |         |         |
|        |         |         |   satis |         |   terms |         |
|        |         | meaning | faction |         |     of  |         |
|        |         |         |     and |         |         |         |
|        |         |         |         |         |   satis |         |
|        |         |         | discuss |         | faction |         |
|        |         |         |     why |         |     and |         |
|        |         |         |     the |         |     ask |         |
|        |         |         |         |         |     the |         |
|        |         |         | results |         |     q   |         |
|        |         |         |         |         | uestion |         |
|        |         |         |    have |         |     of  |         |
|        |         |         |         |         |     why |         |
|        |         |         | meaning |         |     are |         |
|        |         |         |         |         |         |         |
|        |         |         |         |         |    they |         |
|        |         |         |         |         |     sa  |         |
|        |         |         |         |         | tisfied |         |
|        |         |         |         |         |     or  |         |
|        |         |         |         |         |         |         |
|        |         |         |         |         |  dissat |         |
|        |         |         |         |         | isfied? |         |
+--------+---------+---------+---------+---------+---------+---------+

Number range CHAPTER**MODULE 1: MAPPING THE EMS PROJECT**

The following questions are designed to help evaluate the outcomes of
projects that integrate EMSs with a regulatory program. EPA's *Strategy
for Determining the Role of EMSs in Regulatory Programs* provides
guiding principles and a framework to assess what role, if any, EMSs
should play in regulatory programs. Evaluation of the EMS project is
critical to the strategy and helps EPA weigh future policy decisions on
the use of EMSs in regulatory programs.

**I. Summary and Purpose of the EMS Project**

This section provides basic background information on the EMS project,
including a brief description of how the EMS project differs from the
traditional way of doing business. The user should also include the
impetus for the project and its purpose, so that anyone reading about
the project will quickly understand how and why EMSs are being used. The
following questions will help to frame this background information:

1)  Why was the EMS project developed or proposed?

2)  What problem or opportunity does the project address?

3)  To what extent and does the project focus on the following:

    a.  Individual facilities

    b.  Economic sectors or groups of sectors

    c.  Other regulated entities

    d.  Communities

    e.  Tribes

    f.  Other

4)  To what extent is the project intended to:

    g.  Improve technology

    h.  Streamline state/federal regulations

    i.  Improve organizational management/operations

    j.  Make more efficient use of state/local/federal resources

    k.  Improve stakeholder involvement

    l.  Foster organizational change, especially with respect to
        organizational culture

    m.  Improve environmental management practices (e.g., pollution
        prevention, environmental stewardship, environmental data, etc.)

    n.  Consider cross-media impacts or multi-media strategies

    o.  Other

```{=html}
<!-- -->
```
4)  In what way(s) does the project involve new ideas and approaches
    when compared to the current/existing approach?

5)  What programs or policies are impacted by the project, and how?

6)  What are the key resources that the EMS project uses?

7)  What are the key activities that you undertake in your EMS project?

8)  What are the key outputs of your EMS project?

Questions 7 through 9 deal with the key resources, activities and
outputs of the EMS project.

###### 

###### Key Resources

These are programmatic investments available to support the EMS project.
Examples could include FTE, support staff, extramural funding, grants,
outside consultants and equipment. Typically, you include those
resources currently in hand versus projected or anticipated resources.

###### Activities

These are things that are done---activities that you plan---to conduct
the EMS project. Examples include designing marketing tools, holding
training sessions, and conducting surveillance audits.

###### Outputs

These are products or service delivery/implementation targets the
project aims to produce. Examples include EMS templates, brochures
produced, and audit reports. There is some gray area between an activity
and an output. The important distinction is that there should be a
logical link between the resources, activities and outputs that
resembles the diagram below.

II\. Identifying Customers, Partners, Stakeholders of the EMS Project

Answering this set of questions will identify the major participants in
the development and implementation of the project, and their respective
roles and responsibilities. Key individuals who are working together on
the project should be identified as partners. Partners will be those
individuals who are necessary participants in order for the project to
be implemented. For example, in an EMS type project, the state and the
regional EPA office will likely be identified as key partners. In the
case of a community-based project, the community leaders and organizers
who are actively participating in the project should be identified and
considered partners. The roles and responsibilities of federal and state
regulators should be described, key contact personnel identified, and
the process for coordination and collaboration documented. The user
should also identify key customers of the project. An example of a key
customer may be an industrial sector that may benefit from knowing the
results of a facility-specific EMS project. Specificity in identifying
customers of the project will allow the user to target resources,
collect data and communicate results, and craft public involvement
strategies.

The user should also make an attempt to describe the key regulated
entities of the project. For example, if the project is intended to
benefit the government and regulated entities, the user should further
specify which parties in the government that may benefit (e.g.,
inspectors, permit writers), the level of government at which the
project is aimed (e.g., the project is intended to benefit state permit
writers and inspectors), and the types of facilities that may benefit
(e.g., is the project intended to help a sector or an individual
facility?). Key stakeholders in the project are the individuals who may
care about the project and its results, but may not be active in the
everyday implementation and activities of the project. For example, for
an EPA project, key stakeholders may be a community, EPA senior managers
and Congress. For a facility project, senior corporate managers and
shareholders may be key stakeholders to engage. These stakeholders are
important to keep in mind when communicating the results of the project
or in designing the project. The key message in this section is
identifying those you are trying to reach and/or influence and those who
you work with to influence your key customers.

10. Who are the key regulated entities in the project?

11. Who are the key partners of the project?

12. Who are the key customers of the project?

13. Who are the key stakeholders in the project?

14. Who has primary responsibility for designing, overseeing, and
    implementing the EMS project or using the innovative approach or
    tool?

15. Does the project involve delegation of regulatory responsibilities
    from EPA to a tribe or state or from the state to local government?
    (Y/N). If yes, how?

####### Identifying Outcomes

Outcomes are the changes or benefits resulting from activities and
outputs. Outcomes should be described using the following structure to
help set goals and clarify the scope of the project.

*Outcome Structure:*

Short-term outcomes---Changes in learning, knowledge, attitude, and
skills. Example: Facilities learn about the benefits of Environmental
Management Systems (EMSs) as a direct result of training (an activity).

Intermediate outcomes---Changes in behavior, practice or decisions.
Example: Facility identifies and implements three process improvements
in their EMS.

Long-term outcomes---Changes in condition. Example: Facility decreases
wastewater discharge by 20 percent and increases hazardous waste
generation by 10 percent.

This structure is known as the **ABCs of outcomes.** You are looking for
how your project leads to changes in **A**ttitude or the acquisition of
knowledge, which leads to a change in **B**ehavior, which leads to a
change in **C**ondition.

####### 

####### 

####### 

####### 

16. Please list and describe all intended or anticipated short-term
    outcomes.

17. Please list and describe all intended or anticipated intermediate
    outcomes.

18. Please list and describe all intended or anticipated long-term
    outcomes.

The key to describing the anticipated or actual outcomes of your
innovation is to identify those outcomes that are the direct result of
your EMS project (outcomes that you directly control) versus those
outcomes that are greatly influenced by factors outside of the scope of
your EMS project (outcomes that are indirectly controlled). This is
better illustrated in the following example:

*Example A:* Your EMS project is designed to substantially reduce NO~x~
emissions from a particular industrial sector that is heavily
concentrated in one airshed with a high percentage of smog days. You
invest \$100,000 and 2 FTE into the project for activities that include
conducting on-site technical visits and consultation and producing
helpful materials for the sector. Your project is able to hold 10
on-site technical visits and produce 2 workbooks, one in English and one
in Spanish. You work with the Trade Association for this sector to
disseminate the information. You want this sector to adopt the changes
proposed in the workbooks and through on-site assistance. The facilities
get the workbooks and meet with the technical assistance providers
(short-term outcome) and adopt the changes (intermediate outcome). These
changes lead to a 15 percent reduction in NO~x~ from the sector. This is
a long-term outcome that is the DIRECT result of your EMS project. If
you say that your project will reduce smog in the airshed, this is an
INDIRECT long-term outcome. It is true that your proejct did lead to
NO~x~ reductions in one particular sector and can reduce smog in a
particular airshed if all other factors that control smog did not
change. There are many other factors in the environment that contribute
to smog outside of your project. Therefore, it is more important for
your project to be measuring long-term outcomes that are directly
related to your innovation rather than measuring all smog in the airshed
since you would have little confidence in the way in which your project
impacts the airshed when combined with the cumulative environmental
impacts.

+------+-------+-------+-------+------+--------+---------+---------+
| *    | **Act | ##    | **C   | **Sh | **I    | **Lo    | **Lo    |
| *Res | iviti | # Out | ustom | ort- | nter-m | ng-term | ng-term |
| ourc | es/** | puts  | ers** | term | ediate | enviro  | enviro  |
| es** |       |       |       | ou   | outc   | nmental | nmental |
|      | **    | -Pr   | -S    | tcom | omes** | out     | out     |
| \    | Progr | oduce | ector | es** |        | comes** | comes** |
| $100 | ams** | work  | Trade |      | -Faci  |         |         |
| ,000 |       | books |       | -Fa  | lities | \- NOx  | -Smog   |
|      | -De   | in    | A     | cili | adopt  | em      | in the  |
| 2    | velop | Sp    | ssoci | ties | c      | issions | airshed |
| FTE  | work  | anish | ation | l    | hanges | for     | reduced |
|      | books | and   |       | earn | sug    | this    |         |
|      |       | En    | -     | a    | gested | sector  | *I      |
|      | -De   | glish | Indiv | bout | by     | reduced | NDIRECT |
|      | velop |       | idual | w    | wor    |         | C       |
|      | on    | -Co   | Facil | orkb | kbooks | *DIRECT | ONTROL* |
|      | -site | nduct | ities | ooks |        | C       |         |
|      | tech  | on    |       | and  | -Faci  | ONTROL* |         |
|      | nical | -site |       | meet | lities |         |         |
|      | assis | v     |       | with | adopt  |         |         |
|      | tance | isits |       | t    | tec    |         |         |
|      | mate  |       |       | echn | hnical |         |         |
|      | rials |       |       | ical | assi   |         |         |
|      |       |       |       | as   | stance |         |         |
|      |       |       |       | sist | sugge  |         |         |
|      |       |       |       | ance | stions |         |         |
|      |       |       |       | p    |        |         |         |
|      |       |       |       | rovi |        |         |         |
|      |       |       |       | ders |        |         |         |
+------+-------+-------+-------+------+--------+---------+---------+

####### Example A: Logic Model for EMS Project to Reduce Air Emissions

####### IV. EMS Project -- Tools, Drivers, and Barriers

**[Tools for the EMS Project]{.underline}**

The tools that assist an EMS project may have significant influence on
the progress and advancement of the project. This section identifies and
describes tools that have been or will be used during development and
implementation of the project. Tools that may be used to add to the EMS
include but are not limited to: economic incentives, regulatory reform,
smart permitting, pollution prevention, information management and
access, and risk-based standards. Since this user's guide and
accompanying modules refer specifically to EMSs, the question below asks
the user to identify additional tools that are being developed to
support the EMS.

19. What additional tools are employed to supplement the EMS (e.g.,
    economic incentives, regulatory reform, smart permitting, pollution
    prevention, information management and access)? Please describe.

# [Drivers for the EMS project]{.underline}

This question asks what the primary motivating factors that drive the
project forward are. Some drivers may include: the need for regulatory
flexibility to reduce uncertainty within the permitting process, rapid
economic growth, technological/scientific development, increased
environmental awareness, population growth, urbanization, and
international commerce. Drivers that promote the EMS should be
identified, and if possible, ranked according to their significance. For
example, the potential opportunity costs of project delays from air
permitting can be high, creating a demand among companies for permit
flexibility to stay competitive within a global market by engaging in an
EMS permitting project. By identifying the primary drivers for the
project, the user may be better equipped to identify potential adopters
of the project in the event of broad-scale application.

20. Describe the motivating factors for undertaking the project and
    explain how such factors promote the EMS project (e.g., market
    forces, law or policy that promotes the project).

# [Barriers to the EMS Project]{.underline}

Barriers to an EMS project may include the following: technical
challenges consisting of inadequate tools or limited economic
alternatives; lack of facilities or regulated entities willing to
participate; scientific challenges consisting of the absence of key data
or inadequate scientific understanding; institutional challenges such as
a resistance to change, a jurisdictional challenge, or lack of an
authorizing environment; or legal challenges arising from laws,
regulations, or policies that impede the project. Different barriers
require different strategies to create a pathway for the project.
Barriers, and the strategies for overcoming these challenges, should be
identified within this module. The user should identify and address
barriers, or perceived barriers, early in the project to enhance the
chance of success.

21. Describe all challenges to the project and explain how such
    challenges may present barriers.

## V. Describing the Logic of the EMS Project 

Many programs and projects often run into trouble because they lack a
well-articulated road map describing the logic of the program or
project. Having gathered information on the purpose of the project, the
problem or opportunity the project addresses, as well as the customers,
drivers for and barriers to the project, as a next step in the process,
the user may want to develop a logic model that synthesizes the key
activities intended to achieve the goals of the project into a picture
linking inputs to activities and to expected outputs and outcomes. A
logic model is a [diagram]{.underline} and [text]{.underline} that
describes the logical (causal) relationships among program elements and
the problem to be solved, thus defining measurements of success.

Using a logic model helps determine the degree to which project's
activities affect the expected outcomes and can help plan appropriate
measures to achieve the outcomes. Logic models can be created in many
different ways. For an ongoing project, the starting point can be the
elements of the project, which are then organized into their logical
flow. For a new project that is in the planning phase, the starting
point can be the mission and long-term goals of the effort. The
intermediate objectives that lead to those long-term goals are then
added to the model, followed by the short-term outcomes that will result
from those intermediate objectives. The key to the logic model is that
it tells the story of why the project is important, how it will make a
difference, and the expected outcomes as a result of the project. An
example of a logic model is provided in Exhibit 1a on Page 16.

# Contextual Factors

Contextual factors are those program externalities that are outside of
your control and are often surprise changes that can greatly impact your
innovation. Contextual factors can both be positive or negative. For
example, your EMS project has caught the eye of a prominent politician
who requests more money for your program. Conversely, your project may
have lost political favor the next year. These are factors outside of
your control that can change the day-to-day operation of your program.
Question 22 asks you to hypothesize about these potential changes. Some
changes you may be aware of but have not accounted for as yet in the
implementation. For example, you know in one year that your program
manager will be going to school part-time. How does losing a full FTE
impact your program one-year from now?

22. Describe all contextual factors that are *outside of your control*
    that can effect the implementation of your innovation.

Question 23 asks about underlying assumptions of the program. Underlying
assumptions may include contextual factors, but should discuss aspects
of your program that you assume exist and affects the way you implement
your program. Using Example A (on the previous page), you know that
regional smog is based on weather patterns, emissions from stationary
sources and emissions from mobile sources. Your EMS project may be based
on a sophisticated model that allows you to calculate smog formation
based on NO~x~ emissions from a particular sector and accurately predict
how much smog was reduced for a given timeframe. The use of a
sophisticated model allows you to assume certain outcomes and helps
describe how your program chooses to measure its long-term outcomes. The
underlying assumption in this example is that the model is accurately
predicting smog formation.

23\. Describe the underlying assumptions of your program.

# VII. Benefits of Developing a Logic Model

###### Design and Planning Phase

The logic model can help communicate the performance story of the EMS
project and can help build a common understanding of the purpose, goals,
and anticipated outcomes among staff and stakeholders. By answering
questions relating to what the project is trying to achieve, with what
resources, through what customers, the program niche, and the expected
results within a given context, the logic model can help identify
potential pitfalls in the design of the project. The logic model
identifies potential outcomes that may be difficult to achieve based on
the design of the EMS project. For example, if the EMS project is
supposed to result in a behavior change at the management level of the
facility and if there is no built-in measurement system to assess
whether or not facility managers have changed their behavior as a result
of the project, then it will be difficult to measure the overall success
of the project.

###### Implementation Phase

The logic model can be used to further design and develop the project by
helping to identify gaps in the suite of activities and prioritize
programs and resources towards achieving desired end outcomes and goals.
By being explicit about the program theory and assumptions behind a
project, various stakeholders and policy-makers can better understand
the project.

Once completed, the logic model can help the practitioner "manage for
results". By arraying information in a logical sequence the user can
identify and choose appropriate performance measures. Specifically, the
identification of anticipated outcomes will be useful in identifying and
developing performance measures and indicators that can be used to
determine if the project is achieving the stated goals, objectives, and
results. The development of performance measures and the collection of
data can facilitate *program improvement* and allow the user to
communicate the value of the project and influence new program
development. As a planning and evaluation tool, the logic model can help
identify which areas of a program to focus an evaluation.

###### End of the EMS Project Phase

If a logic model of the program was not completed during the design or
implementation phases, producing one at the conclusion of an innovation
and at the outset of an evaluation can be an extremely valuable process.
A logic model at this stage will help describe the operation of the
program to stakeholders and evaluators, identify potential questions to
be asked through an evaluation, and highlight the key areas of program
design and theory for further analysis. If a logic model was created
during an earlier phase, returning to it at this point and comparing how
the innovation actually functioned to the earlier logic model can
provide a starting point for areas to focus on during an evaluation.

*Formal Evaluation*

If a logic model was not completed during the design or implementation
phases, producing one at the conclusion of a project and at the outset
of an evaluation can be an extremely valuable process. A logic model at
this stage will help describe the operation of the program to
stakeholders and evaluators, identify potential questions to be asked
through an evaluation and highlight the key areas of program design and
theory for further analysis. If a logic model was created during an
earlier phase, returning to it at this point and comparing how the
project actually functioned to the earlier logic model can provide a
starting point for areas to focus on during an evaluation.

The logic model can help identify the right evaluation questions to ask
based on the major components of the innovation. The evaluation focuses
on the connection points between the elements of the program. This means
that the practitioner should focus on the "how and why" between the
phases of the program. For example, if the project is supposed to change
the behavior of a target group of individuals in order to get improved
compliance---some evaluation questions to ask are 1) to what extent has
compliance improved; 2) what is the innovation doing to change people's
behaviors; and 3) what else could be causing the change in behavior? The
logic model provides a basis for identifying the major facets or
components of the project being evaluated. This is less difficult if the
practitioner determines the major functions of the project and then
aggregates similar functions into program components. After
[each]{.underline} major function area or component is identified, it
should be described in terms of the resources (inputs) needed to conduct
the activities, (e.g., staff, time, finances, information, equipment,
facilities, etc.) and activities (processes) that will be accomplished
to achieve the objectives, outputs, and outcomes. The following five
steps can help in the development of a logic model of the project.

#### VIII. Steps in the Logic Model Process

[Please note that not all of the steps mentioned below have to be
formalized. Time and resources may limit extent to which implementation
of the logic model process occurs, but at a minimum people involved with
the project should work on a logic model.]{.underline}

**STEP 1: Establish a Stakeholder Work Group and Collect Documents**

An important first step in developing the logic model is to establish a
workgroup comprised of individuals/stakeholders that are knowledgeable
about the project. A stakeholder workgroup can provide a wide breadth of
information and different perspectives and knowledge about the project
that might otherwise be missed if developed by a single individual. Once
convened, the workgroup should review any available documentation about
the project that will provide information on goals and objectives,
costs, anticipated outcomes etc. Sources of program documentation
include, strategic and operational plans, budget requests, current
metrics, past evaluations, evaluations of similar projects or programs,
extant theories (e.g., economic, behavioral sciences), and interviews.
This stage does not have to be overly formal or does not have to be a
large workgroup. The main goal is to make sure that all logical pieces
of the project are accounted for and that the logic model will provide a
close to accurate road map of the project and its intended outcomes.

**STEP 2: Define the Problem and Context for the Project**

Clearly defining the problem the project is designed to address and
understanding the context in which the project is designed to operate
will help the user understand the conditions that may influence the
success or lack of success of the project. Using the answers to the
questions in Module 1, begin to develop a problem or issue statement
that describes the problem(s) the project is attempting to solve or the
issue(s) the project will address. As part of this step, specify the
needs and/or assets that led to the design of the project which
addresses the problem. In addition, state and/or identify desired
results, or vision of the future (including those results out of the
control or not in the direct influence of the project), by describing
the expected near and long-term outcomes. Next, list the possible
factors that will influence change in the affected community and list
general successful strategies or "best practices" that have helped
achieve the kinds of results intended by the EMS project. Last, state
the assumptions behind how and why the change strategies will work.

**STEP 3: Define Elements of the Logic in a Table**

Using the answers developed to the questions in Module 1, use the blank
table included in Module 1 to: 1) describe the resources or influential
factors available to support the innovation activities (*question 7*);
2) describe each of the activities conducted to support the innovation
(*question 8*); 3) identify (for each activity) what outputs (service
delivery/implementation targets) aimed to produce/provide (*question
9*); 4) describe the customer(s) the innovation is intended to reach
through the activities and the partner(s) needed to implement the
innovation (*questions 11 and 12*); and 5) identify the short-term,
intermediate and long-term outcomes the innovation is expected to
achieve for each (*questions 16, 17 and 18*). Exhibit 1a below provides
an example of a completed logic model table. Just as a reminder,
**Remember your ABCs: Short-term outcomes** are described as changes to
**A**ttitudes, knowledge and/or skills of the target customers.
**Intermediate outcomes** are described as changes in **B**ehaviors that
result from the acquisition of knowledge or shift in attitude that flows
from the short-term outcomes. **Long-term outcomes** are outcomes that
result in a change in **C**ondition---e.g., the air is cleaner.

+-------+-----------+-------+---------+---------+---------+----------+
| **R   | **Acti    | ##    | **Cust  | **Sho   | **Inter | **L      |
| esour | vities/** | # Out | omers** | rt-term | mediate | ong-term |
| ces** |           | puts  |         | out     | out     | envir    |
| *     | **P       |       | *(who   | comes** | comes** | onmental |
| (what | rograms** | *     | you     |         | *(what  | ou       |
| you   | *(what    | (what | reach   | *(what  | are the | tcomes** |
| inv   | you do in | you   | and who | are the | inter   |          |
| est)* | the       | prod  | you     | short   | mediate | *(what   |
|       | program)* | uce)* | work    | term    | results | is the   |
|       |           |       | with    | results | you are | ultimate |
|       |           |       | ---indi | you are | looking | impact   |
|       |           |       | viduals | looking | for?--- | of the   |
|       |           |       | who use | for?--- | changes | in       |
|       |           |       | the     | changes | in      | novation |
|       |           |       | outputs | in      | Beh     | on       |
|       |           |       | deve    | At      | avior)* | b        |
|       |           |       | loped)* | titude, |         | ehavior, |
|       |           |       |         | kno     |         | human    |
|       |           |       |         | wledge, |         | health,  |
|       |           |       |         | s       |         | envir    |
|       |           |       |         | kills)* |         | onment-- |
|       |           |       |         |         |         | -changes |
|       |           |       |         |         |         | in       |
|       |           |       |         |         |         | Cond     |
|       |           |       |         |         |         | ition?)* |
+-------+-----------+-------+---------+---------+---------+----------+
| Exam  | Examples: | Exam  | Ex      | E       | E       | Example: |
| ples: |           | ples: | amples: | xample: | xample: |          |
|       | *-        |       |         |         |         | *\*R     |
| *     | Materials | *-Tra | *-Deve  | *-In    | *-Dev   | eduction |
| -EPA* | Dev       | ining | lopers* | creased | elopers | in       |
|       | elopment* | v     |         | aw      | and     | n        |
| *-S   |           | ideo* | *-Bu    | areness | b       | on-point |
| tate* | *-Re      |       | ilders* | of      | uilders | source   |
|       | cruiting* | *     |         | harmful | acquire | po       |
| *-L   |           | -Meet |         | e       | new     | llutants |
| ocal* | *-        | ings* |         | ffects* | techn   | in       |
|       | Training* |       |         |         | ologies | w        |
| *-Pri |           | *-    |         | *-In    | and     | aterway* |
| vate* | *-        | Broch |         | creased | change  |          |
|       | Technical | ures* |         | aw      | pr      | *\*H     |
|       | As        |       |         | areness | actice* | ealthier |
|       | sistance* |       |         | of new  |         | w        |
|       |           |       |         | techn   |         | etlands* |
|       | *         |       |         | ologies |         |          |
|       | -Website* |       |         | and     |         | *\*More  |
|       |           |       |         | ince    |         | fish     |
|       |           |       |         | ntives* |         | clean    |
|       |           |       |         |         |         | beaches* |
|       |           |       |         |         |         |          |
|       |           |       |         |         |         | *\*Clean |
|       |           |       |         |         |         | safe/*   |
|       |           |       |         |         |         |          |
|       |           |       |         |         |         | *swi     |
|       |           |       |         |         |         | mmable/* |
|       |           |       |         |         |         |          |
|       |           |       |         |         |         | *        |
|       |           |       |         |         |         | fishable |
|       |           |       |         |         |         | water*   |
+-------+-----------+-------+---------+---------+---------+----------+

Table 1a---Logic Model of a Surface Water Program

**STEP 4: Develop a Diagram of Logical Relationships and a Narrative**

As a logic model is developed, remember that project components are
often related. The logic model should help graphically depict and
explain the logical relationships that exist between inputs, outputs,
and outcomes. It graphically illustrates what must occur in order for
the project to accomplish its goals. While the boxes represent an
activity, the arrows indicate the connection between the activities. As
the model is developed, limit the words in the diagram,
*[but]{.underline}* attach more detail in separate charts or a narrative
that describes the research underlying the assumptions. Keep in mind
there are many different forms of logic model diagrams. The practitioner
may want to have more than one model that depicts different levels of
detail, different groups of activities, different levels at which
performance is measured, different stakeholder views, or different
theories.

#  

# Exhibit 1b---Level II Logic Model Surface Water Program

**STEP 5: Verify the Logic with Stakeholders**

As a final step, be sure to verify the logic model to ensure that all
aspects of the project have been captured and depicted. The original
stakeholder group can review the logic model or an even broader group of
stakeholders can be employed for this. As the model is reviewed,
consider asking "How-Why" questions. For example, start with a specific
outcome and ask, "How is the outcome expected to be achieved?" Start at
Activities and ask, "Why is this activity important?" Also consider
asking "If-Then" Questions. For example, start at Activities and move
along to Outcomes and ask, "If this activity happens, then what outcome
is expected?"

Number range CHAPTER**MODULE 2: ASSESSING THE ENVIRONMENTAL RESULTS**

**OF THE EMS PROJECT**

Module 1: Mapping the EMS Project asks the user to describe the logic of
the project, and the logic model process describes the relationship
between the goals, activities, outputs, and outcomes of the project.
Module 2 is intended to help the practitioner identify the environmental
goals of an EMS project, link environmental outcomes to those goals,
identify appropriate performance measures, determine methods for
measuring results, and measure the results of the project.[^1] The term
*environmental results* is intended to include [output]{.underline}
measures (e.g., number of facilities committing to a reduction in
greenhouse gases in a voluntary program or number of permits issued),
[outcome]{.underline} measures (e.g., percent reduction of greenhouse
gases over the baseline), and environmental [indicator]{.underline}
measures (e.g., air monitoring data that indicates improvements in air
quality).

Environmental program and project evaluation often falls short on
identifying environmental performance measures, tracking environmental
indicators, and measuring environmental outcomes. **Performance
measures** should be designed at the start of the project and track with
the goals of the project. For example, if the project is testing a new
environmental technology, an example of a performance measure is 50
percent increase in adoption of the technology within two years when
compared to the current level of usage. **Environmental indicators**
help measure the state of air, water, and land resources, the pressures
on them, and the resulting effects on ecological and human health. An
example of an indicator measure is the number of people living in areas
with ozone (8-hour) and particulate matter levels above the National
Ambient Air Quality Standards. **Outcome measures** look at the extent
to which the project is achieving its intended results. There are three
levels of outcome measures---short-term outcomes that look for impacts
and changes in attitude (i.e., increase in environmental knowledge),
intermediate outcomes look at change in behaviors (i.e., skills adopted
from knowledge), and long-term outcomes that measure the change in
conditions (i.e., the overall impact). For example, EPA's programs
should have long-term outcomes that should include measures of
improvement in public health and the environment to show that EPA is
reaching its goals.

Environmental results may also include the benefits of having a cleaner
environment -- typically classified into four types of benefits: human
health, ecological, amenities (i.e., taste, odor, visibility), and
reduced environmental damages (e.g., reduced runoff). Such environmental
results should be identified and to the extent feasible, described
quantitatively. In situations where quantification is not possible, the
user should qualitatively describe the benefits.[^2] Results also may
include or be dependent on behavioral changes that the project may be
trying to address. Behavioral changes can also be described in this
section, however they may be described at different qualitative levels
(e.g., anecdotal evidence versus a survey) depending on the evaluative
rigor necessary for the project. Table 2 below highlights performance
measures across the logic model (discussed in Module 1). This module
will help you identify and define measures for your project; evaluate
the measure in terms of data collection, analysis and reporting; and
choose the most important measures according to the requirements of the
project, importance of the measure and how relevant the measure is to
defining how well the EMS project is working.

Table 2: Types of Performance Measures Across the Spectrum

+--------+---+------------------------------+---+-------------------------+
| **Cate |   | **Definition**               |   | **Examples**            |
| gory** |   |                              |   |                         |
+--------+---+------------------------------+---+-------------------------+
| Reso   |   | Resources consumed by the    |   | Amount of funds, \# of  |
| urces/ |   | organization.                |   | FTE, materials,         |
|        |   |                              |   | equipment, supplies     |
| Inputs |   |                              |   | (etc.).                 |
+--------+---+------------------------------+---+-------------------------+
| Acti   |   | The work performed that      |   | \# of training classes  |
| vities |   | directly produces the core   |   | offered as designed;    |
|        |   | products and services.       |   | Hours of technical      |
|        |   |                              |   | assistance training for |
|        |   |                              |   | staff.                  |
+--------+---+------------------------------+---+-------------------------+
| O      |   | Products and services        |   | #of technical           |
| utputs |   | provided as a direct result  |   | assistance requests     |
|        |   | of program activities.       |   | responded to; \# of     |
|        |   |                              |   | compliance workbooks    |
|        |   |                              |   | developed/delivered.    |
+--------+---+------------------------------+---+-------------------------+
| Cu     |   | Measure of target population |   | \% of target population |
| stomer |   | receiving outputs.           |   | trained; \# of target   |
| R      |   |                              |   | population receiving    |
| eached |   |                              |   | technical assistance.   |
+--------+---+------------------------------+---+-------------------------+
| Cu     |   | Measure of satisfaction with |   | \% of customers         |
| stomer |   | outputs.                     |   | dissatisfied with       |
| Satisf |   |                              |   | training; % of          |
| action |   |                              |   | customers "very         |
|        |   |                              |   | satisfied" with         |
|        |   |                              |   | assistance received.    |
+--------+---+------------------------------+---+-------------------------+
| Ou     |   | Accomplishment of program    |   | Pounds of pollutants    |
| tcomes |   | objectives; attributable to  |   | reduced; Miles of       |
|        |   | program outputs.             |   | beaches cleaned; %      |
|        |   |                              |   | increase in industry's  |
|        |   |                              |   | understanding of        |
|        |   |                              |   | regulatory recycling    |
|        |   |                              |   | exclusion; \# of        |
|        |   |                              |   | sectors that adopt      |
|        |   |                              |   | regulatory recycling    |
|        |   |                              |   | exclusion.              |
+--------+---+------------------------------+---+-------------------------+
| **The  |   |                              |   |                         |
| me     |   |                              |   |                         |
| asures |   |                              |   |                         |
| below  |   |                              |   |                         |
| are    |   |                              |   |                         |
| d      |   |                              |   |                         |
| erived |   |                              |   |                         |
| using  |   |                              |   |                         |
| the    |   |                              |   |                         |
| six    |   |                              |   |                         |
| me     |   |                              |   |                         |
| asures |   |                              |   |                         |
| iden   |   |                              |   |                         |
| tified |   |                              |   |                         |
| ab     |   |                              |   |                         |
| ove.** |   |                              |   |                         |
+--------+---+------------------------------+---+-------------------------+
| Effi   | T |                              | C |                         |
| ciency | h |                              | o |                         |
|        | e |                              | s |                         |
|        | r |                              | t |                         |
|        | a |                              | p |                         |
|        | t |                              | e |                         |
|        | i |                              | r |                         |
|        | o |                              | w |                         |
|        | o |                              | o |                         |
|        | f |                              | r |                         |
|        | t |                              | k |                         |
|        | h |                              | b |                         |
|        | e |                              | o |                         |
|        | a |                              | o |                         |
|        | m |                              | k |                         |
|        | o |                              | p |                         |
|        | u |                              | r |                         |
|        | n |                              | o |                         |
|        | t |                              | d |                         |
|        | o |                              | u |                         |
|        | f |                              | c |                         |
|        | i |                              | e |                         |
|        | n |                              | d |                         |
|        | p |                              | ; |                         |
|        | u |                              | C |                         |
|        | t |                              | o |                         |
|        | t |                              | s |                         |
|        | o |                              | t |                         |
|        | t |                              | p |                         |
|        | h |                              | e |                         |
|        | e |                              | r |                         |
|        | a |                              | i |                         |
|        | m |                              | n |                         |
|        | o |                              | s |                         |
|        | u |                              | p |                         |
|        | n |                              | e |                         |
|        | t |                              | c |                         |
|        | o |                              | t |                         |
|        | f |                              | i |                         |
|        | o |                              | o |                         |
|        | u |                              | n |                         |
|        | t |                              | c |                         |
|        | p |                              | o |                         |
|        | u |                              | n |                         |
|        | t |                              | d |                         |
|        | . |                              | u |                         |
|        | F |                              | c |                         |
|        | o |                              | t |                         |
|        | c |                              | e |                         |
|        | u |                              | d |                         |
|        | s |                              | . |                         |
|        | i |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | n |                              |   |                         |
|        | o |                              |   |                         |
|        | p |                              |   |                         |
|        | e |                              |   |                         |
|        | r |                              |   |                         |
|        | a |                              |   |                         |
|        | t |                              |   |                         |
|        | i |                              |   |                         |
|        | n |                              |   |                         |
|        | g |                              |   |                         |
|        | e |                              |   |                         |
|        | f |                              |   |                         |
|        | f |                              |   |                         |
|        | i |                              |   |                         |
|        | c |                              |   |                         |
|        | i |                              |   |                         |
|        | e |                              |   |                         |
|        | n |                              |   |                         |
|        | c |                              |   |                         |
|        | y |                              |   |                         |
|        | . |                              |   |                         |
|        | R |                              |   |                         |
|        | e |                              |   |                         |
|        | l |                              |   |                         |
|        | a |                              |   |                         |
|        | t |                              |   |                         |
|        | i |                              |   |                         |
|        | n |                              |   |                         |
|        | g |                              |   |                         |
|        | o |                              |   |                         |
|        | u |                              |   |                         |
|        | t |                              |   |                         |
|        | p |                              |   |                         |
|        | u |                              |   |                         |
|        | t |                              |   |                         |
|        | t |                              |   |                         |
|        | o |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | m |                              |   |                         |
|        | e |                              |   |                         |
|        | s |                              |   |                         |
|        | p |                              |   |                         |
|        | e |                              |   |                         |
|        | c |                              |   |                         |
|        | i |                              |   |                         |
|        | f |                              |   |                         |
|        | i |                              |   |                         |
|        | c |                              |   |                         |
|        | r |                              |   |                         |
|        | e |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | u |                              |   |                         |
|        | r |                              |   |                         |
|        | c |                              |   |                         |
|        | e |                              |   |                         |
|        | i |                              |   |                         |
|        | n |                              |   |                         |
|        | t |                              |   |                         |
|        | e |                              |   |                         |
|        | r |                              |   |                         |
|        | m |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | f |                              |   |                         |
|        | c |                              |   |                         |
|        | o |                              |   |                         |
|        | s |                              |   |                         |
|        | t |                              |   |                         |
|        | o |                              |   |                         |
|        | r |                              |   |                         |
|        | t |                              |   |                         |
|        | i |                              |   |                         |
|        | m |                              |   |                         |
|        | e |                              |   |                         |
|        | . |                              |   |                         |
+--------+---+------------------------------+---+-------------------------+
| Produc | M |                              | N |                         |
| tivity | e |                              | u |                         |
|        | a |                              | m |                         |
|        | s |                              | b |                         |
|        | u |                              | e |                         |
|        | r |                              | r |                         |
|        | e |                              | o |                         |
|        | o |                              | f |                         |
|        | f |                              | e |                         |
|        | t |                              | n |                         |
|        | h |                              | f |                         |
|        | e |                              | o |                         |
|        | r |                              | r |                         |
|        | a |                              | c |                         |
|        | t |                              | e |                         |
|        | e |                              | m |                         |
|        | o |                              | e |                         |
|        | f |                              | n |                         |
|        | p |                              | t |                         |
|        | r |                              | c |                         |
|        | o |                              | a |                         |
|        | d |                              | s |                         |
|        | u |                              | e |                         |
|        | c |                              | s |                         |
|        | t |                              | i |                         |
|        | i |                              | n |                         |
|        | o |                              | v |                         |
|        | n |                              | e |                         |
|        | p |                              | s |                         |
|        | e |                              | t |                         |
|        | r |                              | i |                         |
|        | s |                              | g |                         |
|        | o |                              | a |                         |
|        | m |                              | t |                         |
|        | e |                              | e |                         |
|        | s |                              | d |                         |
|        | p |                              | p |                         |
|        | e |                              | e |                         |
|        | c |                              | r |                         |
|        | i |                              | i |                         |
|        | f |                              | n |                         |
|        | i |                              | s |                         |
|        | c |                              | p |                         |
|        | u |                              | e |                         |
|        | n |                              | c |                         |
|        | i |                              | t |                         |
|        | t |                              | o |                         |
|        | o |                              | r |                         |
|        | f |                              | . |                         |
|        | r |                              |   |                         |
|        | e |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | u |                              |   |                         |
|        | r |                              |   |                         |
|        | c |                              |   |                         |
|        | e |                              |   |                         |
|        | ( |                              |   |                         |
|        | e |                              |   |                         |
|        | . |                              |   |                         |
|        | g |                              |   |                         |
|        | . |                              |   |                         |
|        | s |                              |   |                         |
|        | t |                              |   |                         |
|        | a |                              |   |                         |
|        | f |                              |   |                         |
|        | f |                              |   |                         |
|        | o |                              |   |                         |
|        | r |                              |   |                         |
|        | e |                              |   |                         |
|        | m |                              |   |                         |
|        | p |                              |   |                         |
|        | l |                              |   |                         |
|        | o |                              |   |                         |
|        | y |                              |   |                         |
|        | e |                              |   |                         |
|        | e |                              |   |                         |
|        | ) |                              |   |                         |
|        | . |                              |   |                         |
|        | T |                              |   |                         |
|        | h |                              |   |                         |
|        | e |                              |   |                         |
|        | f |                              |   |                         |
|        | o |                              |   |                         |
|        | c |                              |   |                         |
|        | u |                              |   |                         |
|        | s |                              |   |                         |
|        | i |                              |   |                         |
|        | s |                              |   |                         |
|        | o |                              |   |                         |
|        | n |                              |   |                         |
|        | l |                              |   |                         |
|        | a |                              |   |                         |
|        | b |                              |   |                         |
|        | o |                              |   |                         |
|        | r |                              |   |                         |
|        | p |                              |   |                         |
|        | r |                              |   |                         |
|        | o |                              |   |                         |
|        | d |                              |   |                         |
|        | u |                              |   |                         |
|        | c |                              |   |                         |
|        | t |                              |   |                         |
|        | i |                              |   |                         |
|        | v |                              |   |                         |
|        | i |                              |   |                         |
|        | t |                              |   |                         |
|        | y |                              |   |                         |
|        | . |                              |   |                         |
+--------+---+------------------------------+---+-------------------------+
| Cost-E | M |                              | C |                         |
| ffecti | e |                              | o |                         |
| veness | a |                              | s |                         |
|        | s |                              | t |                         |
|        | u |                              | p |                         |
|        | r |                              | e |                         |
|        | e |                              | r |                         |
|        | t |                              | p |                         |
|        | h |                              | o |                         |
|        | a |                              | u |                         |
|        | t |                              | n |                         |
|        | r |                              | d |                         |
|        | e |                              | s |                         |
|        | l |                              | o |                         |
|        | a |                              | f |                         |
|        | t |                              | p |                         |
|        | e |                              | o |                         |
|        | s |                              | l |                         |
|        | o |                              | l |                         |
|        | u |                              | u |                         |
|        | t |                              | t |                         |
|        | c |                              | a |                         |
|        | o |                              | n |                         |
|        | m |                              | t |                         |
|        | e |                              | s |                         |
|        | s |                              | p |                         |
|        | t |                              | r |                         |
|        | o |                              | o |                         |
|        | c |                              | d |                         |
|        | o |                              | u |                         |
|        | s |                              | c |                         |
|        | t |                              | e |                         |
|        | s |                              | d |                         |
|        | . |                              | ; |                         |
|        |   |                              | C |                         |
|        |   |                              | o |                         |
|        |   |                              | s |                         |
|        |   |                              | t |                         |
|        |   |                              | p |                         |
|        |   |                              | e |                         |
|        |   |                              | r |                         |
|        |   |                              | m |                         |
|        |   |                              | i |                         |
|        |   |                              | l |                         |
|        |   |                              | e |                         |
|        |   |                              | o |                         |
|        |   |                              | f |                         |
|        |   |                              | b |                         |
|        |   |                              | e |                         |
|        |   |                              | a |                         |
|        |   |                              | c |                         |
|        |   |                              | h |                         |
|        |   |                              | c |                         |
|        |   |                              | l |                         |
|        |   |                              | e |                         |
|        |   |                              | a |                         |
|        |   |                              | n |                         |
|        |   |                              | e |                         |
|        |   |                              | d |                         |
|        |   |                              | . |                         |
+--------+---+------------------------------+---+-------------------------+
| S      | M |                              | P |                         |
| ervice | e |                              | e |                         |
| Q      | a |                              | r |                         |
| uality | s |                              | c |                         |
|        | u |                              | e |                         |
|        | r |                              | n |                         |
|        | e |                              | t |                         |
|        | o |                              | o |                         |
|        | f |                              | f |                         |
|        | t |                              | t |                         |
|        | h |                              | e |                         |
|        | e |                              | c |                         |
|        | q |                              | h |                         |
|        | u |                              | n |                         |
|        | a |                              | i |                         |
|        | l |                              | c |                         |
|        | i |                              | a |                         |
|        | t |                              | l |                         |
|        | y |                              | a |                         |
|        | o |                              | s |                         |
|        | f |                              | s |                         |
|        | p |                              | i |                         |
|        | r |                              | s |                         |
|        | o |                              | t |                         |
|        | d |                              | a |                         |
|        | u |                              | n |                         |
|        | c |                              | c |                         |
|        | t |                              | e |                         |
|        | s |                              | r |                         |
|        | a |                              | e |                         |
|        | n |                              | q |                         |
|        | d |                              | u |                         |
|        | s |                              | e |                         |
|        | e |                              | s |                         |
|        | r |                              | t |                         |
|        | v |                              | s |                         |
|        | i |                              | r |                         |
|        | c |                              | e |                         |
|        | e |                              | s |                         |
|        | s |                              | p |                         |
|        | p |                              | o |                         |
|        | r |                              | n |                         |
|        | o |                              | d |                         |
|        | d |                              | e |                         |
|        | u |                              | d |                         |
|        | c |                              | t |                         |
|        | e |                              | o |                         |
|        | d |                              | w |                         |
|        | . |                              | i |                         |
|        |   |                              | t |                         |
|        |   |                              | h |                         |
|        |   |                              | i |                         |
|        |   |                              | n |                         |
|        |   |                              | o |                         |
|        |   |                              | n |                         |
|        |   |                              | e |                         |
|        |   |                              | w |                         |
|        |   |                              | e |                         |
|        |   |                              | e |                         |
|        |   |                              | k |                         |
|        |   |                              | . |                         |
+--------+---+------------------------------+---+-------------------------+

Environmental results may also include the benefits of having a cleaner
environment, which can be classified into three types of benefits: 1)
human health, 2) ecological, amenities (i.e., taste, odor, visibility),
and 3) reduced environmental damages (e.g., reduced runoff). Such
environmental results should be identified and, to the extent feasible,
described quantitatively. In situations where quantification is not
possible, the innovation practitioner should qualitatively describe the
benefits.[^3] Results also may include or be dependent on behavioral
changes that the innovation may be trying to address. Behavioral changes
can also be described in this section, however they may be described at
different qualitative levels (e.g., anecdotal evidence versus a survey)
depending on the analytical rigor necessary for the innovation.

# *Design and Planning Phase*

For projects in the planning phase, the user should be able to identify
goals, develop appropriate measures to assess whether the project is
meeting its goals, and *anticipate* environmental outcomes. Often, the
hardest part of designing a project is in choosing appropriate measures
to describe the intended outcomes of the project. Many projects stop at
the measurement of outputs (e.g., number of permits issued or number of
inspections completed), especially in the design phase. The user should
have an eye to linking the output measure to the outcome measures for
intended or expected outcomes (e.g., 10 percent improvement in
compliance rates over the previous year of inspected permitted sources).

# *Implementation Phase*

For EMS projects in implementation (depending on how long the project
has been implemented), the user should be able to measure and report
results as to whether environmental outcomes are meeting or exceeding
expectations when compared to the baseline measures. Baseline measures
will be discussed in greater detail in this module. In addition,
evaluators should look at collecting and reporting qualitative and
quantitative information needed to explain if and how well the project
is working. During implementation, this module may be used to re-check
the data measurement and collection approach depending on the results. A
mid-course analysis may reveal a data gap or incomplete data collection.
The user and the project team may have to decide what the impact of a
data gap is on the long-term course of the project and whether the data
should be collected mid-stream.

*End of the EMS Project Phase*

For mature projects, the practitioner should demonstrate if and how the
project poses a relative advantage over the traditional approach in
terms of environmental results. This module explores possible
differences between anticipated results and actual results in order to
ascertain if and why, the project may or may not be working as intended.
For a project to succeed and possibly be replicated, qualitative
information as well as quantitative data on environmental outcomes is
needed.

*Formal Evaluation*

The practitioner should focus on how the results were achieved and if a
causal link can be made between the project and the results. The
evaluation should address how the project [caused]{.underline} the
intended outcomes to be realized. If the outcomes were not realized, the
evaluation should focus on why they were not realized and how the
project should be modified or improved to realize outcomes in the
future.

This module concentrates on environmental results in contrast to process
efficiencies such as permit streamlining that make more efficient use of
federal, state, or local resources. Such efficiencies could be addressed
in Module 3: Assessing the Costs and Cost Savings of the EMS Project.
Process efficiencies may result in greater environmental protection
because scarce resources are redirected to other environmental problems,
but it may be difficult to track these connections and to determine a
cause and effect relationship.

#### I. Identifying Environmental Goals of the EMS Project

In Module 1, the practitioner identified the problem(s) that the project
was designed to target and asks the user to identify outcomes for the
EMS project in a logic model. Module 2 asks the practitioner to identify
the environment goal or goals that the project intends to achieve
relative to the problem that the project is trying to solve. The
practitioner should ensure that the stated goals appropriately match in
size and scope the nature of the environmental problem that the project
is trying to solve. In addition, the practitioner should ensure that the
environmental goals listed below track with the expected outcomes
identified in Module 1.

1\. What are the specific **environmental goals** that the EMS project
intends to achieve? Please describe.

2.  Do the goals of the project include improving the **environmental
    performance of third parties** (e.g., customers, suppliers, etc.)?

3.  Do the **goals of the project match** the expected/intended outcomes
    of the project?

4.  Do the goals of the project include **cross-media transfers**? If
    yes, how many and what types of cross media transfers are being
    considered?

    II. Measuring the Environmental Results

In order to make sound judgments about the potential benefits of the
project, the user needs to measure the results of the project when
compared to the current practice. That is, the user needs to compare the
current state (or world without the project) to the one in which the
project is in effect to determine whether the project is achieving its
intended outcomes. If the user is in the federal government, measurement
is also central to reporting obligations under the Government
Performance and Results Act (GPRA). The user can adapt the questions in
the module to address the GPRA reporting requirements as well as any
other reporting requirements specific to the project. Below is a set of
overview questions, followed by a discussion of the issues to consider
in answering each question, how to design performance measures, and
examples of how data might be organized to measure environmental
impacts.

*Measurement Approach.* A measurement approach is the method(s) that
will be used to collect data and information on the project. For
example, the user may decide to use focus group interviews to collect
qualitative data on the efficiency of the project and can use
air-sampling data to collect data on the efficacy of the project. The
user must determine what measurement approaches are feasible based on
the available data---anecdotal, quantitative, or qualitative---and the
needs and resources of the project team. The user is more likely to have
greater flexibility at the design phase to choose the appropriate
measurement approaches than if the project is already being implemented.

The availability of quantifiable environmental results depends on the
type of intervention of the project. For example, where a project
results in a change in the level of emissions or discharges that are
already being monitored and tracked for regulatory purposes, it will be
easier to access data (e.g., permitted source will already be monitoring
and tracking emissions through stack tests etc.). If an EMS project is
aimed at an environmental problem for which there exists little or no
data (e.g., non-point source pollution for certain pollutants), it will
be more difficult to quantify the level of improvement with the project.

Qualitative and anecdotal data can support a project and may be
necessary to make the case for transferability of the project. For
example, in the case of a health benefit, it may be possible to
qualitatively assert that the project is expected to be a contributing
factor to a reduced incidence of asthma in children without establishing
a direct correlation between the project and the reduced incidence.
Anecdotal data also provides information on how people perceive the
project is working. An example of anecdotal information might be where a
regulatory agency is testing the merits of an Internet-based public
participation process and stakeholders are asked informally to provide
feedback because it is not feasible to conduct a statistically valid
survey.

*Baseline Data.* Establishing a credible baseline is critical for
measuring the impacts of an EMS project. Baseline data are usually built
into the EMS, and the same baseline information should be used to
measure progress in this module. Developing a baseline involves more
than taking stock of current conditions; it lays the foundation against
which all future environmental progress will be measured. Baseline data
provide a frame of reference for the change that the project proposes.
Characterizing an appropriate baseline involves describing the
conditions that prevail in the absence of the project by looking at
[measures]{.underline}, normalizing measures, [timeframes]{.underline},
[assumptions,]{.underline} and [comparability]{.underline}. It is
important to collect baseline data before the project is implemented.

[Measures:]{.underline} The project's goals should be translated into
measurable parameters, and appropriate metrics should express both
baseline conditions and expectations of future changes. Appropriate
measures to consider are environmental measures (e.g., particulate
matter emitted), economic measures (e.g., tons of cement produced), and
the inter-relation of environmental and economic measures (e.g.,
particulate matter emitted per ton of cement produced) when developing a
baseline and planning for future reporting.

Baseline assessments are most helpful if project goals are addressed and
well defined at the beginning of the project. For example, if a
facility's goal is to reduce Hazardous Air Pollutants (HAPs), it should
clarify whether it is committing to reduce all HAPs, or it is focusing
its efforts on reducing a subset of HAPs. Units of measurement and
numeric expressions should be standardized early on. For example, mass
units (tons, pounds) and actual numbers are appropriate measures to use.
Similarly, measures for assessing economic activity should be
standardized early on. The established environmental and economic
baseline measures should be transparent and as simple as possible.

[Normalizing Measures:]{.underline} Normalizing environmental and
economic data helps to organize the data so that it is still relevant
and meaningful in describing the project despite changes to processes or
practices associated with the project. A common way to normalize data is
on a per unit basis, or using a normalizing factor to adjust performance
to the baseline year.

To normalize data on a per unit basis, simply divide the environmental
quantity by the production measure for the same time period, which is
typically one year (e.g., tons of emissions per tons of product
annually). For example, if a facility produces 2000 tons of cement a
year and 30 tons of air pollution, its per unit pollution is 30/2000 =
0.02 tons of air pollution per ton of cement.

When using a normalizing factor, the factor assigned to the baseline
year is always one (e.g., Year 1). For example, in the first year of
reporting and then in subsequent years, a facility divides its current
year production by its baseline year production to derive the
normalizing factor for that year. The facility then divides its actual
environmental performance by the normalized factor to derive the
normalized quantity. In Year 1, the facility produces 2000 tons of
cement and 30 tons of air pollution. In Year 2, the facility produces
2300 tons of cement and 28 tons of air pollution. Its normalizing factor
for the baseline Year 1 is 1.15 (2300/2000). To calculate Year 2 tons of
air pollution normalized to the baseline Year 1, divide the tons of air
pollution produced in Year 2 by the normalizing factor to get 24.35
(28/1.15) normalized tons of air pollution. The normalized quantity is
less than the actual quantity in Year 1, reflecting that the facility
performed proportionally better than its actual environmental statement
given the increase in production in Year 2.

[Timeframes:]{.underline} Another key decision to make when establishing
a baseline is the appropriate period of time that characterized
"current" or "normal" environmental and economic conditions. Using one
year of recent data or an average of two years data is appropriate when:

-   Economic activity is relatively steady over time.

-   Recent and significant environmental technology upgrades mean that
    older environmental data are no longer applicable to future
    activity.

-   Reporting on past performance related to the same project and is
    providing applicable economic data, essentially providing a longer
    time horizon for baselining.

-   Other facilities are involved in the project and may have one of the
    above circumstances.

It is important that environmental and economic data are reporting in
the same timeframe when establishing a baseline, as well as throughout
the life of the project. Matching multi-year environmental data with
single year economic data has the potential to skew the measurement of
project results.

[Assumptions:]{.underline} The user should consider key assumptions
behind the data collection. For example, if the project will be affected
by impending regulations, the baseline should account or include the
effects of the new requirements in the baseline or explicitly identify
the projected impacts of the regulations. Criteria for making this
determination could include the timeframe of the pending regulatory
change compared with the timeframe for the project, the level of
certainty regarding the change in regulations, and the level of
certainty regarding the effect of the change on the threshold for
compliance.

[Comparability:]{.underline} If the user is interested in comparing a
series of like-projects for transferability potential, then it is
important to standardize the baseline as much as possible to allow for
comparison of data. Specifically, individual projects that are part of a
larger EMS initiative should ideally use the same measures, timeframes,
and regulatory assumptions. Failing to standardize baseline conventions
often sets individual projects down different paths in terms of data
reporting and can lead to great difficulty in comparing the results of
multiple projects, and in analyzing factors that affect project success
or lack of success. The user should consider the following baseline
comparability factors whenever possible:

-   Identify a reasonable compromise format that suits all EMS projects
    within a program.

-   Data is provided in multiple formats -- to better reflect individual
    project needs and one that is suitable for comparison and
    aggregation with other projects.

-   Try to identify the components that may not be comparable to others
    due to existing baseline conditions at the start of the project.

A determination of what entities will be subject to the project or
program allows the user to define the extent to which segments of the
community will be affected and to identify reliable databases from which
to draw information regarding the number and size of such entities.
Alternatively, the user may need to confer with councils, trade
associations, or community groups to determine methods for gathering
data regarding existing conditions. Despite efforts to use the best
available data sources to establish a baseline analysis, the user should
nonetheless identify areas of concern such as the consistency of
variables over space and time, adherence to sampling protocols,
sensitive populations, whether non-compliant or exempt entities are
included, or any other limiting factors. In addition, there may be cases
where the project is providing data in an area where there is little
existing data. In such cases, if appropriate, surrogate data from
research or data from similar experiments may be used. Depending on the
project, the baseline may in fact be the absence of information,
activities, or data.

####### *EMS Measures and Indicators*. This module should be used to list the significant environmental impacts, as determined through the EMS, provide objectives, indicators, and outcomes data. An EMS should include regulatory and non-regulatory significant environmental impacts (total or per unit of production). The EMS should also include baseline measures for the impacts or baseline measurement plans for environmental impacts. These potential impacts should be described (as applicable) for:

-   Products

-   Processes

-   Facilities

-   Supply chain

-   Facility support functions

-   Customers

Environmental performance for an EMS project may cover regulated
entities and third parties. Possible indicators for EMS identified
significant environmental impacts include:

-   *Releases* (wastewater, air pollutants, greenhouse gases, solid
    waste, hazardous waste):

```{=html}
<!-- -->
```
-   Change in total amounts

-   Change in facility risk or indexed release

```{=html}
<!-- -->
```
-   *Resources consumption:* non-renewable energy use, water use, raw
    materials used, reuse and recycling of materials, etc.

-   *Habitat conditions:* Area of land converted/developed, percent of
    land covered by impervious surfaces, etc.

*Environmental Indicators*

5.  For each environmental goal, what qualitative and quantitative
    environmental indicators (i.e., EMS objectives and targets) are
    being used by individual facility EMSs to measure progress/impacts?
    (*Examples include releases---wastewater, air, pollutants,
    greenhouse gases; resource consumption---nonrenewable energy use,
    water use; habitat conditions---area of land converted/developed;
    significant impact areas versus disinvested areas.) Note: The
    indicators should cover both compliance measures [and]{.underline}
    those indicators of environmental performance integrated in the
    EMS.*

6.  What project-wide indicators are being used to measure EMS
    effectiveness? *(Examples include: increased management review and
    attention to environmental outcomes, increased P2 as a result of
    EMS)*

7.  Does the adoption of an EMS change the organization's use of
    environmental indicators (e.g., organization pays more attention to
    additional unregulated indicators)?

8.  What is the measurement approach that will be used to measure
    progress for each environmental goal (e.g., modeling data, in-situ
    experiment, data extrapolation, real-time, one-time observations)?

9.  For each environmental indicator, what is the pre-project "baseline"
    against which progress is measured (e.g., baseline is 5 percent loss
    of product in facility waste streams---the EMS project is to only
    have 1 percent material loss in three years) in the EMS?

    a.  Baseline compliance indicators (e.g., frequency of
        noncompliance)?

    b.  Baseline process indicators (e.g., adoption of pollution
        prevention techniques, increase in senior management review of
        facility environmental footprint, consideration of local
        environmental issues)?

10. For each of the environmental indicators listed above, what is the
    schedule for data collection (daily, weekly, quarterly, annually,
    etc.) and how is it reflected in each entity's EMS or in the EMS
    project?

11. Based on the indicators provided through EMS objectives and targets,
    what have been the environmental impacts of the project (*e.g., 100
    tons of volatile organic compounds emissions have been eliminated to
    date)? Note: Provide both qualitative and quantitative outcomes. The
    project may be of too recent origin for environmental impacts to be
    observable. Provide qualitative outcomes if possible---e.g.,
    increase in senior management review, etc.*

*Environmental Results of the EMS Project.* By comparing the pre-project
baseline environmental results to the results during implementation or
post-implementation, the practitioner can determine the net change. This
will allow for mid-course corrections, if necessary, or a determination
as to whether the project had the desired outcomes.

An organizing table is provided in Exhibit 2. This model table is
neither intended to be comprehensive, nor to anticipate every kind of
project for which practitioners will want to assess environmental
results. It should be modified to suit the user's project as the
elements provided in Exhibit 2 below are for illustrative purposes only.

*Data Sources, Collection, and Verifiability.* It is recommended that
the project team identify early in the process the data sources,
collection and monitoring protocols, frequency of collection, persons
responsible for data collection, and methods for data verification and
quality control. By establishing procedures up front, it will be easier
during implementation and at the evaluation phase of a project to
assemble the proper data to determine credible environmental results.

*Environmental Results*

12. To what extent are the environmental impacts of the project
    consistent with what was expected at the time of design and
    implementation?

13. Are sufficient data available to determine if the project has met
    its environmental goals (e.g., are the data qualitative or
    quantitative or both)?

14. To what extent has the EMS project been an improvement over the
    prior/traditional approach with regard to:

    i.  Human health

    ii. Organizational management

    iii. Community based protection

    iv. Quality of life

    v.  Ecosystem health

    vi. Tribal management

    vii. Environmental Justice communities

    viii. Others

15. How are **environmental results verified and who is responsible for
    verifying results**? How is this reflected in the EMS project or in
    a facility's EMS?

16. How often are environmental results verified? How is this reflected
    in the EMS project or in a facility's EMS?

######## **Cross-Media Performance.** EMS-related projects may determine whether a project leads to significant improvements in overall environmental quality due to decisions to redirect resources from non or less-significant impact areas to address high environmental impact areas, based on an EMS cross-media analysis. Please answer these questions if applicable.

*Cross-Media Analysis*

17. What is the analytical basis used for cross-media shifts for
    facilities with EMSs (e.g. comparative risk based on acute and/or
    chronic toxicity)?

18. Has a cross-media analysis been performed? Does the analysis help
    provide a baseline to measure cross-media performance?

19. What are the anticipated effects of the cross-media transfers?

20. How does the intended goal and indicators compare to the results of
    the cross-media transfers?

21. Have any new multimedia strategies been identified based on the
    cross-media analysis?

Exhibit 2 -- Environmental Results Table

***Project:*** *Pallet Waste-to-Flooring Demonstration Project*

**Problem the project is trying to solve:** Demonstrate the technical
and financial feasibility of recycling waste pallet wood into a value
added flooring product

+---------------+----------+----------+-----------+--------+---------+
| Project       | In       | Pre      | Output    | S      | Impact/ |
| Objectives    | dicators | -project | Metrics   | ources |         |
|               | (EMS     |          |           | of     | Outcome |
| with Goals    | ob       | Base     |           | info   |         |
|               | jectives | St       |           | and    |         |
|               | and      | atistics |           | Calcul |         |
|               | targets) |          |           | ations |         |
+===============+==========+==========+===========+========+=========+
| Produce       | R        | Pallet   | \# square | Prod   | Natural |
| recycle       | egulator | H        | feet of   | uction | r       |
| pallet        | Indicato | ardwoods | flooring  | r      | esource |
| flooring      | rs---fed | used in  | produced  | ecords | conserv |
|               | eral/sta | U.S:     | (from     |        | ation - |
| Goals sq. ft  | te/Local |          | recycled  |        | Es      |
|               |          | 4.5      | pallets)  |        | timated |
|               |          | billion  |           |        | per     |
|               |          | board    |           |        | centage |
|               |          | feet/yr  |           |        | (i      |
|               |          | (1998)   |           |        | ncrease |
|               |          |          |           |        | c       |
|               |          |          |           |        | ompared |
|               |          |          |           |        | to      |
|               |          |          |           |        | ba      |
|               |          |          |           |        | seline) |
|               |          |          |           |        | of      |
|               |          |          |           |        | h       |
|               |          |          |           |        | ardwood |
|               |          |          |           |        | tree    |
|               |          |          |           |        | c       |
|               |          |          |           |        | onserve |
|               |          |          |           |        | (tree   |
|               |          |          |           |        | s/yr)\* |
+---------------+----------+----------+-----------+--------+---------+
|               | Non-re   |          |           |        |         |
|               | gulatory |          |           |        |         |
|               | in       |          |           |        |         |
|               | dicators |          |           |        |         |
+---------------+----------+----------+-----------+--------+---------+
|               |          |          | \# trees  | Calcu  | Per     |
|               |          |          | not cut   | lation | centage |
|               |          |          | for       | based  | of      |
|               |          |          | f         | on     | trees   |
|               |          |          | looring\* | prod   | saved   |
|               |          |          |           | uction | from    |
|               |          |          |           | rec    | har     |
|               |          |          |           | ords\* | vesting |
|               |          |          |           |        | when    |
|               |          |          |           |        | c       |
|               |          |          |           |        | ompared |
|               |          |          |           |        | to      |
|               |          |          |           |        | b       |
|               |          |          |           |        | aseline |
+---------------+----------+----------+-----------+--------+---------+
| Divert Pallet |          | Over     | \# square | Prod   | Conse   |
| wood waste    |          | 305,000  | feet of   | uction | rvation |
| from          |          | tons per | pallet    | r      | of      |
| land          |          | year of  | boards    | ecords | landfil |
| filling/waste |          | wood     | diverted  |        | l/waste |
| management    |          | pallets  |           |        | man     |
| methods       |          | are      |           |        | agement |
|               |          | disposed |           |        | c       |
|               |          | in       |           |        | apacity |
|               |          | l        |           |        | (t      |
|               |          | andfills |           |        | ons/yr) |
|               |          | in NC    |           |        | c       |
|               |          | (1998)   |           |        | ompared |
|               |          |          |           |        | to the  |
|               |          |          |           |        | b       |
|               |          |          |           |        | aseline |
+---------------+----------+----------+-----------+--------+---------+
|               |          |          | Tons of   | Calcul |         |
|               |          |          | Pallet    | ations |         |
|               |          |          | wood      | based  |         |
|               |          |          | diverted  | on     |         |
|               |          |          | from      | prod   |         |
|               |          |          | la        | uction |         |
|               |          |          | ndfilling | r      |         |
|               |          |          |           | ecords |         |
+---------------+----------+----------+-----------+--------+---------+
| Reduce        |          | No       | MTCE      | Model  | \% GHG  |
| greenhouse    |          | st       |           | too    | methane |
| gases through |          | atistics |           | ls\*\* | re      |
| carbon        |          |          |           |        | duction |
| sequestration |          |          |           |        | MTCE    |
| and landfill  |          |          |           |        |         |
| methane       |          |          |           |        |         |
| reduction.    |          |          |           |        |         |
+---------------+----------+----------+-----------+--------+---------+

\*100 board ft/ 1 tree -- USFS estimates based on trees 12" in diameter
(DBH) with 2.5 16 foot long logs. Conversion factors: 0.625 board
ft/ft\^2 of finished flooring (for a 3/8" thick flooring product)

\*\*Metric tons of Carbon Equivalent (MTCE). Emission factors for wood:
(Methane generation in landfill: 0.170 MTCE/wet ton) + (Carbon storage
of wood: 0.21 MTCE wet ton) = 0.39 MTCE per wet ton. Source: SOLID WASTE
MANAGEMENT AND GREENHOUSE GASES, A Life-Cycle Assessment of Emissions
and Sinks, 2nd EDITION, EPA530-R-02-006, May 2002

Number range CHAPTERMODULE 3: Assessing the Costs and Cost Savings of
The EMS Project

This module assists users in determining the costs and cost savings
associated with an EMS project. Many projects are aimed at improving
resource efficiency for the regulator or regulated community so that
scarce resources may be re-allocated to other environmental priorities.
For example, if a permit streamlining project results in less permit
review time by personnel within the regulatory agency, these full-time
employees (FTEs) may now address other important problems. In addition
to calculating resource efficiencies, it may be possible with some EMS
projects to conduct a cost-effectiveness analysis with the goal of
minimizing the costs of achieving particular policy goals. For example,
if an EMS project results in reducing the compliance costs per ton of
pollutant, the project is more cost effective than the traditional way
of doing business. In most cases, the project team will not have the
resources to conduct formal cost-benefit or economic analyses, nor are
such analyses appropriate in all cases.[^4] Module 3 helps assess if the
project is more cost-effective than standard practice and assists the
user in identifying ways in which the project can be more efficient.
This module does not address a costs and benefits assessment of
ecological or environmental attributes (e.g., economic value of a
wetlands or health benefits).

An EMS project may address a number of objectives that can be examined
for costs and cost-effectiveness. Examples include: 1) Decreasing permit
development and implementation times; 2) Decreasing costs (capital,
labor) for permit development and implementation; 3) Decreasing permit
complexity and permit development process; and 4) Redirecting regulatory
oversight from lower to higher priority areas to increase the proportion
of time spent addressing "high risk" activities relative to time spent
addressing "low risk" activities. It is important that the appropriate
indicators (as discussed in Module 2) are developed and that costs and
cost-savings are addressed in the design phase, implementation, and at
the end of the project.

*Design and Planning Phase*

The user should design the project to ensure that information is
collected on the expected cost-savings and costs of the project. It is
important at this phase to collect and assess the baseline costs
[before]{.underline} the project is implemented. For example, the user
should attempt to quantify FTEs, budget, time etc. of the traditional
way of doing business of a traditional permitting system. If the project
is a streamlined facility-wide permit, the design of the project should
include an estimation of the expected cost-savings (e.g., time, money,
FTE etc.) and methods to collect this data so that the cost-savings can
be compared to the traditional costs to demonstrate a relative cost
advantage of the project.

*Implementation Phase*

The user should be tracking information on the cost-savings and costs of
the project. Often, higher transaction costs are associated with the
initial implementation of project. The user should anticipate these
costs, but also track costs to see if those costs decrease with time
over the life of the project, and that expected cost-savings projected
in the design phase are being realized. It is at this phase where the
practitioner could be looking for anecdotal data and impressions
regarding costs and cost-effectiveness. Most importantly, cost and cost
savings information is needed in order to help determine if there needs
to be a mid-course correction of the project and if the project can
eventually be transferable.

*End of the EMS Project Phase*

Consider if a full cost-benefit analysis is needed or warranted. For
more information on economic analyses, please see EPA's *Guidelines for
Preparing Economic Analyses at:*

(<http://yosemite.epa.gov/ee/epa/eed.nsf/Webpages/Guidelines.html>). For
the purposes of this module, looking at the transferability and
evaluation of the project, the user should focus on whether or not the
EMS project poses a relative cost-savings advantage over the traditional
way of environmental protection. Costs and cost savings do not have to
be limited to transaction costs and cost savings, but can also include
the amount of job creation, cost savings incurred by faster time to
market, or property redevelopment benefits, etc.

*Formal Evaluation*

A cost and cost-savings evaluation addresses how much the EMS project or
project components cost, preferably in relation to alternative uses of
the same resources and to the benefits being produced by the EMS
project. The cost evaluation will include a description of the costs and
savings associated with the project as well as an analysis of the
efficiency, productivity and cost-effectiveness of the evaluation. The
evaluation will focus on the "how and why" resources invested achieved
the intended outcomes. The practitioner may decide that a formal
cost-benefit analysis is needed outside of a cost-evaluation.

I. Measuring the Costs and Cost Savings of the EMS Project

Quantifying the costs and cost savings associated with the project may
require use of data sources from different organizations (i.e.,
regulated entity, governmental body, or other stakeholder group). Cost
estimates will likely involve assumptions or uncertainties that users
will want to identify and acknowledge, particularly if the EMS project
is transferable.

*Measurement Approach.* In most instances, the user will use a simple
direct compliance cost method to analyze costs and cost savings. This
approach involves quantifying the compliance costs and cost savings
realized or incurred organization(s) implementing the project. The costs
may include the capital costs associated with new technologies; the
costs of operating and maintaining that new equipment; the costs of
modifying operations to comply with the project; and the costs of
complying with the project's monitoring, record keeping, and reporting
requirements. In addition, the analysis can quantify the costs that
state regulatory authorities and EPA will incur in administering the
project. Analysis of these costs is often likely to provide a reasonable
approximation of the total social costs of the project. A similar
analysis can be performed for the cost savings of the project. For EMS
projects, the practitioner can look at the following potential
indicators for the cost and cost-effectiveness analysis:

-   Permitting cycle time

-   FTEs and dollars spent on permit development and implementation

-   Amount of permit backlog

-   Number of permitting process steps (for the regulated and regulator)

-   Number of (low-priority) inspections avoided

-   Number of (low-priority) permit modifications avoided

-   Rate errors and rework on permits

-   Amount of information (pages) required

-   FTEs and dollars saved from avoiding inspections, permit
    modifications, and/or other low-priority activities

*Baseline Analysis.* Similar to the analysis undertaken to determine
environmental results in Module 2, the user must also characterize
baseline cost conditions to determine the net change in costs. Without a
baseline, there is no frame of reference for the change that the project
proposes and it is difficult to say if the project poses a relative
advantage to the traditional system.

*Data Sources.* Sources of cost and cost savings information should be
identified if possible at the outset of the project. By recording the
staff time incurred by the organization, governmental entities, and
stakeholders during project development and implementation, the user
will have a better estimate of the real costs of the project.

*Costs or Cost Savings of the EMS Project.* By comparing the pre-project
baseline costs to the costs during implementation or
post-implementation, the practitioner can determine the net change as a
result of the EMS project. This will allow for mid-course corrections,
if necessary, or a determination as to whether the project had the
desired outcomes.

1.  What is the measurement approach that will be used to estimate the
    costs and cost savings of the project? What indicators will be used
    *(e.g., compliance measures, materials use, numbers of spills,
    etc.)*?

2.  Does the adoption of an EMS change the organization's use of
    economic performance indicators (e.g., identification of
    environment-related costs and benefits more explicitly for
    management attention)?

3.  What are the pre-project baselines against which costs and cost
    savings are measured?

    a.  Costs of compliance

    b.  Costs of EMS (i.e., design, training, implementation, and
        certification)

    c.  Cost savings of streamlined permitting system

    d.  Cost savings of reallocation of personnel

    e.  Other (e.g., new investments, time to market, competitiveness)

```{=html}
<!-- -->
```
4.  What data sources will be used to measure costs and cost savings?

5.  To what extent has the project resulted in costs or cost savings?

####### Cost Savings of the EMS Project

The user will need to estimate cost savings resulting from the project
in comparison to the cost-savings that would be incurred or generated in
the absence of the project. For the purposes cost savings analyses, cost
savings are represented as savings in time, personnel, capital,
operation and maintenance, transactional costs, and economic activity.

6\. What significant **time savings/savings** has the organization
derived as a result of the EMS project? *(Please describe the key types
of time savings incurred including staff time and contractor savings
involved in activities including project development, implementation,
monitoring, reporting and record keeping, rule revisions, permit
administration, and inspections.)*

a.  Savings to the regulator

b.  Savings to the regulated?

c.  Local community or other stakeholder group savings?

7\. What **significant cost savings in capital, operation, and
maintenance of new equipment, operation and maintenance of existing
equipment, materials, or energy** has the organization derived as a
result of the EMS project?

8\. What **other savings** (e.g., insurance, worker compensation,
creation of jobs etc.) has the organization derived as a result of the
EMS project?

9\. What **economic activity**, if any, has been generated by
implementation of the EMS project (e.g., jobs may be created if a
brownfields site is redeveloped)?

# Costs of the EMS Project

Costs that are frequently feasible to quantify include compliance costs,
government regulatory costs, and transaction costs.[^5] The practitioner
may want to consider the costs and cost differences associated with
implementing an EMS project when compared to the costs of the
traditional approach. The five basic categories of cost include:

-   Real-resource compliance costs: the costs associated with changing
    production processes or with purchasing, installing, operating and
    maintaining new equipment to comply with the project or traditional
    regulations. The costs can be fixed that require an investment over
    a period of time or they can be variable as a per unit cost.

-   Government or stakeholder costs: any public sector administrative,
    training, permitting, monitoring, reporting, or enforcement costs
    associated with the EMS project or traditional approach.

-   Social costs: costs associated with a rise in price of goods or the
    decrease in the production of goods and services as a result of the
    project or traditional approach.

-   Transitional cost: the costs associated with changes in processes or
    production due to the project or traditional approach---these costs
    could be a disruption in production or the costs of retiring old
    equipment and changing to new technology.

-   Indirect costs: the unintended costs the project or traditional
    approach may have on product quality and productivity, or on markets
    for other goods and services.

The user should consider each of these potential costs, however, it is
not always necessary or feasible to quantify costs in each category. For
example, if real-resource compliance costs are likely to be small,
social costs and transitional costs may be insignificant. Similarly, if
the resources available for an analysis are limited, it may not be
feasible to model indirect costs. The user will want to indicate which
costs can be quantified, and which will be addressed qualitatively. An
example of a qualitative cost may be changes in organizational
management to account for an EMS.

Please describe the costs in terms of the real-resource compliance
costs, social welfare costs, transitional costs, and/or indirect costs.

10\. What significant **time costs/investments** has the organization
incurred as a result of the EMS project? *(Please describe the key types
of costs incurred including staff time and contractor costs involved in
activities including project development, implementation, monitoring,
reporting and record keeping, rule revisions, permit administration, and
inspections.)*

a.  Costs to the regulator

b.  Costs to the regulated

c.  Costs to the local community or other stakeholders

```{=html}
<!-- -->
```
11. What significant **costs/investments in capital, operation, and
    maintenance of new equipment, operation and maintenance of existing
    equipment, materials, or energy** has the organization incurred as a
    result of the EMS project?

12. What **other significant costs** (e.g., insurance, worker
    compensation, creation of jobs etc.) has the organization incurred
    as a result of the EMS project?

IV\. Relative Cost Advantage

The user should look at costs and cost savings of the project relative
to the system prior to implementing the project.

13\. If the EMS project were used more widely in the future, how would
the **marginal** (i.e., per EMS project) **savings and costs** of the
project change for **the organization**?

a.  For the regulator

b.  For the regulated

c.  For the local community and other stakeholders

14\. What is the **difference** between the actual project costs and
baseline costs (i.e., costs associated with current regulatory
framework)?

Exhibit 3 below is an organizing table that is intended as a model only.
It is neither intended to be comprehensive, nor to anticipate every kind
of cost associated with a project.

Exhibit 3

+-----------------+-----------------+-----------------+-----------------+
| Costs/Cost      |                 |                 |                 |
| Savings Model   |                 |                 |                 |
| Table           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Category of     | Baseline Costs  | Costs of the    | Net Change:     |
| Costs           |                 | Project         |                 |
|                 |                 |                 | Costs or Cost   |
|                 |                 |                 | Savings         |
+-----------------+-----------------+-----------------+-----------------+
| Real-Resource   |                 |                 |                 |
| Compliance      |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Project         |                 |                 |                 |
| Development     |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Capital Costs   |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Operation and   |                 |                 |                 |
| Maintenance     |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Government      |                 |                 |                 |
| Regulatory      |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Permit Review   |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Inspection      |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Social Costs    |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Cost of goods   |                 |                 |                 |
| with recycled   |                 |                 |                 |
| materials       |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Transitional    |                 |                 |                 |
| Costs           |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
|                 |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
| Indirect Costs  |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+
|                 |                 |                 |                 |
+-----------------+-----------------+-----------------+-----------------+

Number range CHAPTER**MODULE 4: Enforcement and Compliance Assurance**

This module evaluates the project's reporting requirements,
accountability, enforceability and effectiveness compared to the
traditional reporting requirements. Completion of this module may
require input from EPA, state, and local enforcement and compliance
assurance personnel at the design, implementation, and evaluation phases
of the project.

EMS projects can be fairly complex for enforcement and compliance,
especially if the project involves developing permits based on the EMS.
This module asks the user to look at requirements for monitoring,
reporting, and record-keeping to demonstrate compliance and how these
requirements would relate to an EMS project. Potential objectives of an
EMS project include: 1) Redirecting regulatory oversight from lower to
higher priority areas; and 2) Increasing the proportion of time spent
addressing "high risk" activities relative to time spent addressing "low
risk" activities. These objectives are addressed with targeted questions
found at the end of this module.

*Design and Planning Phase*

To ensure that all substantive and procedural requirements of the
project are met, the appropriate scope, timing, and availability of all
monitoring, reporting, and record-keeping (MRR) requirements should be
included in the EMS project at the design phase. For an EMS-based
project, the user should attempt to clearly delineate responsibilities
for the facility, regulator, and expectations for third-party
verification, if any. For example, what will inspectors look for in an
EMS based permit? What compliance information would be covered in the
EMS and what compliance information would belong in the permit?

*Implementation Phase*

During the implementation phase, regulatory authorities should review
and track information submitted by the facility to monitor compliance
and identify problems or trends that may require mid-course adjustments.

*End of the EMS Project Phase*

To verify and evaluate the results of the project, compliance assurance
and enforcement staff may need to complete final record reviews and
conduct a facility inspection or use alternative methods to verify
results. Federal, state, and local regulatory authorities should decide
in advance how to allocate these responsibilities to conserve scarce
resources.

*Formal Evaluation*

To evaluate the enforcement and compliance assurance, it is essential
that the practitioner keep contextual factors in mind when asking how
and why certain results were achieved. For example, the EMS project's
results clearly show a rapid rise in the rate of compliance from a
target sector. The evaluation needs to ask why compliance changed---was
it a direct result of new compliance assistance, was it the result of
the EMS, or was it the result of economic factors or forces outside of
the scope of the EMS project? Enforcement and compliance can have
important short-term outcomes that should be captured and fully
explained in order to make the causal link between the EMS project and
the outcome.

**[Monitoring, Record-Keeping, and Reporting (MRR) to State Agencies,
EPA, and Stakeholders]{.underline}**

The user may choose to standardize the collection and tracking of
monitoring, record-keeping, and reporting information at the outset of
an EMS project to assess compliance with MRR requirements at the design
of the project, throughout implementation, and during evaluation of the
project. Exhibit 4 below is intended to assist EPA, state, and local
regulatory authorities to develop a conceptual framework for verifying
compliance with MRR requirements and to note any deviations. The table
can be modified to meet the needs of the project. For each applicable
requirement, team members will want to identify the monitoring approach
and/or materials use and operating parameter requirements for the
environmental media, specify the frequency of data collection, and
identify the reporting and record-keeping requirements. Depending on the
method of determining compliance (i.e., record review, facility
inspection, etc.), enforcement personnel will determine whether the
project is in compliance with applicable requirements. The user will
want to tailor the table to meet the specific needs of each project, and
may choose to create separate tables for monitoring, reporting, and
record-keeping purposes.

**I. Reporting to State Agencies, EPA, or Stakeholders\
**

###### Design Questions

1.  What is the **legal implementing mechanism** for the project?

2.  What **standard permit conditions or regulatory requirements,** if
    any, will require/have been modified?

```{=html}
<!-- -->
```
3.  What are the specific **requirements for environmental monitoring**
    of this EMS project? Clarify those requirements by law and
    regulation between requirements of the project (i.e., requirements
    over and above those required by law).

4.  What are the specific **requirements for keeping records** of this
    EMS project? Clarify those requirements by law and regulation
    between requirements of the project (i.e., requirements over and
    above those required by law).

5.  What are the specific **requirements for reporting to regulatory
    organizations** regarding this EMS project? Clarify those
    requirements by law and regulation between requirements of the
    project (i.e., requirements over and above those required by law).

6.  What are the specific **requirements for reporting to stakeholders**
    regarding this EMS project? Clarify those requirements by law and
    regulation between requirements of the project (i.e., requirements
    over and above those required by law).

7.  Do the reports have a **required audience(s)**? (Y/N) If yes, please
    identify the audience(s). Clarify those requirements by law and
    regulation between requirements of the innovation (i.e.,
    requirements over and above those required by law).

###### Implementation Questions

8.  To what extent have the specific **requirements for environmental
    monitoring** of this project been met?

9.  To what extent have the specific **requirements for keeping
    records** been met?

10. To what extent have the specific **requirements for reporting to
    regulatory organizations** been met?

11. To what extent have the specific **requirements for reporting to
    stakeholders** been met?

12. Have reports been delivered to the required audiences identified in
    Question 7? (Yes/No) If yes, please list dates and method of
    communication (e.g., Web site, email public notice). If no, please
    provide explanation.

[Compliance Assurance with a "Project Agreement"]{.underline}

For the purposes of this module, the term "project agreement" is being
used to cover EMS type projects. EMS project team members will want to
structure the EMS project agreement carefully to ensure that all
applicable requirements are met and function within the current
regulatory framework, unless rule revisions are contemplated by the
project. Designers of the project agreement will want to address all
substantive requirements (e.g., technology, emissions or effluent
performance, work practice requirements, etc.) and procedural
requirements (e.g., public notification, review, comment processes;
potential termination of the project; and reporting and information
availability requirements).

# Verification

13. How do users ensure that the parties to the EMS project comply with
    the provisions of the EMS project?

```{=html}
<!-- -->
```
a.  How will the organization's performance under the project be
    compared to the performance that could have been obtained under the
    normally applicable regulatory structure?

b.  Who is responsible for verifying compliance and environmental
    performance results and how will it be done?

# III. Practical Enforceability 

The user will want to ensure the practical enforceability of the
project. This is accomplished by developing monitoring, record-keeping,
and reporting requirements that enable regulatory authorities to detect
source compliance with all applicable requirements. Compliance personnel
will find the project to be practically enforceable if sufficient data
regarding the project is available and well organized to perform
compliance verification calculations according to established
procedures. Further evidence of the practical enforceability of a
project can occur in the context of inspections and review of
third-party audits of an EMS. If inspectors find that inspecting the EMS
projects is straightforward and comparable to conducting inspections for
sources with conventional approaches, the project will prove to be
practically enforceable.

14\. What is the **pre-project "baseline"** for enforcement and
compliance assurance against which progress will be (is) measured?

15\. Can an inspector visiting the project site **determine** **historic
and current compliance** from the records maintained on site?

16\. Does the EMS permit, if applicable, **contain a legal obligation**
for the source to adhere to the terms and conditions of the limitation?

17\. Does the permit **rely on the** **efficiency of a control
technology** for compliance with a permit limit? If so, how is that
efficiency determined and shown to be accurate?

18\. Does the EMS project agreement **require the correct type and
amount of information** (in logs, notices, monitoring data, etc.) to
determine the number and duration of any deviations?

19\. How will regulators determine---prior to and throughout the
project---that the facility is **continuing to maintain its EMS**?

20\. Who will audit **EMS conformance?**

21\. Does the project include required **self or third-party compliance
auditing**? If yes, explain.

22\. Do the terms of the project obligate a regulator to exercise its
**enforcement discretion** in specific ways (if so, explain)?

23\. Does the regulatory preserve the requisite statutory **inspection
and enforcement authority** to satisfy EPA-state delegations of
authority?

24\. How, and for what reasons, will the **organization return to
standard permit terms** should it become necessary to terminate the
organization's participation in the EMS project?

####### Redirecting Regulatory Oversight

25. What **screening criteria** (e.g., compliance history or
    participation in leadership programs) are used to ensure that good
    facility partners participate in the EMS project?

26. If applicable, what combination of **EMS conditions and facility
    characteristics** is being used to establish the confidence or the
    analytical basis for redirecting resources (e.g., compliance
    history, transparency of decision-making, quality and degree of
    public involvement, third-party auditing, reporting, etc.)?

27. What is the analytical basis being used for **determining the
    relative priority** **or risk of agency activities** (for the
    purpose of targeting staff time and resources)?

28. What are the **net resource implications** to compliance and
    enforcement personnel/programs of substituting EMS terms for permit
    conditions or linking them to the permit conditions?

**V. Results and Relative Advantage**

29\. Has the project **improved compliance rates** and/or reduced the
length or severity of noncompliance?

30\. To what extent **can the source** in the project **be more/less
easily inspected** to determine compliance than a similar source
operating under conventional approaches? How does this compare to what
is expected?

31\. Does the project **improve on enforcement or enforcement
practices** over the current system (e.g., redirection of oversight
resources)?

32\. Does the adoption of an EMS allow organizations to **remove
regulatory burdens** by moving down in "regulatory status" (e.g., moving
from large quantity generator to small quantity generator status)? If
yes, describe.

Exhibit 4

+----------+--------+---------+-------+----------+----------+-------+
| Model    |        |         |       |          |          |       |
| Table    |        |         |       |          |          |       |
| for      |        |         |       |          |          |       |
| Mon      |        |         |       |          |          |       |
| itoring, |        |         |       |          |          |       |
| Re       |        |         |       |          |          |       |
| porting, |        |         |       |          |          |       |
| and      |        |         |       |          |          |       |
| Record   |        |         |       |          |          |       |
| -Keeping |        |         |       |          |          |       |
| (MRR)    |        |         |       |          |          |       |
+----------+--------+---------+-------+----------+----------+-------+
| Envir    | Moni   | Ma      | Data  | R        | Record   | Compl |
| onmental | toring | terials | Colle | eporting | -Keeping | iance |
| Media    | Ap     | Use and | ction | Requ     | Requ     | Notes |
| and      | proach | Op      | Freq  | irements | irements |       |
| Po       | (conti | erating | uency | for      |          | (sp   |
| llutants | nuous, | Pa      |       | Re       |          | ecify |
| of       | param  | rameter |       | gulatory |          | date  |
| Concern  | etric, | Requi   |       | Aut      |          | of    |
|          | anal   | rements |       | horities |          | r     |
|          | ytical | (e.g.,  |       | and      |          | eport |
|          | te     | appl    |       | Stak     |          | and   |
|          | sting, | ication |       | eholders |          | note  |
|          | com    | rate,   |       |          |          | any   |
|          | posite | per     |       |          |          | d     |
|          | s      | centage |       |          |          | eviat |
|          | ample, | by      |       |          |          | ions) |
|          | grab   | weight) |       |          |          |       |
|          | s      |         |       |          |          |       |
|          | ample) |         |       |          |          |       |
+----------+--------+---------+-------+----------+----------+-------+
| Air      |        |         |       |          |          |       |
| E        |        |         |       |          |          |       |
| missions |        |         |       |          |          |       |
| by       |        |         |       |          |          |       |
| P        |        |         |       |          |          |       |
| ollutant |        |         |       |          |          |       |
| (tons    |        |         |       |          |          |       |
| per      |        |         |       |          |          |       |
| year)    |        |         |       |          |          |       |
+----------+--------+---------+-------+----------+----------+-------+
| Average  |        |         |       |          |          |       |
| Effluent |        |         |       |          |          |       |
| Concen   |        |         |       |          |          |       |
| trations |        |         |       |          |          |       |
| by       |        |         |       |          |          |       |
| Con      |        |         |       |          |          |       |
| stituent |        |         |       |          |          |       |
| (mg/L)   |        |         |       |          |          |       |
+----------+--------+---------+-------+----------+----------+-------+
| H        |        |         |       |          |          |       |
| azardous |        |         |       |          |          |       |
| Waste    |        |         |       |          |          |       |
| G        |        |         |       |          |          |       |
| enerated |        |         |       |          |          |       |
| (pounds) |        |         |       |          |          |       |
+----------+--------+---------+-------+----------+----------+-------+

**MODULE 5: Public Involvement and Stakeholder Feedback**

An EMS project should maintain or enhance existing standards of public
involvement and should not shift the burden of risk to disadvantaged
populations or diminish the traditional levels of safety provided for
workers responsible for creating, maintaining, or implementing the
project. In addition, in order to be accountable to the public, the
project should be enforceable and produce measurable and verifiable
environmental results. The environmental and enforceability evaluation
modules (Modules 2 and 4) seek to understand these latter issues of
concern. The public involvement and accountability module seeks to
understand the former.

EPA defines public involvement as the full range of activities that are
used to engage the American people in decision-making processes. Public
involvement is a progression that starts with outreach to build
awareness and interest. It evolves to information exchange, through
collaboration and recommendation to agreement and decision-making. The
public may include private individuals, environmental or other advocacy
groups, environmental justice groups, indigenous peoples, minority and
ethnic groups, business and industrial interests (including small
businesses), elected and appointed public officials, trade associations,
or research and governmental associations.

Public involvement in the development and implementation of innovative
projects is fundamental to ensure a transparent process that fosters
trust and works to enhance the relationships between the public, the
regulated community, and regulators. Public participation can benefit
the project, the regulated entity, the regulator, and the public by
increasing awareness about innovative projects and their environmental
benefits and impacts, developing measurable and verifiable environmental
results, preventing shifts in risk burdens to disadvantaged populations,
ensuring worker safety and protections are maintained, and enhancing the
level of information available to the public. By involving the public
early in the project, practitioners will have the benefit of the
public's guidance, experience, and input. Not everyone may choose to be
an active participant; however, the goal should be to provide
opportunities for people to engage at every point along the progression.
Individuals and groups should decide for themselves whether, when, and
how to participate. EPA issued a new Public Involvement Policy in May
2003, which contains useful tips on implementing effective public
involvement strategies and helps define different stakeholder groups.
The policy also provides useful tools to assist practitioners in the
public involvement process at the public involvement Web site:
<http://www.epa.gov/publicinvolvement/intro.htm>.

**[Stakeholder Participation]{.underline}**

The first step in stakeholder involvement is to identify key
participants who may be interested in, or affected by, the project.
Depending on the nature of the project and who is implementing the
project, stakeholders may include members of the regulated community,
community groups, environmental or other advocacy groups, governmental
entities, trade associations, and others. The user may also want to
consult with EPA, state, tribal, and local government partners. To alert
the public in low-income and minority communities regarding the
opportunity to become involved in a project, practitioners should
consider using various media such as advertising in local newspapers,
making announcements on radio stations, and communicating through local
institutions such as religious establishments. Please see EPA's Public
Involvement Policy for more information on involving different
stakeholder groups and stakeholder needs. The key questions to ask are:

1.  Who are the key stakeholders?

2.  Have state, tribal, and local government partners been consulted?

3.  Have stakeholders and partners been consulted in the development
    and/or implementation of the EMS project? If yes, describe the
    extent of the participation.

4.  How did stakeholder involvement change as a result of the adoption
    of the EMS or the implementation of the EMS program?

5.  What benefits does this involvement provide (e.g., ideas not
    otherwise considered, more positive community relations)?

6.  What effects does this involvement have on the decisions made by
    organizations or by the program?

7.  If applicable, what specific strategies are being considered to
    ensure the participation of low-income and minority communities?

8.  What is the **pre-project "baseline"** for public involvement and
    accountability against which progress will be measured?

9.  How does the project address **regulatory requireme**nts
    (federal/state/local/tribal) for public involvement?

10. What changes to the **transparency in decision-making** (for the
    regulator and/or the regulated) and the degree of stakeholder/public
    leverage result from the EMS project?

II Constructive Dialogue Approaches

The best means of involving stakeholders in the development and
implementation of the EMS project depends on the number and diversity of
the parties to be consulted, the geographic impacts of the project, the
resources available to engage in consultative processes, and the type of
communication networks generally used by particular stakeholders.
Constructive dialogue approaches may include outreach activities,
information exchange, the solicitation of stakeholder advice or
recommendations, technical workgroups, Web-based dialogues, Citizen
Advisory Committees, and stakeholder negotiations. Collaborative
processes encourage an interactive and dynamic discussion that may lead
to greater clarification of the issues and consensus among the parties.

11. What are the best means of involving stakeholders in the development
    of the EMS project?

12. What types of collaborative processes or other participatory
    practices will be used to solicit input?

III\. Availability of Information

For information to be readily available to stakeholders, it must be both
understandable and accessible. The user must prepare "plain English" (or
other appropriate language) summaries and fact sheets to facilitate
comprehension of otherwise complex environmental concepts. Communication
materials must engage participants at all levels---from members of the
general public to experts in the field---and make available project
materials such as innovative permits, progress reports, annual reports,
emissions, and/or effluent data, etc. Materials must be translated when
appropriate so that diverse populations have access to the information.
In addition, the user must seek ways to ensure the broadest
participation feasible and should work with all identified partners to
enhance information distribution to all potentially interested parties.
Public information meetings can provide a valuable and interactive means
for communication with interested stakeholders.

13. Is information regarding the EMS project readily available to
    stakeholders? If yes, how will it be shared?

14. What changes to the type, scope, amount, quality (accuracy,
    relevance), and timing of information available to the public result
    from the project?

# IV. Stakeholder Feedback

Stakeholder involvement needs to be carefully planned to allow
sufficient time for discussion of the relevant issues with the
stakeholders and to incorporate their feedback in the project. If
possible, how stakeholder feedback will be used in the project should be
communicated and explained to participants from the start of the
project. The user should anticipate both positive and negative feedback
from stakeholders. The feedback may be content-based or about the
process; and the feedback from one stakeholder group may be at odds with
another group. The practitioner should try to understand the feedback,
work to resolve conflicts, and strive for consensus.

15. At what stage in the project process will stakeholders be involved
    to ensure participation and an opportunity to incorporate feedback?

16. To what extent has the practitioner been successful in obtaining
    feedback from the public about the project's design and/or
    implementation?

# V. Responsiveness to Stakeholder Priorities and Concerns

An EMS project may demand a high level of responsiveness to the
priorities and concerns of stakeholders. The user should develop a
process to address the major concerns of stakeholders. If resources are
available, academic experts may provide objective technical assistance
to stakeholder groups or a facilitator can be used to make sure that all
concerns are heard and addressed. It is important at the outset of a
project to determine what resources will be available to address
stakeholder issues.

17\. How will the major concerns of stakeholders be addressed?

18\. Is technical or financial assistance available to facilitate the
participation of particular groups of stakeholders?

19\. Are stakeholders likely to consider their involvement in the
project a positive experience? Why or Why not?

Number range CHAPTER**MODULE 6: ASSESSING THE POTENTIAL
TRANSFERABILITY**

**OF THE EMS PROJECT**

The purpose of this module is to help reduce the uncertainty about an
EMS project's expected consequences and determine its rate of adoption.
In 1962, Everett Rogers wrote the pioneering work, *Diffusion of
Innovations*[^6], which presented a workable framework for diffusing
innovations or innovative thoughts over time. Rogers' work provides a
systematic approach to understanding the nature of innovations and the
existing conditions and culture necessary for accepting, adopting and
implementing innovations and can be easily applied to EMS projects.

In assessing the potential transferability, practitioners should
consider a set of overview questions, followed by a ranking methodology
based on Rogers' innovation-diffusion model and a sample application of
the transferability module. It is recommended that the user first read
through the questions and ranking methodology to understand the approach
and how these two steps work together. Some of the overview questions
will also be addressed in the ranking process and the ranking will
inform responses to the overview questions. This transferability module
is based on Rogers' innovation-diffusion model, which has five
components: 1) relative advantage, 2) compatibility, 3) complexity, 4)
trialability, and 5) observability. This module provides a definition of
each component as it relates to environmental projects and key questions
related to each component. The practitioner will find that the five
other modules also ask questions of relative advantage and those
responses can be applied in this module.

***I. Relative Advantage*** is the degree to which the EMS project is
perceived as being better than the idea it supersedes. The degree of
relative advantage is often expressed as enhanced environmental
protection, reduced risk to public health, costs savings in meeting
regulatory requirements, recognition for being an environmental leader,
administrative streamlining, increased public involvement, or other
benefits over the traditional approach.

Within this component, it is also important to identify who benefits
from the EMS project or program. If all parties to the project including
the regulated community, the public, and federal and state environmental
regulators benefit from the project, the relative advantage can be
easier to ascertain. If, however, regulated entities perceive a relative
advantage while the public perceives a disadvantage, the project may
need to be better communicated to the public and stakeholders or it may
need modification prior to scale-up.

A final question regarding the relative advantage component is whether
additional data is needed to inform this determination. If additional
information is required prior to making this assessment, it may be
necessary to go back and re-check Module 2: *Assessing the Environmental
Results of the* *EMS Project* to see where there are data gaps in data
collection or methodology. It is often the case that more complete
information is necessary to make a determination of the potential for
broad-scale application of the project.

1.  Compared to the traditional way of doing business, what has been the
    measurable impact (positive and/or negative) of the EMS project with
    regard to:

    a.  Environmental protection

    b.  Organizational management

    c.  Economic impacts

    d.  Expedited action

    e.  Public involvement

    f.  Accountability

    g.  Environmental Justice

    h.  Administrative burden

    i.  Other areas

2.  Who benefits from the EMS project?

    a.  What do they gain?

3.  Who incurs costs as a result of the EMS project?

    a.  What costs do they incur?

4.  What additional data are necessary to inform determination of the
    relative advantage of the project?

5.  To what extent does the adoption of an EMS change an organization's
    use of advanced environmental and materials accounting techniques?

    ***II. Compatibility*** is the degree to which the project is
    perceived as consistent with the existing values, past experiences,
    and needs of potential adopters. The user should assess how well the
    project is consistent with organizational needs and goals. This
    module asks the user to look at the feasibility to adopt a project
    given the "culture" of those who are affected by the project, the
    users of the project, or the perceivers of the project.

    As in the relative advantage component, it is important to assess
    the project's compatibility with the multiple organizations that
    participate, or may be interested in participating, in the project.
    For example, compatibility with the culture of the regulated entity
    is as important as compatibility with the culture of the
    environmental regulators. If the project is embraced by the
    regulated community, and rejected by the regulators, the project is
    unlikely to be adopted. It is also possible that a project may be
    embraced at the state level and not at the local level for a variety
    of reasons that may, or may not have to do with the project itself,
    but with the availability of resources.

6.  To what extent is the EMS project consistent with existing
    organizational beliefs, values, and/or management approaches?

7.  What is the level of support for the EMS project from:

    a.  Within EPA

    b.  The affected entity or entities

    c.  Other regulated entities

    d.  State agencies

    e.  Federal agencies

    f.  Local community

    g.  Environmental non-governmental organizations (NGOs)

    h.  Environmental Justice groups

        i.  Local government

8.  To what extent has a similar EMS project been tested before?

    a.  Different sector or industry

    b.  Different media

    c.  Different state, EPA region, local government, tribe

    d.  Different community

9.  Among existing users, to what extent does the EMS project support
    organizational goals, (i.e., department, office or divisional goals,
    community goals)?

10. Among existing users, to what extent are organizational changes
    necessary to enable widespread use of the EMS project (what specific
    changes are necessary)?

11. Among potential users, to what extent does a broader user market or
    audience exist for the EMS project?

12. Among potential users, to what extent does the project need
    modifications to be used more broadly (what specific changes are
    necessary)?

13. Who else might use or be interested in the EMS project (e.g.,
    regulated entities not originally contemplated as users of the
    project, or regulators who might be able to transform the project in
    a creative way for other purposes)?

    a.  Other regulated entities

    b.  Other regulators (tribes, local state, EPA region, EPA
        Headquarters)

    c.  Communities

        ***III. Ease of Adoption*** is the degree to which a project is
        perceived as relatively easy to understand and use. If the
        project is complex, the development of assistance materials to
        assist adoption may be considered. Or, if the project has been
        tested before in a different sector, media, governmental entity,
        or community, are there existing users that would be willing to
        provide testimonials, or existing materials that might prove
        helpful?

14. How readily understood is the EMS project?

15. To what extent is assistance necessary, and available, to understand
    and use the EMS project?

16. If the project needs to be brokered, what assistance products are
    available?

    a.  Are they in development?

    b.  Do they need to be created?

        ***IV. Trialability*** is the degree to which a project may be
        experimented on a limited basis. New ideas that can be tried in
        a phased approach are generally adopted more rapidly than
        projects that are not easily implemented in stages. The user
        should identify, to the extent feasible, how the project might
        be divisible, and in what sequence, to solicit feedback for this
        component.

17. To what extent can the project be tried on a temporary basis (i.e.,
    one month, one year, etc.)?

18. To what extent can the project be tried on a limited scale (i.e.,
    fewer facilities initially or with fewer regulatory authorities)?

    ***V. Observability*** is the degree to which the results of a
    project are visible to potential users of the project. If the
    project is targeted at a small group of technical experts,
    dissemination of the idea in a trade journal may be adequate,
    whereas a project that affects a large municipality will need a
    different strategy to assure optimal visibility and broad-scale
    adoption.

19. To what extent are the EMS project results apparent to others?

# VI. Personal Experience and Observations

This set of overview questions asks fundamental questions that must be
addressed in any assessment of transferability potential, including
whether the project represents an improvement, whether it is ready for
diffusion, what the primary drivers and barriers are to scale-up, how to
best implement the project, and at what organizational level?

20. To what extent can the EMS project be considered to be an
    improvement over the traditional way of doing business? In what
    way(s) was the project an improvement?

21. Is the EMS project mature enough to have a full understanding of its
    advantages and disadvantages?

    a.  If not, when will it be possible to gain a full understanding of
        the advantages and disadvantages of the EMS project?

22. What are the primary lessons learned from testing and analyzing the
    EMS project that pertain to its broad-scale application?

23. What is the potential for broader application of the EMS project?

    a.  Could the EMS project be used to address another problem?

24. What are the primary barriers to broader application of the EMS
    project?

25. What are the critical implementation elements needed to overcome the
    barriers to broader application of the EMS project?

26. How would the EMS project best be applied?

    a.  What steps could be taken to facilitate more widespread
        application of the EMS project?

b\. What steps could reduce the transaction costs of the diffusion?

c.  What elements should be scaled-up?

d.  What elements should be changed?

e.  How might other users be identified?

f.  Are there unique circumstances that could impact broader application
    of the EMS project (e.g., window of opportunity)?

    27. Are there resource limitations, if any, which would constrain
        broad-scale application?

    28. At what level---national, state, or local---should the project
        be applied?

```{=html}
<!-- -->
```
b.  What are the appropriate mechanisms for such application?

```{=html}
<!-- -->
```
29. How do users feel about the voluntary incorporation of EMSs with the
    regulatory system?

## [Innovation-Diffusion Model: Using a Transferability Scale]{.underline}

By assessing the potential transferability of an EMS project by the five
diffusion components, the user will be able to better identify
candidates for broad-scale application. Those projects with high scores
on all or more diffusion components are likely to be better candidates
for broad-scale application than those projects with low scores. This
ranking methodology should provide the user some insight into the
potential transferability of a project, but it will not substitute for
the judgment of experienced practitioners and it does not guarantee the
predicted results. Projects can take on a life of their own despite the
best predictions. For example, an EMS project that ranks as a "low" on
the transferability scale may be able to be scaled-up due to
unanticipated events. The converse is true for those projects that may
seem to be highly transferable, and may end up sidelined in the end. The
ranking table should be used to emphasize the strengths of the project
and to continue to improve on the weaknesses. The table can also be used
to help identify priority projects for scale-up. If the user has
multiple projects, but limited resources, the ranking table can help
assist decisions to scale-up those projects which are highly
transferable first.

**Relative Advantage:** If an EMS project were perceived as resulting in
significant environmental benefits and cost savings when compared to the
traditional way of doing business, it would be ranked as high on the
transferability scale for this component. Conversely, if the project
yields environmental results that are no better than the traditional
approach and the costs are increased, it would be ranked as low for the
relative advantage component.

A more difficult case is when environmental results are superior, but
costs are significantly increased. In this case, the user may want to
identify the benefits that are being ranked (i.e., rank the relative
advantage for environmental results as high and the relative advantage
for costs as low). Or, the practitioner may choose the "moderate" rank
to reflect the competing considerations. In either case, the
practitioner is advised to explain the rankings so that the rationale is
transparent.

**Compatibility:** An initial ranking regarding compatibility should
begin with the existing users of the project, and then address
compatibility of the project with potential adopters. For example, in an
EMS project a state project manager may want to do the initial ranking
with other staff and then do a comparability ranking with the
participating regulated entities. Existing users and, to the extent
feasible, potential adopters should be consulted to ascertain whether
the project is, or is likely, to be consistent with organizational
beliefs and management approaches. If not, are there specific changes
that would make the project more compatible? Again, if there are
differences in the compatibility rankings among the participants, or
between existing users and potential users, note these differences and
provide explanations, if possible. The practitioner should use these
differences to honestly assess how feasible adoption of the project will
be, and how to communicate differently with parties to address
differences.

**Ease of Adoption:** A project will be ranked high on the ease of
adoption component if potential adopters readily understand it and
little assistance is needed to implement the project. If the EMS project
is complex, difficult to understand, and requires considerable
assistance to inform adoption of the project, the project will be ranked
low on the ease of adoption component.

**Trialability:** For this diffusion component, a high ranking means
that the project may be tested on a temporary basis, or on a limited
scale, before being fully adopted. A low ranking would indicate that the
project must be tried with a large number of facilities or over a
significant period of time in order to see results, implying that
significant and sustained resources (e.g., capital, personnel) are
required before the project can be fully adopted.

**Observability:** A high ranking means that the project is very visible
to the targeted users and a low ranking means that greater efforts will
need to be made to increase the project's visibility.

  --------------------- ----------------- ----------------- -----------------
  **Transferability                                         
  Scale**                                                   

  **Project-Diffusion   **High**          **Moderate**      **Low**
  Components**                                              

  **Relative                                                
  Advantage**                                               

  **Compatibility**                                         

  **Ease of                                                 
  Adoption**^1^                                             

  **Trialability**                                          

  **Observability**                                         

  ^1^ For ease of                                           
  adoption, "high"                                          
  responses are                                             
  positively related to                                     
  a project's rate of                                       
  adoption, whereas                                         
  "low" responses are                                       
  negative related. For                                     
  trialability, "high"                                      
  responses are                                             
  positively related.                                       
  --------------------- ----------------- ----------------- -----------------

Number range CHAPTER

[^1]: Module 1: Mapping the EMS Project summarizes the overall goal(s)
    of the project, which may include environmental, economic, and
    process-related goals. In this module, the user should focus on the
    environmental goal(s) of the project and whether or not the project
    is achieving its environmental goal(s).

[^2]: This module is not intended to result in a valuation of benefits
    that would occur in the context of a formal benefit-cost analysis.
    Instead, the analyst will assemble available information and use it
    to guide the development of the project.

[^3]: This module is not intended to result in a valuation of benefits
    that would occur in the context of a formal benefit-cost analysis.
    Instead, the analyst will assemble available information and use it
    to guide the development of the innovation.

[^4]: For more information on complete cost-benefit assessments and on
    preparing economic analyses, please see EPA's National Center for
    Environmental Economics Web site.
    http://yosemite.epa.gov/ee/epa/eed.nsf/Webpages/Guidelines.html

[^5]: Economists assign other kinds of costs such as social welfare
    losses (i.e., rise in price or a decrease in the output of goods and
    services as a result of the project that raises prices for consumers
    and may decrease a producer's revenues), transitional costs (i.e.,
    the value of resources that are displaced because of
    innovation-induced reductions in production), and indirect costs
    (i.e. the adverse effects the project may have on product quality
    and productivity, or on markets for other goods and services). These
    types of costs or cost savings are difficult to quantify and
    generally will not be available to the user.

[^6]: ^1^ ^1^Rogers, E. *Diffusion of Innovations.* 4^th^ Edition. The
    Free Press, New York: 1995.