Document ID: EPA-HQ-OA-2007-0607-0003
Agency: epa
Document Type: Supporting & Related Material
Title: 
Posted Date: 2020-08-12T04:00Z

Supporting Statement for

Information Collection Request No. 2255.01

 “Performance Measurement Reporting for Training 

and Education/Outreach” 

PART A OF THE SUPPORTING STATEMENT

1.   	Identification of the Information Collection

This is a request for approval of a new generic Information Collection
Request (ICR).  The purpose of the ICR is to authorize the use of
reporting tools to enable the U.S. Environmental Protection Agency (EPA)
to measure the performance of training, education and outreach
activities performed under EPA’s assistance agreements (including
grants and cooperative agreements).  EPA Order 5700.7, “Environmental
Results under EPA Assistance Agreements,” calls on the EPA program
offices to, among other things, link their assistance agreements to the
Agency’s Strategic Plan and Government Performance Results Act (GPRA)
architecture and ensure that outputs and outcomes are appropriately
addressed through their assistance agreements.  Program offices must
also ensure that interim and final performance reports submitted by
assistance agreement recipients under 40 CFR Parts 30 and 31 address
their progress in achieving the agreed-upon outputs and outcomes.

EPA has developed some generic reporting tools to supplement the
existing performance reports submitted by assistance agreement
recipients under 40 CFR Parts 30 and 31.  The new reporting tools are
designed to collect performance measurement information on two types of
activities performed by assistance agreement recipients:  training and
education/outreach.  Specifically, EPA has created generic questions and
reporting formats that will be made available to the EPA programs and
assistance agreement recipients.  The tools are designed to measure the
extent to which the assistance agreement recipients achieve the
short-term and intermediate outcomes stated in their work plans for
trainings and education/outreach.  The questions and formats are
flexible enough for use by any of the EPA programs.  

An assistance agreement recipient would review the reporting formats and
generic questions, select the format and questions that are appropriate
for its needs, and assemble them into a questionnaire.  The recipient
would then submit its proposed questionnaire and supporting materials to
EPA, which would forward them to OMB after a review for completeness and
compliance.  After receiving OMB approval through an expedited clearance
process, the assistance agreement recipient would distribute the
questionnaire and collect, tally, and report the responses to EPA.  Use
of the forms would not be required, but used at the discretion of the
responsible program and assistance agreement recipients. 



EPA is requesting a generic clearance for the following reasons:

(1)	EPA needs a generic clearance to help satisfy its performance
measurement and reporting responsibilities under the 1993 Government
Performance Results Act (GPRA) and OMB’s Program Assessment Rating
Tool.  Under these requirements, the Agency tracks and reports on the
performance of its programs, including its assistance agreements.  The
generic tools would assist EPA in the collection of standardized
performance measurement data that can be compared, aggregated, and
reported consistently across EPA’s assistance agreements and
environmental programs.  Without a generic clearance, it is doubtful
that EPA would be in a position to collect standardized measurement data
that can be aggregated and reported across its programs.

Refer to Section 2(a) for a further explanation of the need for the
reporting tools in this ICR.

(2)	This ICR sets forth a comprehensive plan for the collection of
information, and thereby provides full transparency to the public on how
the collections will be developed and carried out.  Briefly, the
reporting tools include the following information:

Reporting formats, generic questions, and detailed instructions for
assistance agreement recipients to prepare questionnaires in accordance
with OMB requirements and guidance.

Reporting form for assistance agreement recipients to tally and report
the collected data to EPA.

Detailed guidance for assistance agreement recipients to develop strong
data collection strategies and sample designs (e.g., recommended
procedures for maximizing survey response rates).

Detailed guidance for assistance agreement recipients to prepare the
supporting paperwork that will be submitted to OMB in the expedited
clearance request (e.g., a “Part B Document” that proposes a data
collection strategy).

Refer to the document, “Developing and Using Information Collection
Instruments for Assessing the Effectiveness of Training, Education, and
Outreach Activities under EPA’s Assistance Agreements,” for the
tools.  Because the tools are detailed and comprehensive, they will
ensure a high level of consistency in how each collection is
implemented.  For example, the tools spell out the generic questions to
be asked on a questionnaire and the approaches for collecting responses.
 Because of this transparency, the tools minimize the necessity to
provide public notice-and-comment for each proposed collection under
this ICR.

(3)	The reporting tools are designed to make OMB’s expedited review as
straightforward as possible.  Specifically, the tools are designed to
ensure that the questionnaires and collection approaches submitted under
the expedited review are designed in accordance with OMB requirements
and other best practices on a consistent basis.  For example, in
preparing their questionnaires, assistance agreement recipients will be
provided with generic questions and reporting formats to use.  Most
questions must be included on a form exactly as they are presented in
the tools, while the others allow for minor modifications only.  This
will ensure that each questionnaire submitted to OMB for expedited
approval includes only those questions and format that have been
approved by OMB in this ICR.  In addition, the detailed guidance on data
collection methods will ensure that the collection strategies are
designed in accordance with OMB’s guidelines and expectations for
statistically valid results.  Finally, EPA has developed an Agency
review process in which several EPA staff, including a Gatekeeper, will
review each proposed clearance request for compliance and completeness
before submittal to OMB.  Because of these factors, EPA is confident
that the likelihood of difficulties or surprises during the OMB
expedited review will be greatly minimized.

(4)	A primary purpose of the reporting tools is to measure the
satisfaction of individuals that received a training or
education/outreach.  OMB has indicated that generic clearances may be
appropriate for surveys that measure customer satisfaction of products
and services offered under government programs.  Individuals would be
asked to complete a questionnaire after receiving a training or
education/outreach (e.g., a brochure, video, or guidance document).  A
questionnaire would ask whether the individuals intend to follow, or
have followed, the recommendations offered by the training or
education/outreach.  It also may ask such questions as whether the
trainer was prepared, the training materials were acceptable, and the
training objectives were met.  These types of questions will let
assistance agreement recipients and EPA know whether the individuals
found the training or education/outreach to be satisfactory, beneficial,
and useful for their needs. 

(5)	The reporting tools do not raise any controversial or special
circumstances that would require particular justification and review. 
The questionnaires would include straightforward, generic questions.  In
addition, as stated above, the questionnaires and collection strategies
submitted for expedited approval would closely mirror the ones included
in this ICR.  Hence, they would not raise issues that are controversial,
difficult, or unfamiliar to OMB.

2.   	Need For/Use of the Collection

2(a) Need.  As stated above, EPA needs the reporting tools to help
satisfy its responsibilities under the 1993 Government Performance
Results Act (GPRA) and OMB’s Program Assessment Rating Tool (PART). 
The GPRA requires Federal agencies to develop goals and objectives,
measure their performance, and communicate information about their
performance to Congress and the public.  Under the PART, OMB and Federal
agencies assess and improve the agencies’ program performance to
achieve better results.  The PART reinforces the results-oriented
performance measurement framework developed under the GPRA.  EPA issued
Order 5700.7 to ensure that EPA’s assistance agreements are
results-oriented and are aligned with the Agency’s goals and
objectives.  

EPA needs the reporting tools to collect information to determine if
assistance agreement recipients achieve their short-term and
intermediate outcomes and further the mission and strategic goals of the
Agency, in accordance with the above obligations.  The tools also are
needed to identify deficiencies in the design and implementation of
trainings and education/outreach under assistance agreements, as well as
the corrective actions needed.  For example, the tools include questions
that ask training participants to rate their satisfaction with various
aspects of the training.  From a larger perspective, such performance
information will help EPA make better funding decisions and hold
assistance agreement recipients more accountable for the quality of
their products and services.  

2(b) Use.  EPA will use the information collected to determine if the
training, education, and outreach being performed under its assistance
agreements achieve their outcomes and outputs and satisfactorily further
the Agency’s mission and overall strategic goals.  EPA also will use
the information to provide stakeholders with reliable and uniform data
on the performance of its assistance agreements.  In addition, EPA will
use the information
潴椠敤瑮晩⁹敤楦楣湥楣獥愠摮渠捥獥慳祲挠牯敲瑣癩
⁥捡楴湯⁳湩琠敨琠慲湩湩Ⱨ攠畤慣楴湯‬湡⁤畯牴慥
档戠楥杮瀠牥潦浲摥‮䘠湩污祬‬偅⁁楷汬甠敳琠敨椠
普牯慭楴湯琠⁯癥污慵整椠獴映湵楤杮搠捥獩潩獮愠摮
瀠楲牯瑩敩⁳湡⁤慭敫戠瑥整⁲敤楣楳湯⁳湩琠敨映瑵
牵⹥

3.  	Description of Forms

The following identifies the information collection instruments under
this ICR.  Burden estimates are summarized in Exhibit A.

(1)	“Post-Training Questionnaire” Format.  This collection
instrument would be provided to training participants immediately at the
conclusion of a training course.  This instrument would measure
short-term outcomes.  Participants would be asked to provide feedback on
changes relating to their awareness, understanding, and anticipated
behavior as a result of the training.  The format includes a number of
required questions (i.e., questions that must be included on every
questionnaire), and optional questions that may be included at the
assistance agreement recipient’s discretion.  All required questions,
as well as many optional questions, use a 5-point agree/disagree
response scale.  However, three other types of responses are used:  1)
two-part answers that require selection of a follow-up response for
cases in which the respondent is not completely satisfied; 2) general
comments that request open-ended responses; and 3) anticipated behavior
questions that request selection of appropriate response options.  A
questionnaire will include not more than 30 questions.

(2)	“Follow-Up Questionnaire: Prospective” and “Follow-Up
Questionnaire: Retrospective” Formats.  The prospective format
explores an individual’s intended changes in behavior in response to
an outreach effort that has just taken place and before the individual
has had an opportunity to take action (e.g., survey of attendees at a
conference presentation).  The retrospective format explores an
individual’s past behavior in response to a training or outreach
activity that has taken place some time ago (e.g., 3-6 months ago). 
Each form includes one required question.  Several optional questions
also may be included.  Some of the questions are to be answered using a
5-point agree/disagree response scale. Two other types of responses also
are used: 1) multiple selection questions that permit respondents to
elaborate on the reasons for their actions (or anticipated actions); and
2) anticipated behavior questions that require selection of appropriate
response options. A questionnaire will include not more than five types
of questions.

(3)	“Assistance Agreement Recipient Reporting Form.”  Assistance
agreement recipients would use the Assistance Agreement Recipient
Reporting Form to report the collected responses to EPA.  The form has
been created in Microsoft Excel to facilitate the tallying of data. The
form consists of several spreadsheets with formulas that perform simple
calculations automatically.  A recipient would provide general
information in the form (e.g., name of person originating the form).  It
also would enter raw data from the questionnaires into the spreadsheets
and the data would be tallied.

4.  	Reporting and Recordkeeping Requirements

This ICR does not include any requirements to report or keep records. 
Use of the forms in this ICR would be at the discretion of the EPA
programs and assistance agreement recipients.  This ICR also does not
affect the existing reporting or recordkeeping requirements for
assistance agreements at 40 CFR Parts 30 and 31.  (EPA has accounted for
the burden of these existing requirements in its currently approved ICR,
“General Administrative Requirements for Assistance Programs” (ICR
No. 0938.11)).  

5.  	Respondents and Information Requested

5(a).  The primary recipients of EPA assistance agreements are State and
local governments, Indian Tribes, educational institutions, and
not-for-profit institutions. 

Prior to initiating a collection under this ICR, OMB approval must be
obtained through an expedited approval process.  EPA has developed tools
to assist assistance agreement recipients prepare an expedited clearance
request for submittal to EPA.  A request would include the following
information:

(1)	A cover letter or memorandum addressed to the appropriate EPA
Project Officer that includes the following information:

A brief overview of the assistance agreement (e.g., grant or cooperative
agreement name, objectives, general activities to be performed,
duration);

A description of the particular training and/or outreach activit(ies)
that will be evaluated (e.g., content, expected outcomes, method of
delivery of information, number/size of events);

Identification of the type of questionnaires to be used; and

A brief overview of the survey approach. 

(2)	A “Part B Document” that provides a description of the proposed
approach for collecting the information, including data analysis
methods, if any; 

(3)	An estimate of the total burden to the assistance agreement
recipient and its respondents; and 

(4)	The proposed questionnaire(s).  Burden will be defined on the
questionnaires.  They also will display the OMB control number and
clearance expiration date and inform respondents of its legal
significance (see 5 CFR 1320.5(b)).  In addition, every form will have
instructions clarifying its use.  

Once the EPA Project Officer receives the request, he or she would
review it for completeness and compliance.  The request would then be
forwarded to the EPA Gatekeeper in the Office of Grants and Debarment
who also would review the request.  The Gatekeeper also would be
available to answer questions from the programs, monitor the process
generally, and perform other necessary duties (e.g., tracking the
cumulative burden hours requested under this ICR).  The request would
then be submitted to EPA’s Office of Environmental Information, which
oversees and coordinates all Agency ICRs.  Following its review, the
request would be submitted to OMB for expedited review.  After OMB
approval, the recipient would distribute the questionnaire, compile the
responses, and report the responses to EPA using the Assistance
Agreement Recipient Reporting Form.

5(b).  EPA will encourage assistance agreement recipients to use
electronic methods whenever possible to simplify the reporting process
for their respondents.  This could include, for example, use of email,
fax, and/or web-based options.  However, some assistance agreement
recipients and the individuals they serve have limited technological
capacity.  In these cases, alternative methods will be offered to
enhance participation.

5(c).  See the reporting tools for specific data elements.

6. 	 The Information Collected—Agency Activities, Collection
Methodology, and Information Management

6(a).  The reporting forms have been reviewed thoroughly to identify and
eliminate duplication in reporting.  Similar information is simply not
available elsewhere.

6(b).  The information will be collected using recommended formats.  The
information collected will be reviewed and retained by EPA.

6(c).  The information collected will be reviewed in accordance with the
requirements of 40 CFR Parts 30 and 31.  

7.   	Nonduplication, Consultations, and other Collection Criteria

7(a).  There is no other source for this information.

7(b).  Administering EPA assistance agreements requires constant
recipient contact (e.g. written, telephone, face to face, etc.).  The
EPA Office of Grants and Debarment is both sensitive and responsive to
recipient concerns involving burden, duplication, availability of data,
clarity of instructions, etc.  The reporting tools have been subject to
close examination for burden as well as comment from participants in
pilot tests that EPA conducted.  They also have been closely reviewed by
an EPA working group that met periodically as the tools were being
developed.  Refer to Section 7(g) for additional information on the
pilot tests and EPA working group.

7(c).  No less frequent collection is possible in accordance with OMB
requirements.

7(d).  The information collection is consistent with 5 CFR 1320.5(d).

7(e).  To minimize the burden on small businesses, EPA has developed
clear reporting instructions, easy-to-use reporting templates, and other
resources (e.g., electronic spreadsheet tool) to help assistance
agreement recipients and their respondents compile and report the
information to EPA easily.  However, assistance agreement recipients
that prefer to submit data manually will be permitted to do so.

7(f).  No sensitive information is collected under this ICR, and
assistance agreement recipients are encouraged to make every effort to
protect respondent privacy.  However, assistance agreement recipients
are not asked to promise confidentiality since, under certain
circumstances, it is conceivable that respondents could be identified.

7(g).  Consultations - EPA conducted pilot test of the reporting tools
to identify needed improvements and collect burden estimates.  The
pilots lasted about two months and involved all of the reporting
formats.  The table below identifies the EPA assistance agreement
programs participating in the pilots, as well as the reporting formats
piloted.

Assistance agreement recipients volunteered for the pilots.  They were
provided with all of the questionnaire development materials, generic
questions, Assistance Agreement Recipient Reporting Form, and associated
instructions.  They were asked to read the instructions, prepare a
questionnaire, and distribute the questionnaire to a limited number of
respondents (e.g., 2 or 3 of their trainees).  

Each of the respondents completed a questionnaire.  They then completed
a Questionnaire Development Feedback Form.  The Questionnaire
Development Feedback Form collected their input on areas where the
collection instrument could be improved (how its questions could be
improved; what additional questions should be asked, etc.).  It also
collected estimates of their burden.

After the assistance agreement recipients compiled the completed
questionnaires, they completed the Assistance Agreement Recipient
Reporting Form.  They then completed a Grantee Reporting Form Feedback
Form.  The Grantee Reporting Form Feedback Form collected their input on
areas where the form could be improved.  It also collected estimates of
burden.

The assistance agreement recipients submitted all of this information to
EPA for evaluation under the pilots.  In general, their feedback focused
on the need to simplify, clarify and/or streamline the tools in specific
ways.  EPA addressed their suggestions, as appropriate.  In addition,
EPA incorporated their burden estimates into this ICR, as appropriate.

EPA ensured that no identical questions were asked of more than nine
persons across all of the pilots.  

Pilot Tests of Reporting Tools  

Name of Assistance Agreement Program in Pilot	Collection Instruments
Piloted

1. Tacoma-Pierce County Health Department, Training under the Community
for a Renewed Environment (CARE) Program	Post-Training Questionnaire and
Assistance Agreement Recipient Reporting Form

2. Outreach under SouthWest Area Neighborhood Association (SWAN) -
Healthy Homes Program	Follow-up Questionnaire and Assistance Agreement
Recipient Reporting Form

3. Cleveland Tenant’s Organization Training Seniors on Eliminating
Hazards in Homes	Follow-up Questionnaire and Assistance Agreement
Recipient Reporting Form

4. National Environmental Education Foundation	Follow-up Questionnaire
and Assistance Agreement Recipient Reporting Form

5. American Medical Association	Post-Training Questionnaire and
Assistance Agreement Recipient Reporting Form

In addition, EPA held several discussion sessions with a working group
consisting of EPA personnel representing the Agency’s major program
areas (e.g., air, water, waste) in developing the reporting tools.  EPA
held an upfront kickoff meeting to hear their initial feedback on
information needs.  As the tools were developed, additional sessions
were held to solicit their input.  A final discussion session was held
to get their final input and concurrence on the finalized tools.  This
process ensured that the tools meet the needs and expectations of the
EPA programs and that the information collected has a purpose and
practical utility.

8.   	Estimating the Burden and Cost of the Collection

8(a).  An estimation of the burden hours is included in Exhibit A.  EPA
estimates that, each year, the Post-Training Questionnaire will be
administered under 240 assistance agreements and that 90 individuals on
average will receive a questionnaire from each agreement.  EPA estimates
that, each year, the Follow-up Questionnaires (prospective or
retrospective) will be administered under 480 assistance agreements and
that 105 individuals on average will receive a questionnaire from each
agreement.  

Results from the questionnaires will be reported to EPA on the
Assistance Agreement Recipient Reporting Form.  EPA estimates that this
form will be submitted quarterly.

These assumptions are reflected in Exhibit A.

8(b).  RESPONDENTS: The total annual burden hours for respondents is
estimated to be 12,024.  This includes 4,320 hours for assistance
agreement recipients and 7,704 hours for the individuals who are asked
to complete a questionnaire (e.g., training participants).

Assistance agreement recipients.  EPA estimates that 80% of
recipients’ hours (3,456) will be spent by their secretarial/clerical
staff and the remaining 20% (864) by the recipients’ professional
staff.

Therefore:	Secretarial/Clerical Wage Rate:  $15.94 x 1.41 = $22.48.

Professional Wage Rate:  $39.22 x 1.41 = $55.30.

                              	

Thus: 	3,456 hours x $22.48 = $77,691 and 864 hours x $55.30 = $47,779.

Total: 	$125,470 (labor costs).

Individuals who complete a questionnaire (e.g., training participants). 
Assistance agreement recipients provide training and outreach/education
to a very broad range of entities.  This includes, for example,
individuals and organizations in the public sector (e.g., states,
Tribes, local governments) as well as the private sector (e.g.,
businesses, non-profit organizations, households).  Any of them could be
asked to complete a collection instrument under EPA’s reporting
approach.

	Therefore:	Wage Rate, All Workers:  $19.88 x 1.41 = $28.03.

	Thus: 		7,704 hours x $28.03 = $215,943.

	Total:		$215,943 (labor costs).

Total respondent costs.  Based on the above estimates, the total annual
respondent costs are estimated to be $341,413 (i.e., $125,470 +
$215,943).

8 (c)  FEDERAL:  The total annual burden hours for EPA is estimated to
be 1,800.

	8(d).  Bottom Line Burden Estimate.

          	Total burden hours for respondents = 12,024.

          	Total burden hours for EPA = 1,800.

8(e).  Without the data compiled by the use of the forms, it would be
difficult for EPA to evaluate the performance of the training,
education, and outreach activities performed under its assistance
agreements.  

9.   Start-up/Capital Costs and Operation and Maintenance Costs -
Start-up costs for the information collections in this ICR are minimal
and are part of customary and usual recipient business expenses.  There
are no capital costs related to these collections.  Therefore, total
start-up and capital costs are zero (0). There are no O&M costs
required.

10.  Special Circumstances for Change  –  Under no circumstances will
EPA add, delete or modify any of the questions or reporting formats in
this ICR without prior OMB approval.  

From time to time, EPA may review the list of “response options”
that are available to assistance agreement recipients, to ensure they
are relevant and responsive to their needs.  As indicated in Section 3
of this supporting statement, both post-training and follow-up
questionnaire formats offer the opportunity to include a generic
behavior question about the particular actions that respondents have
taken, or expect to take, as a result of the training or outreach
activity.  An example of the question is as follows:  “Because of
training, I plan to take the following actions that I would not have
taken otherwise.”  Following this generic question, a series of
response options would be listed.  Respondents would be asked to place a
check mark next to each response option that applies.  Refer to Section
5 of the instructions in the reporting tools for additional information.
 

EPA may expand the list of response options based on input from the EPA
programs, should the need arise, subsequent to OMB approval of this ICR.
 EPA notes that expanding the list of response options would not affect
the burden under this ICR, nor would it alter the generic question that
has been approved by OMB.  Specifically, it would not increase the
length or complexity of a questionnaire or the number, type or
complexity of questions asked on a questionnaire.  The only effect would
be to give assistance agreement recipients response options
better-tailored to their individual programs’ needs that could be
included on a questionnaire form, if deemed appropriate.  Finally, it is
important to note that OMB would have an opportunity to review the new
response options when they are submitted during the expedited review.

11.  Federal Register Announcement – A notice was published on page
39805 of the July 20, 2007 Federal Register, announcing that EPA was
beginning to prepare this ICR.  No comments were received in response to
this announcement.

12.  Burden Statement - The annual public reporting burden for this
collection of information is estimated to be six hours per assistance
agreement.  This includes time for preparing the questionnaire and
related materials for OMB approval, and collecting, summarizing, and
reporting the results to EPA.  The annual public reporting burden is
estimated to range from five to 10 minutes for an individual to complete
a questionnaire.  There is no public recordkeeping burden for this
information collection.  Burden means the total time, effort, or
financial resources expended by persons to generate, maintain, retain,
or disclose or provide information to or for a Federal agency.  This
includes the time needed to review instructions; develop, acquire,
install, and utilize technology and systems for the purposes of
collecting, validating, and verifying information, processing and
maintaining information, and disclosing and providing information;
adjust the existing ways to comply with any previously applicable
instructions and requirements; train personnel to be able to respond to
a collection of information; search data sources; complete and review
the collection of information; and transmit or otherwise disclose the
information.   

	

To comment on the Agency’s need for this information, the accuracy of
the provided burden estimates, and any suggested methods for minimizing
respondent burden, including the use of automated collection techniques,
EPA has established a public docket for this ICR under Docket ID No.
EPA-HQ-OA-2007-0607, which is available for public viewing at the Office
of Environmental Information Docket in the EPA Docket Center (EPA/DC),
EPA West, Room 3334, 1301 Constitution Ave., NW, Washington, DC.  The
EPA Docket Center Public Reading Room is open from 8:30 a.m. to 4:30
p.m., Monday through Friday, excluding legal holidays.  The telephone
number for the Reading Room is (202) 566-1744, and the telephone number
for the Office of Environmental Information Docket is (202) 566-1752. 
An electronic version of the public docket is available online at
www.regulations.gov.  Use www.regulations.gov to submit or view public
comments, access the index listing of the contents of the public docket,
and to access those documents in the public docket that are available
electronically.  Once in the system, select “search,” then key in
the docket ID number identified above.  Also, you can send comments to
the Office of Information and Regulatory Affairs, Office of Management
and Budget, 725 17th Street, NW, Washington, DC 20503, Attention:  Desk
Office for EPA.  Please include the Docket ID No. EPA-HQ-OA-2007-0607.

PART B OF THE SUPPORTING STATEMENT

Each expedited clearance request submitted to OMB will include a Part B
Document that provides a description of the proposed approach for
collecting information, including data analysis methods, if any.  To
complete the document, assistance agreement recipients would refer to
the guidance in the reporting tools, including Attachment A (guidance on
collection strategies, including sample design) and Attachment B
(guidance on preparing the Part B Document).  These guidances suggest
best practices and recommendations for the design and implementation of
a successful survey.  The guidances offer user-friendly reference
materials and approaches on, for example, how to determine a minimum
acceptable sample size, maximize response rates, and obtain
representative responses, so that statistically valid results are
obtained.  Examples of acceptable approaches are also provided, to
minimize the burden imposed on assistance agreement recipients and
ensure that they understand the level of detail needed to obtain OMB
approval.  Any questions that arise can be directed to the EPA Project
Officer and/or the Gatekeeper for assistance.

The length and complexity of the Part B Document will depend on the
proposed survey approach.  If a sampling approach will be used instead
of a census, for example, more information is generally required in
order to explain how the representativeness of the sample will be
ensured.  If a particular section of the Part B Document is not
applicable to a particular collection, this will be indicated briefly.

1.	Survey Objectives, Key Variables, And Other Preliminaries

(1)	Survey Objectives

The Part B Document will describe the survey objectives.  The objectives
will be stated in the form of issues or questions for resolution. In
general, survey objectives will be limited to a reasonable number, based
on the assistance agreement recipient’s and EPA’s information needs.
 

(2)	Key Variables

The Part B Document will briefly describe the characteristics of the
individuals being surveyed that could affect the way in which they
respond to the survey questions.  The description will try to give the
reviewer a sense of how these variables might influence the survey
results.  However, if the key variables are not known, this will be
explained briefly (e.g., “insufficient information exists on
population characteristics”).  



(3)	Sampling Approach

This section will state whether a census or sampling approach will be
used. If a sampling approach is proposed, details will be presented in
accordance with the procedures laid out below.

(4)	Feasibility

The feasibility of the proposed survey effort will be described briefly.
 This entails a brief discussion of likely problems and how they will be
addressed.

2. 	Survey Design

This section will describe the mechanics of the survey design – from
how the respondent pool is selected to what type of questions will be
asked in the survey.  If a sampling approach is being proposed, all four
information parameters below will be addressed.  If a census is
proposed, only the last parameter will be addressed.

(1)	Target Population and Coverage

This section will briefly describe the overall group of individuals or
organizations that are the focus of the survey (i.e., the “target
population”).  This section also will briefly describe if any key
subgroups within the target population will not be surveyed.  

(2)	Sample Design

The following issues will be addressed with regard to the sampling
design if a census approach will not be used:

Indication of the sampling frame.  Guidance is provided in the tools on
sample frame selection.

Indication of the sample size, including how it was derived.  Guidance
is provided in the tools on determining an appropriate sample size for a
simple analysis of the number of respondents scoring at or above a
pre-determined benchmark level.  A user-friendly reference table is
included that identifies the appropriate sample size range for this type
of analysis based on the total population size, eliminating the need for
calculations. Assistance agreement recipients that wish to conduct more
sophisticated analyses would need to establish appropriate sample sizes
accordingly.  

Brief description of the sampling method.  The generally recommended
approach is a random, or probability, sampling.  However, in some
situations a more complex method (e.g. stratified random sampling) may
be appropriate. This section will discuss why the selected method is
being proposed, including an explanation why the proposed approach will
yield results that are representative of the population sampled and will
meet the objectives of the survey.

Brief description of multi-stage sampling, if applicable.  It is
doubtful that most recipients will use multi-stage sampling for the
questionnaires.  If multi-state sampling will not be used, this will be
indicated briefly.  

Brief description of stratification variables, if applicable, including
how they were selected and how they will improve the survey estimates. 
It is doubtful that most recipients will use stratification for the
questionnaires in this guidance.  If stratification will not be used,
this will be indicated briefly.  

(3)	Precision Requirements

This section will describe the precision targets and sampling error.  It
also will discuss the potential sources of nonsampling error, the steps
to be taken to minimize its impact on sample estimates, and what effect
it may have on the value of the survey as a decision-making tool.  

(4)	Description of Questionnaire

The questions included in the questionnaire will be described briefly,
including a general indication that they are expected to achieve the
survey objectives.  In addition, the steps taken to ensure the
reliability and usefulness of the design will be described.  

3.	Pretests And Pilot Tests

This section will include the following boilerplate text:

“EPA conducted a pilot test of the generic questions and questionnaire
formats to identify and resolve potential problems in regard to
effectiveness, clarity, feasibility, and burden/usability.  The pilot
test lasted about two months and all of the questionnaire formats were
tested, as well as the Assistance Agreement Recipient Reporting Form. 
EPA solicited feedback from assistance agreement recipients as well as
those individuals who completed a questionnaire (e.g., training
participants), and addressed their feedback in finalizing the forms. 
See Section 7(g) in the supporting statement of ICR 2255.01,
“Performance Measurement Reporting for Training and
Education/Outreach,” for additional information on the pilot tests. 
In no case were identical questions asked of more than nine persons.”

4.	Collection Methods And Follow-up

This section will describe methods to be used for collecting data,
including any follow-up activities.  

(1)	Collection Methods

This section will identify which collection method(s) have been selected
and why.  Guidance is provided in the tools on data collection methods.

(2)	Survey Response And Follow-up

This section will state the target response rate and describe how the
actual response rate will be measured.   Guidance is provided in the
tools for assistance agreement recipients to maximize response rates,
including use of the Dillman method for following up on surveys that
have not been submitted, offering incentives, and offering anonymity to
respondents if possible.

  

5.	Analyzing And Reporting Survey Results

Proposed data entry procedures and quality control measures will be
described. This will include a description of the procedures to be used
for accurately counting the data and entering it on the data reporting
forms provided by EPA.  If additional analysis will be conducted, these
will be described.

EXHIBIT A

BURDEN HOURS FOR REPORTING ACTIVITIES*

*Notes:	Column D = A x B x C

Column F = A x B x E

EXHIBIT B

Forms covered in this ICR include:

“Post-Training Questionnaire” Format

“Follow-Up Questionnaire: Prospective” Format

“Follow up Questionnaire: Retrospective” Format

“Assistance Agreement Recipient Reporting Form”



 Although this ICR describes the process for assistance agreement
recipients to develop the clearance request, it is possible that an EPA
program may choose to develop a clearance request for a questionnaire
that it has developed on behalf of its assistance agreement recipients. 
The information submitted in an EPA clearance request would be the same
as that submitted by assistance agreement recipients in their requests.

  Bureau of Labor Statistics; May 2006 National Occupational Employment
and Wage Estimates.  Available at:    HYPERLINK
"http://www.bls.gov/oes/current/oes_stru.htm" 
http://www.bls.gov/oes/current/oes_stru.htm .  These rates were updated
to current levels using an Employment Cost Index from the Bureau of
Labor Statistics, "Table 5a. Employment Cost Index (Wages and Salaries
Only), Civilian Workers by Occupation and Industry Group (Not seasonally
adjusted);" Employment Cost Index Historical. 
http://www.bls.gov/news.release/eci.t05.htm.		

	

 Bureau of Labor Statistics; "Table 1. Civilian workers, by major
occupation group - management, professional, and related;" Employer
Costs for Employee Compensation - September 2007.  Available at: 
http://www.bls.gov/news.release/pdf/ecec.pdf.		

			

萏ː葞ː

愀Ĥ

urrent/oes_stru.htm .  These rates were updated to current levels using
an Employment Cost Index from the Bureau of Labor Statistics, "Table 5a.
Employment Cost Index (Wages and Salaries Only), Civilian Workers by
Occupation and Industry Group (Not seasonally adjusted);" Employment
Cost Index Historical.  http://www.bls.gov/news.release/eci.t05.htm.

 Bureau of Labor Statistics; "Table 1. Civilian workers, by major
occupation group - all workers," Employer Costs for Employee
Compensation - September 2007.  Available at:    HYPERLINK
"http://www.bls.gov/news.release/pdf/ecec.pdf" 
http://www.bls.gov/news.release/pdf/ecec.pdf .

 Office of Personnel Management; 2008 General Schedule (GS) Base (Hourly
Rate) for GS 11, Step 9 ($29.22).  Available at:    HYPERLINK
"http://www.opm.gov/oca/08tables/pdf/gs_h.pdf" 
http://www.opm.gov/oca/08tables/pdf/gs_h.pdf .  This was also fully
burdened (x 40%), yielding a wage rate of $40.91.

  PAGE   1 

                                                                        
 

1

                                                                 A -    

    

1

                                                                 B -