Document ID: EPA-HQ-OECA-2009-0124-0005
Agency: epa
Document Type: Supporting & Related Material
Title: 
Posted Date: 2009-07-06T04:00Z

ICR SUPPORTING STATEMENT:  Part B

Auto Body Compliance Assessment Pilot Project 

Part B applies only to the Pilot project; methodologies described here
will not apply to surveys used for anecdotal purposes in support of the
CRC activities.

Survey Objectives, Key Variables, and Other Preliminaries

Survey Objectives

The purpose of the pilot study is to evaluate whether and how EPA’s
specific compliance assistance activities in Region 1 helped owners or
operators of auto body shops improve their operations with respect to
Subpart HHHHHH National Emission Standards for Hazardous Air Pollutants:
Paint Stripping and Miscellaneous Surface Coating Operations at Area
Sources (NESHAP Subpart HHHHHH, hereinafter referred to as the Surface
Coating Rule.).  Specifically, the study has been designed to assess
whether or not the assistance provided helped owners or operators
understand these and other applicable environmental regulations, and
whether or not the assistance helped owners or operators implement
operational changes that improved environmental practices; including
changes in behavior that resulted in the auto body shop either returning
to compliance with regulations or taking steps toward achieving
compliance.  

The questions contained in the surveys are designed to obtain
information on (1) environmental performance related to current
hazardous waste management and training requirements under RCRA, (2)
environmental performance related to air emissions control requirements
associated with the recently promulgated Surface Coating Rule, (3)
environmental compliance assistance received by government agencies or
other entities, and (4) perceptions regarding the factors that influence
shop behaviors related to environmental performance.  Some of these
behaviors are related to current regulatory requirements, while others
will be required in the future or are purely voluntary.   Survey-based
measurements related to these behaviors will be referred to in this
document as measures of “environmental performance.”  In addition,
the survey will collect information on the type of compliance assistance
received, i.e., the extent to which the shop has been affected by
outreach, site visits, or other types of assistance from EPA or others. 
  

There are two objectives in collecting this information:

The primary objective is to assess the degree to which EPA’s
compliance assistance activities lead to, or are associated with,
improvements in environmental performance at auto body shops.

A secondary objective of the survey is to assess the validity of
information about environmental performance data collected through
telephone surveys.  EPA typically uses telephone surveys to gather
information about environmental performance; however OMB has expressed
concerns that such data is subject to potential self-reporting and
non-response bias.  The current survey will involve on-site data
collection designed to verify telephone responses and evaluate potential
non-responses bias.  

Key Variables:

The key variables in the survey involve shops’ environmental
performance related to employee training, air pollutants (specifically
addressing requirements of the Surface Coating Rule), RCRA generator and
waste determination requirements, emergency procedures, and the type of
compliance assurance received by shops.  

Key variables related to employee training are as follows:

Whether the shop has a training program focused on proper hazardous
waste management procedures, and whether these training records are
available.

Whether spray technicians have been properly trained, and whether these
training records are available.

Key variables related to air emissions are as follows:

Type of spray guns used to apply coatings.

Manufacturer of spray guns.

Configuration of area where spray coatings are applied (e.g., whether
spray coatings are applied in an enclosed, ventilated spray booth, and
how long this booth has been in operation).  

Capture efficiency of booth exhaust system for area where spray coatings
are applied.

Whether particle filter is used on spray booth exhaust and condition of
filter.  

Configuration of paint mixing room (e.g., whether mixing rooms are
enclosed and ventilated, and how long they have been in operation).

Configuration of prep station for spray coating (e.g., whether prep
stations are enclosed and ventilated, and how long they have been in
operation).

Methods used for cleaning spray guns, and how long these methods have
been in use.

Whether the shop uses waterborne paints, and for what purposes.

Amounts of different types of coatings applied in the shop each week.

Primary supplier of paint coating products.

Whether shop has MSDS and coating formulation data for solvents and
coatings, and whether this documentation is available.

Whether coatings used by the shop contain chromium, lead, cadmium,
nickel, or manganese.

Whether paint stripping products used by the shop contain methylene
chloride, and whether shop has minimization plan for methylene chloride.

Whether the respondent is aware of the Surface Coating Rule, and if so,
when and how the shop first heard of these requirements.

Key variables related to RCRA requirements are as follows:  

How much hazardous waste the shop generates.

Whether all hazardous waste streams have been identified.

Whether hazardous wastes are properly labeled and stored in an
appropriate manner.

Whether hazardous wastes are shipped to an appropriate off-site
facility.

Key variables related to emergency procedures are as follows:  

Whether the shop has implemented emergency procedures.

Whether documentation related to emergency procedures is available.

	Key variables related to the type of compliance assurance are as
follows:

Sources of information about how to comply with federal and state
environmental regulations (e.g., trade associations, paint
manufacturers, regulatory agencies, etc.).

Whether shop has been visited by a government environmental official
within the last six months, and type of government official that
visited.

Whether shop has made any changes in its operations in the last year,
and if so, what changed and why.

Sampling Approach:

The survey will involve a probability sample of auto body shops.  As
described in detail in Section B.2, the probability sample will be taken
within the context of a study designed to evaluate the impact of
compliance assistance on environmental performance. The Agency chose not
to conduct a census due to the prohibitive costs and the unnecessary
burden that would be imposed on auto body shops.  The Agency has
concluded that its information needs can be met through a probability
sample and that the sampling error will not compromise the objectives of
the survey.  Section B.2.c provides additional information about
precision.   

The survey is being designed and conducted with the assistance of a
contractor:

Contractor	Contractor Roles

Industrial Economics, Incorporated

2067 Massachusetts Avenue

Cambridge, MA 02140	Study design

Questionnaire development

Statistical sample design

Random assignment

Database design

Feasibility:

The Agency believes it has developed a strong study design that will
meet the objectives described above.  The survey and overall study
design is the result of a two-year collaborative effort involving an
Agency inter-office workgroup; consultants with expertise in program
evaluation, survey design, and statistics; and independent peer
reviewers.  The Agency has allocated sufficient funds to complete the
survey effort, and the data will be available within a timeframe that is
acceptable to users within the Agency.  

The risk of failure has been minimized through the collection of
multiple types of data.  As discussed in the next section, the study
involves two different approaches to policy evaluation (random
assignment experiment and comparison group design) and two different
data collection modes (telephone and on-site surveys).  Furthermore, the
survey will gather data on many different measures of environmental
performance.    

The data collection approach has been designed to minimize the impact of
non-response and self-reporting bias.  In order to address these
potential sources of bias in the telephone survey, the data collection
effort incorporates (1) on-site follow-ups with telephone respondents
and non-respondents and (2) on-site verification of self-reported
environmental performance.  The Agency expects minimal non-response bias
in the on-site component of the survey effort, as past site visit
cooperation rates have been near one hundred percent. 

The Agency does foresee a potential risk related to the type and
quantity of compliance assistance received by the comparison group in
the long-term component of the study.  As we discuss below, our approach
to assessing the long-term impact of compliance assistance requires a
comparison group that receives limited, if any, compliance assistance
from the Agency.  However, the Agency cannot control compliance
assistance provided by states, local governments, private vendors, or
trade associations.  If these groups replace EPA compliance assistance
with substantial compliance assistance efforts of their own, the
long-term study may be compromised.  

Survey Design

This section contains a detailed description of the survey design,
including a description of the target population, sampling frame, sample
size, stratification variables, sampling methods, precision targets,
non-sampling error, and questionnaire design.  The section begins with
an overview of the study design, as the survey is best viewed within the
overall context of the policy evaluation.  The schedule for the various
components of the study is depicted graphically in Appendix A.

Short-Term Experiment: Impact of Compliance Assistance Outreach and
Workshops

The short-term impact of compliance assistance outreach and workshops
will be evaluated through a random-assignment experiment involving auto
body shops in risk-based clusters of Massachusetts communities.  The
auto body shops will be randomly assigned to either a treatment group
(hereafter, Group A) or a control group (hereafter, Group B) (Exhibit
B-1).  The shops in the treatment group will receive (1) a multimedia
guidebook providing a summary of air, water, and RCRA requirements
impacting auto body shops in Massachusetts, (2) a brochure summarizing
the Surface Coating Rule requirements, (3) an invitation to attend
workshops covering the requirements of the Surface Coating Rule, and (4)
a copy of the presentation slides that will be used at the workshops. 
EPA anticipates that a total of approximately 50-100 shops will
participate in the workshops.  The shops in the comparison group will
not receive the mailings, and they will not be invited to the workshops.

Shortly after the workshops have been completed, EPA personnel will
assess environmental performance through a combination of telephone
interviews and on-site visits with probability samples of shops from
each of the two groups.  The impact of compliance assistance will be
assessed by comparing the estimated environmental performance for Group
A to the estimated environmental performance for Group B. 

Exhibit B-1:  Assessing Short-Term Impact of Compliance Assistance
Outreach and Workshops

Assessing the Validity of Telephone Surveys

The validity of telephone survey responses will be evaluated by using a
two-phase sampling approach to assess environmental performance in the
short-term experiment (Cochran, 1977, pp. 327-355) (Exhibit B-2).  In
the first phase, EPA will measure performance through a telephone survey
with a sample of shops.  In the second phase, site visits will be
conducted with samples of telephone survey respondents and
nonrespondents.  The site visits will determine shop environmental
performance through direct observation and a review of on-site records. 
Site visits at respondent shops will allow us to adjust for any
potential self-reporting bias in the telephone survey, while site visits
at non-respondent shops will allow us to adjust for any potential
non-response bias.  By comparing performance levels estimated using data
from this two-phase approach (combining data from the telephone survey
with data from follow-up site visits) with performance levels estimated
using data from the telephone survey alone (excluding the site visit
data), the bias resulting from relying entirely on telephone survey
responses can be determined.  

In addition, we anticipate that the supplemental information obtained
through the telephone survey will improve the precision of our
performance estimate relative to estimates derived from site visits
alone.  If the performance estimates from this two-phase sampling
approach provide substantial precision gains relative to the performance
estimates obtained from site visits alone, the Agency will consider
using the two-phase approach in the 2011 component of the long-term
study. 

Exhibit B-2:  Schematic for Assessing the Validity of Telephone Surveys

Long-Term Study: Impact of Compliance Assistance Package

The long-term impact of a more comprehensive compliance assistance
package will be evaluated through a comparison group design involving
auto body shops in risk-based clusters of Massachusetts communities
(hereafter, Group C) and a similar population of auto body shops in a
comparison location (hereafter, Group X).  By “long-term” impact, we
are referring to the cumulative impacts of a package of compliance
assistance activities provided over approximately two years.  The
post-treatment measurement for assessing long-term impacts will occur at
a point in time when the Surface Coating Rule requirements are in
effect.  

The shops in Group C will be offered a full suite of compliance
assistance activities related to RCRA and surface coating requirements,
including outreach, workshops, and site visits: 

Outreach: All shops will receive a multimedia guidebook providing a
summary of air, water, and RCRA requirements impacting auto body shops
in Massachusetts; a brochure summarizing the Surface Coating Rule
requirements; and a copy of the presentation slides that will be used at
the workshops.

Workshops: All shops will be offered an opportunity to participate in
compliance assistance workshops led by EPA personnel. 

Site Visits: Eighty randomly selected shops will be offered compliance
assistance through site visits by EPA personnel in 2009.  The purpose of
these 2009 site visits is two-fold: they will be used to assess
environmental performance and, after the on-site interview has been
completed, they will be used to deliver compliance assistance based on
interviewer observations.  An additional 60 shops will receive
compliance assistance visits in 2010; most will be randomly selected,
but some shops may request a site visit after hearing about workshops or
site visits at neighboring shops. 

The shops in Group X will receive limited, if any, compliance assistance
from EPA or state environmental assistance programs regarding RCRA
requirements or the Surface Coating Rule.  

In each of the two locations, site visits by qualified personnel at
random samples of shops will be used to estimate performance before and
after compliance assistance has been provided.  This will result in four
separate estimates of performance:  

Group C 2009 Pre-treatment

Group C 2011 Post-treatment

Group X 2009 

Group X 2011 

The impact of compliance assistance will be assessed by comparing the
change in performance in Group C with the change in performance in Group
X.  (See Exhibit B-3)  (i.e., through a “difference-in-differences”
approach, see Bloom, 2005, pg. 179).

Exhibit B-3:  Schematic for Assessing Long-Term Impact of Compliance
Assistance Package

Target Population and Coverage:

The population of interest for the pilot study is auto body shops
located in risk-based clusters of communities in Massachusetts and Group
X that are subject to the recently promulgated Surface Coating Rule. 
Auto body shops are subject to the Surface Coating Rule if they are paid
to complete at least two surface coatings each year on vehicles or
pieces of mobile equipment.  

Risk-based clusters are characterized as groups of geographically
contiguous communities, each centered around a core urban area, that
have elevated cancer and non-cancer risks from air pollution according
to National Air Toxics Assessment (NATA) data.  The Agency has chosen to
focus on risk-based clusters of communities in this study because we
believe that the need for auto body compliance assistance may be greater
in these communities, given elevated risks to residents from other
sources. 

The Agency is currently considering two potential locations for Group X:
Oklahoma and the Tidewater area of Virginia.  The characteristics of
these locations are being evaluated, and the comparison location will be
selected prior to beginning the study.  The evaluation criteria used in
selecting Group X include:  

number of shops located in risk-based clusters;

characteristics of risk-based clusters compared to risk-based clusters
in Massachusetts;

state regulations related to RCRA and air emissions; and

expected compliance assistance from EPA, state, or other entities.

At the conclusion of the study, EPA will use additional data gathered in
the survey to assess the adequacy of Group X as a comparison group and
to identify any compliance assistance received by Group X shops.    

Sample Design:

2 (b) (i) Sampling Frame

The sampling frame for the survey is auto body shops located in
risk-based clusters in Massachusetts and Oklahoma/Virginia.  Auto body
shops are defined as businesses with Standard Industrial Classification
(SIC) 7532 – “Automotive Body, Paint, and Interior Repair and
Maintenance” – that are listed in either Dunn & Bradstreet or
Reference USA.  SIC 7532 comprises establishments primarily engaged in
repairing or customizing automotive vehicles, such as passenger cars,
trucks, and vans, and all trailer bodies and interiors; and/or painting
automotive vehicles and trailer bodies.  The risk-based clusters
centered in Worcester, MA, Lawrence, MA and Boston, MA were excluded
from the sampling frame because for each of these communities aggressive
assistance outreach and or enforcement activities had been conducted for
a number of years (Lawrence and Boston) or being planned (Worcester) at
the local or federal levels. 

2 (b) (ii) Sample Size

The proposed sample sizes for the study are provided in Table B-1.  In
the short-term experiment we will select 400 shops for the telephone
survey and 80 shops for the on-site survey.  In the long-term study, we
will select an additional 160 shops for the on-site survey: 40 shops in
2009 and 120 shops in 2011.  Sample sizes were determined by weighing
expected costs against expected precision gains from larger samples (see
Section B.2.c.i).  In addition, an effort was made to balance total
on-site survey effort between the two survey years.  If resources allow,
the agency plans to double the sample sizes for the on-site survey.

The Agency is planning to conduct an interim telephone survey with 200
Group C shops in 2010 to gauge the progress of auto body shops toward
achieving compliance with the new Surface Coating Rule requirements. 
This interim telephone survey will help the Agency determine whether
additional outreach activities (e.g., additional mailings, workshops, or
site visits) may be needed prior to the 2011 compliance date.    

Although the Agency currently does not plan to conduct telephone surveys
in 2011, telephone surveys will be added in 2011 if the two-phase
sampling approach substantially increases the precision of the
environmental performance estimates substantially without adding undo
complexity to the study.  If telephone surveys are conducted in 2011,
the sample sizes will be 200 in each state, for a total of 400
additional shops.  The Agency does not propose to conduct a phone survey
in Group X in 2009 due to potential measurement effects (i.e., the phone
survey itself could act as a form of compliance assistance, which would
bias results from this comparison group at the baseline measurement.)

Table B-1: Summary of Sample Sizes

Shops Sampled	Sample Size for Telephone Survey	Sample Size for On-Site
Survey

Group A 2009	200	40

Group B 2009 a	200	40

Group X 2009	0	40

Group C 2010	100	0

Group C 2011	0b	60

Group X 2011	0b	60

Notes:

a Group B serves two purposes: it is the control group for the
short-term experiment, and it provides the pre-treatment estimate of
Group C environmental performance for the long-term experiment.

b As discussed in the text, telephone surveys may be conducted in 2011
if the two-phase sampling approach is successful in 2009.

2 (b) (iii) Stratification Variables

Sampling for the telephone and site-visit surveys in Groups A and C will
be stratified (using proportional allocation) by receipt of face-to-face
compliance assistance.  The face-to-face compliance assistance stratum
is defined as all shops that either attended an EPA workshop or received
an EPA compliance assistance site visit as part of this effort.  This
stratification will ensure that both types of shops (i.e., those that
receive face-to-face assistance and those that do not) will be included
in the sample.  The size of the strata will be unknown until compliance
assistance has been provided.  

In addition, the two-phase sampling approach used in Groups A and B will
involve stratification in the second sampling phase.  That is, the shops
sampled for the telephone survey will be stratified into telephone
respondents and telephone non-respondents for the on-site survey.   This
stratification into respondents and non-respondents will ensure that
shops from both groups will be sampled for the on-site survey, ensuring
that the Agency will be able to assess potential self-reporting bias and
potential non-response bias in the telephone survey.   Once again, the
Agency expects to use proportional stratification in allocating the
sample among the strata. (Exhibit B-4 illustrates the implementation of
both stratification approaches within Group A.)  

Exhibit B-4:  Stratified Sampling Approach for Group A 

(Includes all Shops Invited to Attend Workshops)

2 (b) (iv) Sampling Method

The sampling method for Group X in 2009 and 2011 will be simple random
sampling from the population of shops.  In order to minimize potential
measurement effects associated with the 2009 interviews, the 2009 sample
will be excluded from the sampling frame prior to drawing the sample for
2011.  

The sampling method for Group C in 2011 will be random sampling
stratified by receipt of face-to-face compliance assistance, with simple
random sampling used within each stratum.  

The sampling method for Groups A and B in 2009 will be somewhat more
complex.  In Group A, the survey will be completed in two phases (see
Exhibit B-4).  In the first phase, a stratified random sample of 200
shops will be selected for the telephone survey.  There will be two
strata for the telephone survey: shops that attended a workshop and
shops that did not.  In the second phase, site visits will be conducted
at a stratified random sample of 40 shops selected for the telephone
survey.  Thus, the second phase will involve four distinct strata,
defined by workshop attendance/non-attendance and response/non-response
to the telephone survey.   

In Group B, the survey will also be completed in two phases.  However,
as none of the Group B shops will have attended workshops prior to the
telephone survey, a simple random sample of 200 shops will be selected
for the telephone survey in Group B.  Site visits will then be conducted
at a stratified random sample of 40 of the shops selected for the
telephone survey, with shops stratified by response/non-response to the
telephone survey.      

2 (b) (v.) Multi-Stage Sampling

Multi-stage sampling will not be used in the study.

Precision Requirements:

2 (c) (i) Precision Targets

To satisfy EPA’s decision-making needs, the study has been designed to
provide a minimum detectable effect (MDE) not exceeding 29 percentage
points for binary measures of shop performance for the long-term
evaluation and a MDE not exceeding 22 percentage points for the
short-term evaluation.  The MDEs are calculated using a 10 percent
significance level, a one-sided hypothesis test, a power level of 80
percent, assumed population sizes of 600 in each group, and an
assumption of no improvements in precision through the use of
regressors.   In addition, MDE depends on shop environmental
performance, telephone survey response rate, and the degree of agreement
between the telephone and on-site measurements.  The Agency believes
that the telephone survey response rate will likely be between 20% and
40%, while the degree of agreement between telephone and on-site
measurements will likely be between 80% and 95%.  As the performance
rate will vary across the various measures of performance, we use 50%,
70%, and 90% in our calculations.  Under these assumptions, the MDE for
the short-term experiment ranges from 13% to 22%, while the MDE for the
long-term comparison ranges from 17% to 29% (Tables B-2 and B-3).

Table B-2: Minimum Detectable Effect for Short-Term Experiment

Agreement between Telephone and On-Site Measurement	Performance = 50%
Performance = 70%	Performance = 90%

	RR =  20%	RR =  30%	RR =  40%	RR =  20%	RR =  30%	RR =  40%	RR =  20%
RR =  30%	RR =  40%

80%	22.4%	22.0%	21.7%	20.8%	20.6%	20.4%	n/a	n/a	n/a

85%	22.1	21.7	21.2	20.5	20.2	19.8	n/a	n/a	n/a

90%	21.8	21.2	20.5	20.2	19.6	19.1	13.9%	13.9%	13.9%

95%	21.5	20.7	19.8	19.8	19.1	18.3	13.3	13.0	12.8%

Notes: 

RR = telephone survey response rate

Performance = environmental performance rate

Table B-3: Minimum Detectable Effect for Long-Term Comparison

Agreement between Telephone and On-Site Measurement	Performance = 50%
Performance = 70%	Performance = 90%

	RR =  20%	RR =  30%	RR =  40%	RR =  20%	RR =  30%	RR =  40%	RR =  20%
RR =  30%	RR =  40%

80%	29.2%	29.1%	28.9%	26.8%	26.8%	26.7%	n/a	n/a	n/a

85%	29.1	28.9	28.7	26.7	26.6	26.5	n/a	n/a	n/a

90%	29.0	28.7	28.5	26.6	26.4	26.2	17.7%	17.7%	17.7%

95%	28.9	28.5	28.2	26.5	26.2	25.9	17.5	17.4	17.3%

Notes: 

RR = telephone survey response rate

Performance = environmental performance rate

The Agency is currently exploring the possibility of doubling the
on-site sample sizes for the study.  The MDEs associated with doubling
on-site sample sizes are presented in Tables B-4 and B-5.  The MDE for
the short-term experiment ranges from 9% to 15%, while the MDE for the
long-term comparison ranges from 12% to 20%.

Table B-4: Minimum Detectable Effect for 

Short-Term Comparison with On-Site Sample Sizes Doubled 

Agreement between Telephone and On-Site Measurement	Performance = 50%
Performance = 70%	Performance = 90%

	RR =  20%	RR =  30%	RR =  40%	RR =  20%	RR =  30%	RR =  40%	RR =  20%
RR =  30%	RR =  40%

80%	15.4%	15.3%	15.1%	14.3%	14.2%	14.1%	n/a	n/a	n/a

85%	15.3	15.1	14.8	14.1	14.0	13.8	n/a	n/a	n/a

90%	15.1	14.8	14.5	13.9	13.7	13.4	9.5%	9.5%	9.5%

95%	14.9	14.5	14.1	13.7	13.4	13.0	9.2	9.1	8.9

Notes: 

RR = telephone survey response rate

Performance = environmental performance rate

Table B-5: Minimum Detectable Effect for 

Long-Term Comparison with On-Site Sample Sizes Doubled

Agreement between Telephone and On-Site Measurement	Performance = 50%
Performance = 70%	Performance = 90%

	RR =  20%	RR =  30%	RR =  40%	RR =  20%	RR =  30%	RR =  40%	RR =  20%
RR =  30%	RR =  40%

80%	19.8%	19.7%	19.6%	18.2%	18.1%	18.1%	n/a	n/a	n/a

85%	19.7	19.6	19.5	18.1	18.0	18.0	n/a	n/a	n/a

90%	19.6	19.5	19.4	18.0	17.9	17.8	11.9%	12.0%	12.0%

95%	19.6	19.4	19.3	17.9	17.8	17.7	11.8	11.8	11.7

Notes: 

RR = telephone survey response rate

Performance = environmental performance rate

 

2 (c) (ii) Non-Sampling Error

The Agency will use several quality assurance techniques to maximize
response rates, response accuracy, and processing accuracy to minimize
nonsampling error:

The study relies heavily on data collected through on-site surveys, with
many of the key measurements being obtained directly through interviewer
observations rather than through survey questions.  This will minimize
measurement error, eliminate self-reporting bias, and reduce
non-response bias.

To maximize response in on-site surveys, selected shops will be visited
during normal business hours, and shops will be able to reschedule the
interview if the interviewer arrives at an inconvenient time.  In the
past, EPA has obtained response rates near 100% during these type of
facility visits.

To maximize response in telephone surveys, initial contacts will be
attempted shortly after shops open for business, as personnel are
typically busiest from mid-morning to late afternoon.  If the shop
indicates that the time is not convenient, the interview will be
rescheduled.  If the shop does not answer the telephone, at least three
additional callbacks will be attempted on different days and at
different times.  

When telephone surveys are used, the Agency will verify telephone
measurements through follow-up site visits with samples of telephone
respondents and non-respondents, allowing the Agency to address any
potential self-reporting bias and non-response bias.  

To minimize measurement error and interviewer bias, on-site interviewers
will follow objective, written protocols in collecting data on shop
environmental performance.  In addition, the questions have been
organized to mimic a typical shop walk-through, thus minimizing the time
required to collect the necessary data elements.

The data collection instruments have been thoroughly reviewed by
numerous survey experts.  A pre-test has been conducted, as discussed in
section B.3.

Standardized software will be used for sample selection.

Data will be 100 percent independently keyed and verified.

	

Questionnaire Design:

The questions contained in the surveys are designed to obtain
information on (1) environmental performance related to current
hazardous waste management and training requirements under RCRA, (2)
environmental performance related to air emissions control requirements
associated with the recently promulgated Surface Coating Rule, (3)
environmental compliance assistance received by government agencies or
other entities, and (4) perceptions regarding the factors that influence
shop behaviors related to environmental performance.  Most of the
questions are designed to produce binary (i.e., yes/no) indicators of
environmental performance for use as dependent variables in the
statistical analysis.  

There are two different questionnaires, one for the on-site survey and
one for the telephone survey.  The on-site survey is designed to be
consistent with the general flow of a shop walk-through, with the
interviewer obtaining data on environmental performance through his or
her own observations and through targeted questions of shop personnel. 
The telephone survey focuses on a subset of the data items that are
included in the on-site survey, as the survey needs to be relatively
brief to discourage hang-ups.  The telephone survey focuses mainly on
environmental performance measures that can be later verified
independently through interviewer observations on site.  

Both questionnaires were reviewed by survey experts at Industrial
Economics and Abt Associates, and by EPA experts in program evaluation,
program review, statistics, and survey design. Copies of the survey
instruments are attached in Appendices B and C.

3. 	Pretests and Pilot Tests

The survey instrument was pretested on auto body shops in Boston,
Massachusetts, which is not included in the sampling frame for the
proposed survey. Several of the questions from the two survey modes are
identical, so the pretest was limited to a total of nine shops across
the two modes: the on-site survey was pretested on five shops, while the
telephone instrument was pretested on four shops.  The five shops used
to pretest the on-site survey were selected from a list provided by the
Boston Public Health Commission.  The four shops used in the phone
survey pretest were selected from a list derived from Dunn & Bradstreet
and Reference USA (SIC 7532).  The selected shops provide a range of
operation sizes (from “mom-and-pop” shops to national chains) and
locations within the city.  After the pretest, the survey instruments
and instructions were revised to address pretest observations regarding
question wording, clarity of interviewer instructions, question flow,
and survey length.

No pilot tests are planned for the survey.  

4.	Collection Methods and Follow-Up

Collection Methods:

A combination of site visits and telephone surveys will be used to
collect data from the sampled auto body shops.  Site visits that combine
survey questions with interviewer observations will be the primary data
collection approach.  This approach was selected by the Agency because
it avoids potential self-reporting bias and the response rate for site
visits is expected to be extremely high.  Interviewers for the site
visits will be Agency employees for Group C , and EPA-trained
contractors for Group X.  These interviewers will follow a written set
of procedures during all site visits, and they will be trained in the
collection of environmental performance data from auto body shops.

Due to high costs associated with site visits, the Agency prefers as a
general matter, to rely on telephone surveys in collecting environmental
performance data for its national compliance assistance activities. 
Thus, a subset of the on-site surveys will be preceded by telephone
surveys as part of an evaluation of the reliability of environmental
performance data collected through telephone surveys.  

Survey Response and Follow-up:

The target response rate for the on-site survey is 95 percent,
calculated as completed interviews divided by eligible shops sampled. 
Past experience by the Agency with site visits at auto body shops
indicates that surveyors are rarely denied entry, and the non-response
rate is expected to be nearly non-existent.  Shops will be allowed to
reschedule the site visit if the interviewer arrives at an inconvenient
time.  

The target response rate for the telephone survey is between 20 and 40
percent, calculated as completed interviews divided by eligible shops
sampled.  (Although the target response rate is somewhat low, the study
will evaluate non-response bias by conducting follow-up site visits at a
sample of non-respondent shops.)  Up to three attempts will be made to
reach each sampled shop.  If an answering machine picks up during the
first attempt, the interviewer will call again within the next day or
two.  If the second attempt is unsuccessful, the interviewer will leave
a message.  If the respondent does not return the call within two days,
the interviewer will attempt a third and final call.  As with the site
visits, shops will be able to reschedule the telephone interview for a
more convenient time.  In addition, if no one at the shop speaks
English, an attempt will be made to have an interviewer who speaks the
preferred language contact the shop.  

5. 	Analyzing and Reporting Survey Results

Data Preparation:

The Agency will use 100% double-keyed data entry for all survey data. 
Data will be entered into a Microsoft Access database.  Editing will
consist of automated logic and range checks, and checks for missing
data.  Missing environmental performance data will not be imputed and
will result in sample size reductions for the relevant analyses. 
Missing data on covariates will be imputed using standard methods such
as means and regression.    

Analysis:

This section describes the three general types of quantitative analyses
that will be conducted to evaluate (1) the short-term impact of
compliance assistance, (2) the long-term impact of compliance
assistance, and (3) the validity of telephone surveys.  In addition to
these three quantitative analyses, the Agency will analyze responses to
qualitative survey questions focused on respondents’ perceptions of
the factors that influence behaviors related to environmental
performance.  The results of the analyses will be relevant for the
population of auto body shops located in risk-based clusters in
Massachusetts; the Agency does not intend to use the results of the
pilot study to draw conclusions about the compliance assistance program
as a whole. 

Short-Term Impact of Compliance Assistance Outreach and Workshops	

The short-term impact of compliance assistance will be estimated as the
difference in performance between Groups A and B.  Depending upon the
type of variable, group performance will be described either as a
proportion (for binary variables, such as whether or not the shop is
using appropriate spray booths for painting) or as a mean (for
continuous variables, such as the number of gallons of paint a shop uses
per month). 

We will use two different approaches to estimate this difference.  The
first approach will involve a simple comparison of the two-phase
performance estimate for Group A to the two-phase performance estimate
for Group B.  This approach allows us to take advantage of any precision
gains provided by the application of the two-phase approach.  For any
given group, the two-phase estimate of performance is given by:

 

where:

r	= effective sample size for telephone survey

n	= initial sample size for the telephone survey

 	= average performance for respondent shops (telephone measurement)

 	= average performance for sample of respondent shops (site visit
measurement) 

 	= average performance for sample of respondent shops (telephone
measurement)

 	= average performance for sample of nonrespondent shops (site visit
measurement)

Group A will be stratified by receipt of face-to-face compliance
assistance.  As a result, for Group A, we will need to develop separate
two-phase estimates of performance for each stratum, then estimate the
overall performance for Group A as the weighted sum of these two
performance estimates, where the weights are equal to the proportion of
the population represented by each stratum. 

The second approach will involve estimating the difference within the
context of a multivariate regression, with variables included to control
for factors that EPA anticipates may impact performance.  Examples of
such factors include shop size (e.g., number of paint jobs per week),
number of years in operation, and geographic location.  For example, for
performance measures that are continuous, the site visit results will be
used to estimate the following type of model:

 

where

 	= performance measure for shop i

 	= a constant

 	= a binary (0/1) variable equal to one if the shop is in the treatment
group (Group A) and equal to zero otherwise

 	= a binary (0/1) variable equal to one if the shop attended a workshop
and equal to zero otherwise.

 	= a set of J shop characteristics expected to impact performance 

 	= a random error term representing the effect of unknown shop
characteristics that may impact performance

 , where f is the fraction of shops in Group A that attended a workshop.
 

Long-Term Impact of Compliance Assistance Package

The impact of Region 1’s two-year compliance assistance effort
(outreach, workshops, and shop visits) will be estimated as the
difference between 1) the change in performance for Group C and 2) the
change in performance for Group X.   We will use two different
approaches to estimate this difference.  The first approach will involve
a simple comparison of the change in performance for Group C to the
change in performance for Group X.   

The second approach will involve estimating the difference within the
context of a multivariate regression, with variables included to control
for other observable factors that EPA anticipates may impact
performance.  For example, for performance measures that are continuous,
the analysis of compliance assistance impacts will involve estimating
models of the following form:

 

where

 	= performance measure for shop i

 	= a binary (0/1) variable equal to one if the measurement is from the
treatment group in 2009 and equal to zero otherwise

 	= a binary variable equal to one if the measurement is from the
treatment group in 2011 and equal to zero otherwise

 	= a binary variable equal to one if the measurement is from the
comparison group in 2009 and equal to zero otherwise

 	= a binary variable equal to one if the measurement is from the
comparison group in 2011 and equal to zero otherwise

 	= a set of J shop characteristics expected to impact performance
(e.g., size of shop, number of years in operation, or geographic
location)

 	= a random error term representing the effect of unknown shop
characteristics that may impact performance

   have analogous interpretations.  The regression-adjusted
difference-in-differences estimator of the long-term impact of
compliance assistance is given by:

 

Assessing Telephone Survey Bias

 ):

 

 ), 

 

The performance for non-respondents is estimated as the performance rate
for the on-site visits with non-respondents.  The performance for
respondents is estimated by adjusting the overall average performance
rate from the telephone survey.  The adjustment is calculated from the
subsample of telephone survey respondents that also received site
visits, and it is equal to the difference between the average
performance rate as measured by the telephone survey and the average
performance rate as measured on-site.

Reporting Results:

A report containing the questionnaire, sampling plan, calculations, and
results (including variances and response rates) will be prepared.  The
report will be made available to the Agency and the public through the
following means:

A printed report, which will be distributed to all interested offices at
EPA. Compliance assistance providers from EPA and other organizations
will be alerted to the report via available electronic means (e.g.
listservs). Additional copies will be made available to the general
public through the National Technical Information System (NTIS)

An electronic copy of the report will be posted to EPA’s Office of
Enforcement and Compliance Assurance Website (  HYPERLINK
"http://www.epa.gov/compliance/assistance" 
www.epa.gov/compliance/assistance ), and other Agency websites will link
to the report (e.g., Region 1, OAQPS).

EPA will provide OMB with a copy of the final report, as part of its
continuing commitment to “conduct a survey every three years of a
statistically-valid sample of compliance assistance recipients to
measure behavior changes resulting from compliance assistance.”



REEFERENCES

Bloom, Howard S. Learning More from Social Experiments: Evolving
Analytic Approaches. Russell Sage Foundation. 2005.

Cochran, William G. Sampling Techniques (3rd Edition), John Wiley &
Sons. New York:. 1977. 

Orr, Larry L. Social Experiments: Evaluating Public Programs with
Experimental Methods. Thousand Oaks, CA: Sage Publications. 1999.

 These workshops will be organized by EPA together with local partners
(e.g., fire departments), and they will vary in content, duration, and
location.  However, at least one hour of all workshops will be dedicated
to presenting the new requirements associated with the Surface Coating
Rule.  A standard PowerPoint presentation will be used to cover material
related to the Surface Coating Rule.  

 Workshops and mailings will be provided to the control group shops
shortly after the survey described above has been completed to ensure
that all shops ultimately have the opportunity to receive compliance
assistance.  If any shops in the control group learn about the earlier
series of workshops and indicate to EPA that they would like to
participate, they will be encouraged to attend workshops at a later
date.  If they insist on attending the earlier series of workshops, they
will be allowed to do so and tracked as “crossovers” for the
analysis (see, e.g., Orr 1999, pg 211-213).  We anticipate that only a
small number of shops, if any, will insist on attending the earlier
series of workshops.  

 Group C is a combination of Groups A and B from the short-term
evaluation.  After the conclusion of the short-term experiment, the
shops in Group B will receive outreach materials and an offer to attend
a workshop identical to the workshops that were initially offered only
to Group A.  This will make Groups A and B equivalent with respect to
compliance assistance received, making the distinction between the two
groups irrelevant for the long-term experiment.  

 Compliance assistance will be delayed by approximately three months for
a subset of the Group C shops (i.e., until the end of the short-term
experiment).     

 Compliance assistance site visits will consist of a post-survey
“debrief” with the shop representative to review areas where
environmental performance could potentially be improved, including areas
related to the Surface Coating Rule requirements.  In addition, the
interviewer will provide the shop representative with (1) a guidance
document focused on multimedia compliance and environmental best
practices and (2) a brochure related to the Surface Coating Rule
requirements.  The entire site visit (including the survey and debrief)
is expected to last approximately 45 to 85 minutes.

 We are not planning a panel approach, where we would measure the same
shops at multiple points in time.  A panel approach is not feasible in
this case due to potential measurement effects.  

 Elevated Risk - NATA data was broken into five classes using the
Natural Breaks method -
http://webhelp.esri.com/arcgisdesktop/9.2/index.cfm?TopicName=Natural_br
eaks_(Jenks), towns that intersected any of the top three categories
were designated as elevated risk.  This was done for both Cancer Risk
and Non-Cancer Risk.  

 See Tables B-4 and B-5 for the MDEs associated with doubling the sample
sizes.

 Our calculations assume that any disagreement between the telephone and
on-site measurements is due to exaggeration of environmental performance
in the telephone interviews.  That is, we assume that telephone
respondents will not report unsatisfactory environmental performance
when the shop’s performance is in fact satisfactory.    

 The impact of performance rate on MDE is symmetric around 50%, so that
the MDEs associated with a 70% performance rate are identical to the
MDEs associated with a 30% performance rate (and similarly with 90% and
10%).

 Additional information regarding the selected shops is available from
Kenneth Harmon, (202) 564-7049.

 If the Agency doubles the sample size, EPA-trained contractors will
need to supplement the work of Agency employees for Group C.

 Note that although all four group dummies have been included in the
model, the constant term has been excluded. 

 Expanding the Use of Outcome Measurement for EPA’s Office of
Enforcement and Compliance Assurance, Report to OMB, July 31, 2006, p.
30.

Part B, Page   PAGE  1