**Maine Department of Environmental Protection**

**Implementation of a Voluntary Auto Body/Repair Environmental Results
Program**

**Quality Assurance Project Plan**

**Maine Department of Environmental Protection**

**Office of Innovation and Assistance**

**17 State House Station**

**Augusta, Maine 04333-0017**

**Sara M. Lippert**

**Phone: (207) 287-4432**

**Fax: (207) 287-2814**

**Email: sara.m.lippert@maine.gov**

# A PROJECT MANAGEMENT

## A1. Approval Sheet

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Sara Lippert Date

Maine DEP

Environmental Specialist II

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Ron Dyer Date

Maine DEP

Director, Office of Innovation and Assistance

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Julie Churchill Date

Maine DEP

Assistant Director, Office of Innovation and Assistance

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Steve DiMattei Date

US EPA

QA Chemist

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

George Frantz Date

US EPA

EPA-NE Project Officer

## A2. Table of Contents

[]{.mark}

[]{.mark}

[A PROJECT MANAGEMENT 1](#a-project-management)

[A1. Approval Sheet 1](#a1.-approval-sheet)

[A2. Table of Contents 2](#a2.-table-of-contents)

[A3. Distribution List 4](#a3.-distribution-list)

[A4. Project/Task Organization 4](#a4.-projecttask-organization)

[A5. Problem Definition/Background 6](#a5.-problem-definitionbackground)

[Rationale for initiating the project
6](#rationale-for-initiating-the-project)

[Objectives of the project 6](#objectives-of-the-project)

[Regulatory information, applicable criteria and action limits
6](#regulatory-information-applicable-criteria-and-action-limits)

[A6. Project/Task Description 6](#a6.-projecttask-description)

[Project overview. 6](#project-overview.)

[Project summary and work schedule.
7](#project-summary-and-work-schedule.)

[Geographic focus 9](#geographic-focus)

[Resource and time constraints 9](#resource-and-time-constraints)

[A7. Quality Objectives and Criteria
9](#a7.-quality-objectives-and-criteria)

[Detailed performance measures 9](#detailed-performance-measures)

[Quality objectives 11](#quality-objectives)

[A8. Special Training/Certification
11](#a8.-special-trainingcertification)

[A9. Documents and Records 12](#a9.-documents-and-records)

[Report format/information 12](#report-formatinformation)

[Document/record control 12](#documentrecord-control)

[Other records/documents 12](#other-recordsdocuments)

[Storage of project information 12](#storage-of-project-information)

[Backup of electronic files 13](#backup-of-electronic-files)

[B DATA GENERATION AND ACQUISITION
13](#b-data-generation-and-acquisition)

[B1. Sampling Process Design (Experimental Design)
13](#b1.-sampling-process-design-experimental-design)

[B2. Sampling Methods 13](#b2.-sampling-methods)

[Preparation of data collection instruments
14](#preparation-of-data-collection-instruments)

[B3. Sample Handling and Custody 14](#b3.-sample-handling-and-custody)

[Data entry QA procedures 14](#data-entry-qa-procedures)

[B4. Analytical Methods 14](#b4.-analytical-methods)

[B5. Quality Control 15](#b5.-quality-control)

[Crosschecking data 15](#crosschecking-data)

[Data anomalies 15](#data-anomalies)

[Quality control statistics 15](#quality-control-statistics)

[B6. Instrument/Equipment Testing, Inspection and Maintenance
15](#b6.-instrumentequipment-testing-inspection-and-maintenance)

[B7. Instrument/Equipment Calibration and Frequency
15](#b7.-instrumentequipment-calibration-and-frequency)

[B8. Inspection/Acceptance for Supplies and Consumables
15](#b8.-inspectionacceptance-for-supplies-and-consumables)

[B9. Non-Direct Measurements (I.e., Secondary Data)
15](#b9.-non-direct-measurements-i.e.-secondary-data)

[Key resources/support facilities needed
16](#key-resourcessupport-facilities-needed)

[Determining limits to validity and operating conditions
16](#determining-limits-to-validity-and-operating-conditions)

[B10. Data Management 16](#b10.-data-management)

[C ASSESSMENT/OVERSIGHT 17](#c-assessmentoversight)

[C1. Assessment and Response Actions
17](#c1.-assessment-and-response-actions)

[C2. Reports to Management 17](#c2.-reports-to-management)

[D DATA REVIEW AND EVALUATION 18](#d-data-review-and-evaluation)

[D1. Data Review, Verification and Validation
18](#d1.-data-review-verification-and-validation)

[D2. Verification and Validation Methods
18](#d2.-verification-and-validation-methods)

[D3. Evaluating Data in Terms of User Needs
18](#d3.-evaluating-data-in-terms-of-user-needs)

[Mathematical and statistical formulae
20](#mathematical-and-statistical-formulae)

[Approach to managing unusable data
23](#approach-to-managing-unusable-data)

## A3. Distribution List

Each person listed on the approval sheet and each person listed under
Project/Task Organization will receive a copy of this Quality Assurance
Project Plan (QAPP). Individuals taking part in the project may request
additional copies of the QAPP from personnel listed under Section A4.

This document has been prepared according to the United States
Environmental Protection Agency publication *EPA Requirements for
Quality Assurance Project Plans* dated March 2001 (QA/R-5).

## A4. Project/Task Organization

Personnel involved in project implementation are listed in Table 1, and
shown as an organization chart in Figure 1.

+----------------------+-------------------+--------------------------+
| **Table 1: Project   |                   |                          |
| Implementation       |                   |                          |
| Personnel**          |                   |                          |
+----------------------+-------------------+--------------------------+
|                      |                   |                          |
+----------------------+-------------------+--------------------------+
| **Individual**       | **Role in         | **Organizational         |
|                      | Project**         | Affiliation**            |
+----------------------+-------------------+--------------------------+
| Sara Lippert         | ERP Project       | Maine DEP                |
|                      | Manager           |                          |
+----------------------+-------------------+--------------------------+
| Ron Dyer             | ERP Management    | Maine DEP                |
|                      | Support           |                          |
| Julie Churchill      |                   |                          |
+----------------------+-------------------+--------------------------+
| Michael Crow         | Contractor        | The Cadmus Group, Inc.   |
|                      |                   |                          |
| Richard Krop         |                   |                          |
+----------------------+-------------------+--------------------------+
| Ed Cousins           | Bureau of Air     | Maine DEP                |
|                      | Quality           |                          |
| Peter Carleton       | Representative    |                          |
|                      |                   |                          |
| Louis Fontaine       |                   |                          |
+----------------------+-------------------+--------------------------+
| Scott Whittier       | Bureau of         | Maine DEP                |
|                      | Remediation and   |                          |
| Mike Hudson          | Waste Management  |                          |
|                      | Representative    |                          |
+----------------------+-------------------+--------------------------+
| Erich Kluck          | Bureau of Land    | Maine DEP                |
|                      | and Water Quality |                          |
| Sterling Pierce      | Representative    |                          |
+----------------------+-------------------+--------------------------+

The Maine DEP ERP Project Manager will be responsible for the following
activities:

-   Identify facility universe

-   Create sampling approach for Pollution Prevention (P2) audits

-   Construct sampling strategy/schedule for P2 audits

-   Develop web page for DEP web site

-   Develop workbook and checklists

-   Conduct P2 audits

-   QA management

-   Conduct outreach with regulated industry and internal/external
    stakeholders

-   Maintain official, approved QAPP

-   Data entry

-   Database management

-   Issue quarterly and annual reports to U.S. EPA

The Maine DEP ERP Management Support will be responsible for the
following activities:

-   Project oversight

-   Assist in coordination of interdepartmental meetings

-   Coordinate with senior management within each bureau

-   Policy review

The Cadmus Group, Inc. will be responsible for the following activities:

-   Help develop statistical methodology and Environmental Business
    Practice Indicators (EBPIs)

-   Help develop sampling options

-   Develop statistical methodology

The Maine DEP Bureau Representatives will be responsible for the
following activities:

-   Help develop EBPIs

-   Help develop workbook, self-certification checklist, P2 audit
    checklist

-   Conduct outreach at ERP workshops

The participating facilities will be responsible for submitting
self-certification materials and, if applicable, return-to-compliance
forms.

**Figure 1: Project Organizational Chart**

## 

## 

## A5. Problem Definition/Background

### Rationale for initiating the project 

Maine contains over 4,000 registered motor vehicle auto body and auto
repair shops in accordance with 1997 U.S. Economic Census
([www.census.gov)](). This is a significant amount of facilities and
many are located in our densely populated areas that are classified in
accordance with air quality non-attainment areas. This project is a
voluntary innovative permitting system based on the Environmental
Results Program (ERP) model for a currently non-permitted sector.

### Objectives of the project 

The main goals of this project is to promote pollution prevention
concepts, increase public and industry awareness of environmental health
concerns particularly in environmental justice, and measure increased
environmental compliance. Because there are a significant amount of auto
body businesses in Maine, it is anticipated a large impact will be made
on the aggregate reductions in hazardous waste and air emissions. The
focus area of our outreach will be the most populated areas in Maine
(southern coastal), which also includes air quality non-attainment
areas. The aggregate reduction impact in the Southern Maine region will
not only improve the environment but also reduce exposures to the mass
public to air toxins from both point source and fugitive emissions.

The measures we will be using to evaluate the project will include:

-   Quantity of Air pollution reduced or mitigated including VOCs and
    HAPS through proper use of paints and cleaners. Changes in practices
    to high efficiency/transfer paint techniques and use of alternative
    cleaners.

-   Rate of compliance with key Hazardous Waste regulations through
    manifest tracking and compliance checks.

-   Quantity of water pollution reduced or mitigated- this will include
    compliance with the underground injection control program,
    wastewater discharge licenses issued and possibly secondary data
    from the program.

-   Compliance rate will be assessed through initial and final
    compliance site assessments.

-   Reduction in risk to the community focused in environmental justice
    neighborhoods within the Southern Maine regional air non-attainment
    region.

```{=html}
<!-- -->
```
-   Work with RCRA compliance staff on streamlining two existing
    hazardous waste policies/licenses (abbreviated license and
    rag/wipes)

### Regulatory information, applicable criteria and action limits

-   Compliance with state and federal air regulations

-   Compliance with state and federal hazardous waste regulations

-   Compliance with state and federal water regulations

## A6. Project/Task Description

### Project overview. 

This project will allow Maine DEP to explore whether an approach modeled
upon the Environmental Results Program (ERP) can help achieve these
goals, while improving regulatory cost-effectiveness. The ERP is an
innovative approach to solving high-priority environmental problems in
industry sectors largely comprised of small businesses. The ERP concept
combines technical assistance, self-certification, audits, and
statistically based performance measurement in order to reduce
environmental impacts of business.

The promise of ERP is that it will cost-effectively reduce environmental
impacts of small businesses that may present a substantial cumulative
environmental risk. Businesses targeted so far by ERP include gas
stations, auto salvage yards, auto body and mechanical repair shops, dry
cleaners, and printers. ERP can help environmental agencies identify
previously unknown facilities, measure performance, increase regulatory
efficiency, and help improve overall environmental performance. ERP is
in part designed to help facilities that want to comply but do not
understand their requirements, and evidence suggests that ERP can
motivate firms to comprehensively review their environmental performance
and take needed action to come into compliance and adopt best practices.

### Project summary and work schedule. 

This project\'s major tasks and timeline are outlined in the table
below.

  -------------------- ------------------------------------ ----------- ------------
  **Table 2: Schedule                                                   
  of major project                                                      
  tasks**                                                               

  **Task Name**        **Task Description**                 **Start     **End Date**
                                                            Date**      

  Receive funding      Receive EPA grant funding for        October     October 2004
                       project                              2004        

  Hire staff           Hire staff for project               October     October 2004
                                                            2004        

  Train staff          Multi-media training for staff       October     October 2005
                                                            2004        

  Timeline development Develop a Gantt chart detailing flow October     October 2004
                       of work and timelines                2004        

  Outreach             Coordinate and implement a           October     December
                       multi-state workshop for the         2004        2004
                       compliance staff and states awarded              
                       State Innovation Grants                          

  Facility             Determine the exact characteristics  October     December
  identification       of facilities to be included in this 2004        2004
                       project, and compile a list of                   
                       facilities from reliable sources.                

  Outreach             Convene internal stakeholders (DEP   October     August 2005
                       bureaus)                             2004        

  Outreach             Convene external stakeholders        October     August 2005
                       (Compliance Advisory Panel, Auto     2004        
                       Associations)                                    

  Develop incentives   Develop incentives to attract        October     August 2005
                       volunteers into the ERP              2004        

  Statistical          Develop 10-20 Environmental Business October     July 2005
  methodology          Practice Indicators (EBPIs) in       2004        
                       coordination with key DEP compliance             
                       representatives. Development of a                
                       statistical methodology to drive                 
                       performance measurement and                      
                       analytical tasks.                                

  P2 audit checklist   Develop P2 audit checklist           October     April 2005
                                                            2004        

  Self-certification   Develop multi-media compliance       October     May 2005
  checklist            checklist for self-certification     2004        

  Database development Develop an ERP database to house     March 2005  August 2005
                       indicator data                                   

  QAPP finalization &  Finalize QAPP based upon results of  February    August 2005
  approval             the measures identification,         2005        
                       statistical methodology, and data                
                       management tasks. Primary data                   
                       collection will not occur before                 
                       relevant parts of the QAPP are                   
                       finalized and approved by EPA.                   

  Review existing      Review existing compliance records   October     Ongoing
  compliance records   within the department to avoid       2004        throughout
                       audits at facilities when feasible               project
                       with recent or ongoing compliance                
                       issues                                           

  Sampling approach    Design sampling approach such that   January     May 2005
                       we can determine if there are any    2005        
                       statistically significant changes in             
                       environmental compliance                         

  Test audit           Run a test audit to assure that      July 2005   August 2005
                       audit checklist questions are                    
                       appropriate                                      

  Workbook             Develop and complete a draft         October     August 2005
                       compliance workbook                  2004        

  Workshops            Conduct two workshops in the         November    January 2005
                       Southern Maine Regional Area to      2005        
                       educate facilities regarding ERP                 
                       workbooks/self-certification                     
                       checklists                                       

  RCRA policy issues   Identify and assist in developing    August 2004 November
                       suggestions and streamlines on HW                2005
                       RCRA policy issues developed and                 
                       drafted: 1. Wipes/rags treatment                 
                       options & 2. Licensing technology                

  Baseline audits      Audits at facilities to establish a  August 2005 October 2005
  (establishing a      baseline for performance measures.               
  performance measures Facilities selected at random from               
  baseline)            the entire targeted population,                  
                       based upon sample design from                    
                       statistical methodology.                         

  Baseline analysis    Analysis of audit data to establish  November    December
                       a baseline for the project\'s        2005        2005
                       performance measures.                            

  Self-certification   Implementation of a voluntary        November    January 2005
                       facility self-certification          2005        
                       approach. Self-certification refers              
                       to the submission of a legally                   
                       binding record of a facility's                   
                       compliance and beyond-compliance                 
                       practices.                                       

  Analysis of          Analysis of self-certification data, February    March 2005
  self-certification   with primary purpose of identifying  2005        
  results              opportunities for selective                      
                       follow-up (next step).                           

  Beyond compliance    Initiate contracts to facilities     March 2006  September
  contracts            that are high performers interested              2006
                       in beyond compliance education,                  
                       through soliciting volunteers at ERP             
                       workshops and certification reviews              

  Selective follow-up  Selective follow-up with             March 2005  May 2006
                       self-certifying facilities, based                
                       upon analysis of self-certification              
                       data. Selective follow-up may                    
                       include phone calls, audits and                  
                       enforcement. Selective follow-up is              
                       not typically based upon a random                
                       sample.                                          

  Post-certification   Follow up random post-certification  March 2005  June 2006
  audits               audits                                           

  Data analysis        Tabulate scores for EBPIs and total  July 2006   August2006
                       compliance per facility (aggregate               
                       EBPIs), industry (aggregate EBPIs)               

  Data analysis        Tabulate accuracy analysis scores    July 2006   August 2006
                       for self certification vs. audits                

  STEP-UP sign up      Auto body & repair facilities begin  September   October 2006
                       the sign up process with the STEP-UP 2006        
                       Program                                          

  Track RCRA policy    Track streamlines on HW RCRA policy  August 2004 October 2006
  issues               issues developed and assist the                  
                       department in carrying the policies              
                       forward                                          

  Case study           Develop and complete case study of   August 2006 October 2006
                       ERP project (final report)                       

  Reporting to EPA     Reporting shall include quarterly,   October     October 2006
                       annual and final reports.            2004        
  -------------------- ------------------------------------ ----------- ------------

### Geographic focus 

The focus of the project will be the Southern Maine air quality
non-attainment areas in York, Cumberland and Sagadahoc counties. This
includes the cites/towns of: Alfred, Arundel, Berwick, Biddeford,
Buxton, Eliot, Hollis, Kennebunk, Kennebunkport, Kittery, North Berwick,
Old Orchard Beach, Saco, Sanford, York, Brunswick, Cape Elizabeth,
Casco, Cumberland, Falmouth, Freeport, Gorham, Gray, New Gloucester,
North Yarmouth, Portland, Scarborough, South Portland, Standish,
Westbrook, Windham, Yarmouth, Bath, Bowdoinham, Richmond, Topsham, West
Bath, and Woolwich.

### Resource and time constraints 

The budget for the ERP project is \$152,000 federal funding and \$47,000
state funding. This will be a two year project that will run October
2004 to October 2006.

## A7. Quality Objectives and Criteria

### Detailed performance measures 

This project is primarily interested in the following list of likely
performance measures. Note that one of the tasks of this project
involves revisiting and reaffirming/revising these draft performance
measures.

Regulatory Indicators:

-   Does any airborne dust from sanding leave the business premises?
    (i.e. open windows, open doors, unfiltered exhaust vents)

-   Does the shop carry out coating processes in a spray booth to
    contain paint emissions and over spray?

-   Does painting and/or painting take place in areas outside of a spray
    booth?

-   Do any airborne emissions from painting/coating leave the business
    premises? (i.e. open windows, open doors, unfiltered exhaust vents)

-   Are solvents, contaminated rags or other VOC- and HAP-containing
    materials stored in closed containers when not in use?

-   Does the shop generate any of the following wastes? \[Check all that
    apply. For checked items, indicate how you dispose of the wastes.\]

> Waste: gasoline, brake fluid, lead acid batteries, paint, paint
> thinner, oil, parts cleaning solvents, batteries, fluorescent lamps,
> mercury switches, cathode ray tubes

-   What is the greatest monthly amount of hazardous waste generated by
    the facility in the last 12 months?

-   Does the shop properly label containers of hazardous waste
    (including waste paint, waste paint thinner and waste parts cleaning
    solvents)? \[Labels must indicate the contents of the container,
    must say \"hazardous waste,\" must list the date of each deposit
    into the container and the date the container becomes full.\]

-   Does the shop containerize the wastes (like rags and other absorbent
    material) generated from cleanups of spills and dispose of it as
    hazardous waste?

-   Have you ever filled out or signed a hazardous waste manifest?

-   Does the shop ever add hazardous wastes such as waste gasoline,
    solvents or paint thinners into the waste oil?

-   Have inactive floor drains been properly sealed/closed?

-   Does the shop have any active floor drains?

-   Does the shop conduct vehicle maintenance and repair in areas (bays)
    with unsealed floor drains?

-   Are active and inactive floor drains registered with the MEDEP?

-   Does the shop discharge wastewater (other than sanitary wastewater)
    to bathroom/kitchen sinks, toilets, showers, shop wash basins,
    emergency showers, eyewash stations, or other non-industrial
    wastewater outlets?

-   If shops washes vehicles outside, where does the wash water
    discharge to? (ditch, stream, wetland, vegetated buffer, storm
    drain, other) \[Check all that apply.\]

Beyond Compliance Indicators:

-   Does the shop train all employees in the proper use and handling of
    paints and coatings according to the manufacturers' recommendations
    to minimize air emissions?

-   Does the shop utilize enclosed spray gun cleaning, VOC recycling
    systems or other spray gun cleaning methods to reduce or eliminate
    VOC emissions?

-   Does the shop employ a training program in the proper use and
    handling of solvents and waste products to minimize air emissions?

-   Does the shop burn waste oil in a waste oil furnace?

### Quality objectives 

Quality objectives for these performance measures will be developed as
part of the Measures Identification and Statistical Methodology tasks.

This determination will take into account both the best practices for
similar projects and the resources available for this project. In part,
the ERP Management Support will rely upon EPA\'s *Generic Guide to
Statistical Aspects of Developing an Environmental Results Program*
(2003) for advice in making decisions related to the optimizing the
following aspects of data quality for this project:

-   Precision

-   Bias

-   Representativeness

-   Completeness

-   Comparability

-   Sensitivity (if applicable)

## A8. Special Training/Certification

The Maine DEP will develop and deliver mandatory and voluntary training
sessions to key parties to ensure quality data collection, to the extent
practicable.

Mandatory intensive in-person training sessions will be delivered to the
following individuals to ensure quality data collection:

-   inspectors who will be collecting baseline and post-certification
    data

-   data-entry personnel who will be processing data from audits and
    self-certification responses

-   QA/QC personnel (if any additional training is needed to familiarize
    them with the project)

-   Individuals who will be compiling the database containing the
    universe of facilities

Each session will cover proper data collection and QA procedures.
Training will be augmented by debriefing personnel shortly after their
tasks have begun, to correct and clarify appropriate practices.

Voluntary intensive in-person training sessions will be offered to the
self-certifying facilities. Facilities will also be provided with clear
written instructions on how to prepare and submit data, and they will be
able to call a phone number to ask anonymous questions if they wish.

The ERP Management Support is responsible for ensuring that all
personnel involved with data generation (including state personnel,
contractors, and partners) have the necessary QA training to
successfully complete their tasks and functions. The ERP Management
Support will document attendance at all training sessions. Attendance
records for voluntary trainings may not include names, given
privacy/confidentiality concerns.

The ERP Management Support is also responsible for ensuring the
self-certification materials sent to facilities clearly document how
facilities should properly prepare and submit their data.

## A9. Documents and Records

### 

### Report format/information 

The format for all data reporting packages will be consistent with the
requirements and procedures used for data validation and data assessment
described in this QAPP.

### Document/record control

The recording media for the project will be both paper and electronic.
The project will implement proper document control procedures for both,
consistent with Maine DEP\'s Quality Management Plan. For instance,
hand-recorded data records will be taken with indelible ink, and changes
to such data records will be made by drawing a single line through the
error with an initial by the responsible person. The ERP Project Manager
will have ultimate responsibility for any and all changes to records and
documents. Similar controls will be put in place for electronic records.

The Maine DEP ERP Project Manager shall retain all updated versions of
the QAPP and be responsible for distribution of the current version of
the QAPP. The Maine DEP ERP Project Manager and the Maine DEP ERP
Management Support will approve annual updates. ERP Management Support
shall retain copies of all management reports, memoranda, and all
correspondence between the Maine DEP and all project personnel
identified in A4.

### Other records/documents 

Other records and documents that will be produced in conjunction with
this project include:

-   Audit checklists and reports

-   Self-certification forms

-   Return-to-compliance forms

-   Non-applicability forms

-   Enforcement documentation

-   Facility outreach materials, including workbook, fact sheets,
    brochures, etc.

-   Amended QAPP

-   Readiness reviews (see below)

-   Data handling reports

-   Quarterly and annual progress reports to EPA

-   Project final report (to include discussion of QA issues
    encountered, and how they were resolved)

### Storage of project information 

The electronic information will be saved to a network server that is
backed up daily. All paper copies will be filed in a secure file cabinet
in the Office of Innovation and Assistance until 3 years after the grant
close date.

### Backup of electronic files 

The information will be collected and organized in a dedicated secure
database, which will be backed up daily.

# B DATA GENERATION AND ACQUISITION

## B1. Sampling Process Design (Experimental Design)

A key task in this project will be to develop a sound statistical
methodology for collecting and analyzing facility data, in order to draw
inferences related to the selected performance measures. The major
quality objective will be to collect representative data that truly
reflect the conditions of the universe of facilities that this ERP
focuses upon. Facility data is of two types: (1) audit data, which will
be collected by trained Maine DEP inspectors from randomly sampled
facilities, and (2) self-certification data[^1], which will be collected
from facilities through a mail survey process.

To meet the precision targets of the survey, the sample for each round
of inspections will include between 50 and 75 shops. A simple random
sample of shops will be selected. Each shop will be assigned a unique
random number. Research Randomizer (http://www.randomizer.org/form.htm)
will then be used to select random numbers from 1 through 159 (the
amount of shops in the target universe). The numbers from Research
Randomizer will be compared to the random numbers assigned to the shops,
and those will be the shops that will be audited.

## B2. Sampling Methods

As described above, the primary data collected and used by this ERP will
come from a survey data collection process. The Maine DEP will conduct
simple random Pollution Prevention (P2) audits of facilities in the
universe for both the baseline and post certification audits.

Based on budgetary and other constraints, the DEP will be able to
collect data from 50 to 75 shops. The margin of error in estimating the
percent of the eligible shops that use a specific environmental business
practice at the 95 percent confidence level depends on the number of
eligible shops in the sample. The following table gives the largest
possible error for several different sample sizes assuming that all
shops in the sample are eligible (that is, the practice being measured
applies to all shops in the sample).

+-----------------------------------+-----------------------------------+
| **Sample Size**                   | **Margin of Error at 95% level**  |
|                                   |                                   |
|                                   | **will be less than or equal      |
|                                   | to:**                             |
+-----------------------------------+-----------------------------------+
| 50                                | 11.2%                             |
+-----------------------------------+-----------------------------------+
| 59                                | 10.0%                             |
+-----------------------------------+-----------------------------------+
| 75                                | 8.2%                              |
+-----------------------------------+-----------------------------------+

DEP also wishes to compare results between the two rounds of
inspections. If 50 shops are inspected in each round, and the business
practice in question is relevant for all 50 shops, there is at least an
80 percent probability of detecting a real difference of 27 percent (at
a 95 percent confidence level). If 75 shops are included in each sample,
there is at least an 80 percent probability of detecting a real
difference of 22 percent (at a 95 percent confidence level). Smaller
observed differences can also indicate a real genuine change, but with
less than 80 percent probability (The probability of detecting a real
difference is often referred to as \"power.\" In this case, the
recommended power for the test is 80 percent).

Please note that, as a general rule, if the number of eligible
facilities for a given question is smaller than the sample size, then
the effective sample size is lowered. Consequently, the margin of error
and confidence interval for that question will be larger than if the
question applied to all facilities. The size of the difference between
the two rounds of inspections that indicates a real difference also will
be larger. Also, as the observed proportion moves away from 50%, the
margin of error or confidence interval associated with that proportion
will get smaller.

### Preparation of data collection instruments 

All data collection instruments will be subject to multiple rounds of
review by relevant internal and external stakeholders to help assure the
collection of high-quality and representative data. Data collection
instruments will be prepared in accordance with the guidance on data
collection instruments provided in EPA\'s *Generic Guide to Statistical
Aspects of Developing an Environmental Results Program* (2003).
Specifically, preparation will follow the checklist for data collection
instruments provided in an appendix of that guide.

## B3. Sample Handling and Custody

Upon completion of paper checklists, inspectors will sign the
checklists. Inspectors will enter data from paper checklists into the
electronic database. Facilities will mail signed forms into Maine DEP,
where data-entry staff will input data into the electronic database.

Chain of custody is not relevant to this project.

### 

### Data entry QA procedures

Procedures for entering hand-written data into the database will follow
standard quality assurance procedures (e.g., 100% verification using
independent double key entry), consistent with Maine DEP\'s Quality
Management Plan. Detailed quality assurance procedures for data entry
and acceptance will be prepared during the development and
implementation of a data management strategy.

## B4. Analytical Methods

This project will follow well-recognized statistical analytical methods
for survey samples. No physical tests or chemical analyses are
anticipated for this project.

## B5. Quality Control

This project will undertake the following specific steps to
measure/estimate the effect of data errors, consistent with Maine DEP\'s
Quality Management Plan.

### 

### Crosschecking data

Completed questionnaires will be subjected to 100 percent editing review
in preparation for data entry. Every response field will be examined to
check skip patterns, clarify handwriting, and identify any potential
problems. Each form will be entered with 100 percent verification; in
other words, using independent double-key entry. Primary data collection
forms will be designed in such a way to allow internal crosschecking of
data by comparing answers of different questions to each other, and such
crosschecking will be automatic for electronically entered data.
Further, post-certification audits will offer the opportunity to compare
audit results with self-certification results, if the facilities sampled
have submitted self-certification forms.

### Data anomalies

Procedures for handling data anomalies (such as outliers and missing
data) will be handled based on guidance prepared in the project-specific
statistical methodology.

### Quality control statistics

After the data are entered, several automated checks can be conducted.
Questions that required continuous variables should be checked to ensure
that the answers are within acceptable ranges. Consistency among answers
also can be checked; e.g., if an affirmative response to one question
precludes an affirmative response to another question, the automated
checks can confirm the answers are consistent.

The quality control statistics to be used in this project are described
in more detail in section D3.

## B6. Instrument/Equipment Testing, Inspection and Maintenance

This section is not relevant to this project. The project will not
involve such scientific instruments and equipment.

## B7. Instrument/Equipment Calibration and Frequency

This section is not relevant to this project. The project will not
involve such scientific instruments and equipment.

## B8. Inspection/Acceptance for Supplies and Consumables

This section is not relevant to this project. The project will not
involve such supplies and consumables.

## B9. Non-Direct Measurements (I.e., Secondary Data) 

This project will rely upon secondary data to identify the facilities in
the target population.

  ----------------- ----------------- ----------------- -----------------
  **Table 3:                                            
  Non-Direct                                            
  Measurements                                          
  (I.e., Secondary                                      
  Data)**                                               

                                                        

  **Data Sources**  **Intended Use**  **Rationale for   **Acceptance
                                      Use**             Criteria**

  Maine DEP         Identifying the   Commonly accepted All records will
  database of       target            source of         be accepted
  facilities        population, for   facility list     unless sample
                    the sample                          response
                                                        indicates
                                                        facility should
                                                        not be part of
                                                        target
                                                        population. Maine
                                                        DEP will
                                                        crosscheck any
                                                        facility that
                                                        self-identifies
                                                        as non-applicable
                                                        to this project.
  ----------------- ----------------- ----------------- -----------------

### 

### Key resources/support facilities needed

Maine DEP will require access to the data sources mentioned above, and
this information will be managed within the database created/utilized
for the overall project. Maine DEP does not anticipate any obstacles to
this approach.

### Determining limits to validity and operating conditions 

Database containing the list of targeted facilities will be designed
such that the original source for all facility data is marked, and
procedures will be in place such that only the ERP Project Manager can
officially remove a facility entry from the target population. In such
cases, facility entry will not be deleted from the database but will be
marked as non-applicable, and corrective data will be provided in fields
parallel to the original data.

## B10. Data Management

As part of this project, Maine DEP and The Cadmus Group, Inc. will
develop a data management strategy, and amend the QAPP based upon the
strategy. The ERP Management Support is responsible for ensuring that
that strategy is developed and that the QAPP is amended to reflect that
strategy. The strategy will be consistent with the existing Maine DEP\'s
Quality Management Plan. Once amended, this QAPP section on data
management will provide information on the following issues:

-   Data management scheme, from field to final use and storage

-   Standard recordkeeping and tracking practices, and document control
    system (citing relevant agency documentation)

-   Data handling equipment/procedures that will be used to process,
    compile, analyze, and transmit data reliably and accurately

-   Individuals responsible for elements of the data management scheme

-   Process for data archival and retrieval

# C ASSESSMENT/OVERSIGHT

## C1. Assessment and Response Actions

The ERP Project Manager will conduct a Readiness Review immediately
prior to the five major data collection tasks: identifying targeted
facilities, baseline audits, self-certification, targeted follow-up and
post-certification audits. The ERP Project Manager will report findings
to the ERP Management Support, who will take corrective action (if any
is necessary) before the data collection task begins. Further, the ERP
Management Support and ERP Project Manager will thoroughly debrief
project implementation staff a short time after beginning their
respective implementation tasks, to identify emerging/unanticipated
problems and take corrective action, if necessary.

## C2. Reports to Management

Three kinds of reports will be prepared: readiness reviews (described
above), regular quarterly and annual progress reports, and project final
report. Progress reports will note the status of project activities and
identify whether any QA problems were encountered (and, if so, how they
were handled). Project final report will analyze and interpret data,
present observations, draw conclusions, identify data gaps, and describe
any limitations in the way the data should be used.

**Project QA Status Reports**

  ----------------- ----------------- ----------------- -----------------
  **Type of         **Frequency**     **Preparer**      **Recipients**
  Report**                                              

  Amended QAPP      Once, before      Maine DEP ERP     All recipients of
                    primary data      Project Manager   original QAPP
                    collection begins                   

  Readiness Review  Before each major Maine DEP ERP     Maine DEP ERP
                    data collection   Project Manager   Management
                    task                                Support

  Progress Report   Quarterly         Maine DEP ERP     U.S. EPA Project
                                      Project Manager   Officer (Copying
                                                        US EPA OPEI)

  Progress Report   Annually          Maine DEP ERP     U.S. EPA Project
                                      Project Manager   Officer (Copying
                                                        US EPA OPEI),
                                                        stakeholders

  Final Project     Once              Maine DEP ERP     U.S. EPA Project
  Report                              Project Manager   Officer (Copying
                                                        US EPA OPEI),
                                                        stakeholders
  ----------------- ----------------- ----------------- -----------------

# 

# 

# D DATA REVIEW AND EVALUATION

## D1. Data Review, Verification and Validation

This QAPP shall govern the operation of the project at all times. Each
responsible party listed in Section A4 shall adhere to the procedural
requirements of the QAPP and ensure that subordinate personnel do
likewise.

This QAPP shall be reviewed at least annually to ensure that the project
will achieve all intended purposes. All the responsible persons listed
in Section A4 shall participate in the review of the QAPP. The ERP
Management Support and ERP Project Manager are responsible for
determining that data are of adequate quality to support this project.
The project will be modified as directed by the ERP Management Support.
The ERP Management Support shall be responsible for the implementation
of changes to the project and shall document the effective date of all
changes made.

It is expected that from time to time ongoing and perhaps unexpected
changes will need to be made to the project. The ERP Management Support
shall authorize all changes or deviations in the operation of the
project. Any significant changes will be noted in the next report to
EPA, and shall be considered an amendment to the QAPP. All verification
and validation methods will be noted in the analysis provided in the
final project report.

## 

## D2. Verification and Validation Methods

To confirm that QA/QC steps have been handled in accordance with the
QAPP, a readiness review will be conducted before key data
collection/analysis steps, and data handling reports will be prepared
after each step. These reviews and reports will be consistent with Maine
DEP\'s Quality Management Plan. Standard statistical tests (described
below in Section D3) will be used to determine the extent to which
inferences can be drawn from the sample data.

## D3. Evaluating Data in Terms of User Needs

This section was developed consistent DEP\'s Quality Management Plan and
EPA\'s *Generic Guide to Statistical Aspects of Developing an
Environmental Results Program* (2003).

Analysis of the Sample Data

The P2 audit checklist requires three types of responses.

1\. **Dichotomous or yes/no responses.** The inspector indicates whether
the shop uses the practice in question or not. For example, the
questionnaire asks under the air section "Does the shop carry out
coating processes in a spray booth to contain paint emissions and
over-spray?"

The majority of questions on the P2 audit checklist are yes/no
questions. These data can be used to estimate the proportion of
facilities that use a particular practice in a single round of
inspections or differences in proportions between the two rounds.

2\. **Categorical response** in which the inspector checks all that
apply. For example, the questionnaire asks "Does the shop use any
painting and coating application techniques that minimize paint and
coating use? \[Check all that apply.\]" The possible techniques include
HVLP Spray Gun, Electrostatic Spray Gun, Laser Touch, and Other.

Each categorical response must be converted into a series of yes/no
responses that represent all the possible ways the question can be
answered. If the inspector can check up to four responses, for example,
there are 16 possible ways the question can be answered. Continuing with
the example of painting and coating techniques, the possible answers
are:

1\. None

2\. HVLP Spray Gun Only

3\. Electrostatic Spray Gun Only

4\. Laser Touch Only

5\. Other Only

6\. HVLP Spray Gun and Electrostatic Spray Gun only

7\. HVLP Spray Gun and Laser Touch only

8\. HVLP Spray Gun and Other only

9\. Electrostatic Spray Gun and Laser Touch only

10\. Electrostatic Spray Gun and Other only

11\. Laser Touch and Other only

12\. HVLP Spray Gun, Electrostatic Spray Gun, and Laser Touch only

13\. HVLP Spray Gun, Laser Touch, and Other only

14\. HVLP Spray Gun, Electrostatic Spray Gun and Other only

15\. Electrostatic Spray Gun, Laser Touch, and Other only

16\. HVLP Spray Gun, Electrostatic Spray Gun, Laser Touch, and Other

Each of these 16 combinations becomes a yes/no variable (i.e., the
facility either uses the combination or does not.)

Another approach for summarizing these categorical responses is to
indicate the number of different practices used by each facility.
Continuing with the painting and coating example, the response to the
question can be used to create other yes/no variables, such as:

1\. The facility uses none of the practices

2\. The facility uses exactly one of the practices

3\. The facility uses exactly two of the practices

4\. The facility uses exactly three of the practices

5\. The facility uses all four of the practices

6\. The facility uses at least one of the practices

7\. The facility uses two or more of the practices

8\. The facility uses three or more of the practices

3\. **Continuous variables.** For example, under general hazardous
waste, the questionnaire asks "What is the greatest monthly amount of
hazardous waste generated by the facility in the last 12 months?"

Using mean and standard deviation, the ResultsAnalyzer.XLS spreadsheet
will calculate the 95 percent confidence interval for the mean for
continuous data from a single sample. It also will conduct hypotheses
tests for differences in means between samples.

Proportions will be used to characterize the responses to the first two
types of questions. Means will be used to characterize the third type of
data. Associated 95 percent confidence intervals can be estimated for
each. The Cadmus Group, Inc. has provided Maine DEP with an Excel
spreadsheet (ResultsAnalyzer.XLS) that will calculate the estimated
proportion and 95 percent confidence intervals for the first two types
of data. Given a mean and standard deviation, it will calculate the 95
percent confidence interval for means as well.

### 

### Mathematical and statistical formulae

Confidence Intervals for a Proportion from a Single Sample

The score interval for a proportion is given by:

\(3\)
$\frac{p + Z^{2}2n}{1 + Z^{2}n} \pm \frac{Z\sqrt{\frac{p(1 - p)(1 - nN) + Z^{2}4n}{n}}}{1 + Z^{2}n}$

Where:

*p* is the proportion of eligible facilities in the sample that use a
specific practice.

*Z* is the standard normal deviate corresponding to the allowable
probability that the confidence interval will not include the true
proportion.

*n* is the number of eligible facilities in the sample to which the
practice applies.

*N* is the number of facilities in the region of the state.

Confidence Intervals for a Mean

The confidence interval for a sample mean is given by:

\(4\) $\overline{x} \pm t\sqrt{\frac{s^{2}}{n}(1 - nN)}$

Where:

$\overline{x}$is the sample mean among eligible facilities that use a
specific practice.

*t* is the value of the t-distribution with n-1 degrees of freedom
corresponding to the allowable probability that the confidence interval
will not include the true proportion.

*s* is the standard deviation from the sample.

Hypothesis test for comparison of proportions from two samples

The test statistic is equal to:

\(5\)
$\frac{\delta - 0\text{.}5\left( 1n_{1} + 1n_{2} \right)}{\sqrt{p(1 - p)\left( 1n_{1} + 1n_{2} \right)}}$

Where:

$\delta$is the difference in proportions between the two samples,
*p~1~*-*p~2~.*

*n~1~* is the size of the first sample.

*n~2~* is the size of the second sample.

*p* is the average proportion of the two samples that use the practice
in question,\
(*n~1~p~1~* + *n~2~p~2~*)/(*n~1~* + *n~2~*).

The null hypothesis is that the proportions of eligible facilities in
the population that use the practice in the two time periods are
essentially the same. The alternative hypothesis is that the proportion
is greater in the second time period than in the first (i.e., it is a
one-sided test). The critical value is the value of a standard normal
distribution corresponding to the allowable probability that the null
hypothesis is rejected when it is true.

Hypothesis test for comparison of means from two samples

The test statistic is equal to:

\(6\)
$\frac{\delta}{\sqrt{\frac{\left( n_{1} - 1 \right)s_{1}^{2} + \left( n_{2} - 1 \right)s_{2}^{2}}{\left( n_{1} + n_{2} - 2 \right)}\left( 1n_{1} + 1n_{2} \right)}}$

Where:

$\delta$is the difference in means between the two samples,
*p~1~*-*p~2~.*

*s~1~* is the standard deviation from the first sample.

*s~2~* is the standard deviation from the second sample.

The null hypothesis is that the means in the two time periods are
essentially the same. The alternative hypothesis is that the mean is
greater in the second time period than in the first (i.e., it is a
one-sided test). The critical value is the value of a t-distribution
corresponding to the allowable probability that the null hypothesis is
rejected when it is true.

Continuous Variables

The sample mean is the total quantity divided by the number of
*eligible* facilities:

\(1\) $\overline{x} = \frac{\sum_{i = 1}^{n}x_{i}}{n}$

Where $\overline{x}$is the sample mean for the continuous variable *x*,
*x~i~* is the observed quantity of *x* for shop *i*, and *n* is the
number of facilities for which the practice is relevant. The standard
deviation is a measure of the variability in the data and is given by:

\(2\)
$s = \sqrt{\frac{\sum_{i = 1}^{n}\left( x_{i} - \overline{x} \right)^{2}}{n - 1}}$

Excel or a statistics package may be used for these calculations. The
Excel function average() computes the average value of the range of
cells listed in the parentheses. The Excel function stdev() will
calculate the standard deviation of the range of cells listed in the
parentheses.

Sample Size Calculations

The formula for the sample size required to estimate a proportion with
specified precision target for a single sample from an infinite
population is:

\(7\)
$n_{I} = \left( \frac{p(1 - p)}{2\delta^{2}} - 1 \right)Z^{2} + \left( \frac{Z^{2}}{2\delta} \right)\sqrt{1 - 2p(1 - p) + \left( \frac{p(1 - p)}{\delta} \right)^{2}}$

Where:

*p* is the proportion.

$\delta$is the half-width of the confidence interval.

*Z* is the standard normal deviate corresponding to the allowable
probability that the confidence interval will not include the true
proportion.

Corrected for a finite population, the required sample size is given by:

\(8\) $n_{F} = \frac{n_{I}}{1 + \frac{n_{I} - 1}{N}}$

*N* is the size of the population. The formula for the sample size
required to test for a difference in proportions between two samples is:

\(9\)
$n_{u} = \frac{\left( Z_{1 - \alpha}\sqrt{(r + 1)p(1 - p)} + Z_{1 - \beta}\sqrt{(r + 1)(p - \delta 2)\left( 1 - (p - \delta 2) \right)} \right)^{2}}{\delta^{2}}$

Where:

*Z~1-α~* is the standard normal deviate corresponding to the allowable
probability that the null hypothesis of the test will be rejected when
it is true.

*Z~1-β~* is the standard normal deviate corresponding to the allowable
probability that the null hypothesis of the test will be accepted when
it is false.

*r* is the ratio of the size of the second sample to the size of the
first sample.

$\delta$is the minimum detectable difference between the proportions in
the two samples.

Equation 8 assumes the underlying data are continuous. To account for
the fact that the data are not continuous, apply the following
continuity adjustment:

\(10\)
$n_{a} = \frac{n_{u}}{4}\left( 1 + \sqrt{1 + 4n_{u}\delta} \right)^{2}$

### Approach to managing unusable data

Maine DEP\'s data collection approach is expected to minimize the
incidence of unusable data with regard to facility audits. The facility
audits will be conducted by a trained Maine DEP professional
knowledgeable about the intent of the questions. This should minimize
item non-response, a primary potential source of unusable data. In the
event that facilities do not have information on hand during the audit
visit, DEP will follow-up with the facility at a later date to collect
information. Unusable data due to illegibility will also be minimized
because the DEP auditor will be involved in inputting the audit data
into an electronic format. In some instances, unusable data may occur if
a facility refuses to provide information to help answer a question.

In the event of unusable data for a particular facility on a particular
question, the record will be flagged to indicate non-response (and
differentiated from records for which a particular question was not
relevant). Further, the reason for non-response will be noted for later
analysis if necessary. Finally, the record will be dropped from the
analysis of that question: i.e., the sample size for that question will
be reduced. Doing so will make it more difficult to hit precision
targets for a particular question.

In the event that the amount of unusable data for a particular question
is larger than 5%, DEP will report the percentage, along with
conclusions regarding whether the incidence of unusable data creates a
biased data set and therefore reduces data representativeness. For
example, the data could be considered biased if unusable data were
substantially more likely to come from noncompliant facilities than from
compliant facilities.

[^1]: Includes data from self-certification forms, return-to-compliance
    forms, and non-applicability forms.