**Minnesota Pollution Control Agency**

**Environmental Results Program Applied to Feedlots**

**Quality Assurance Program Plan**

**Minnesota Pollution Control Agency**

**Division of Technology, Assistance and Education**

**520 Lafayette Road**

**St. Paul, MN 55155**

**Project Manager Contact Information:**

**Kate Brigman**

**Minnesota Pollution Control Agency**

**Regional Division**

**1230 South Victory Drive**

**Mankato, MN 56001**

**Voice 507-389-1775**

**Fax 507-389-5422**

**kate.brigman@pca.state.mn.us**

# A PROJECT MANAGEMENT

## A1. Approval Sheet

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Kate Brigman Date

MPCA, Regional Division

Project Manager

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Luke Charpentier Date

MPCA, Environmental Analysis and Outcomes Division

Quality Assurance Officer

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

Bob Finley Date

MPCA, Regional Division

Feedlot Programs Manager

Principal Investigator

## A2. Table of Contents

[A PROJECT MANAGEMENT 2](#a-project-management)

[A1. Approval Sheet 2](#a1.-approval-sheet)

[A2. Table of Contents 3](#a2.-table-of-contents)

[A3. Distribution List 4](#__RefHeading___Toc75848716)

[A4. Project/Task Organization 4](#a4.-projecttask-organization)

[A5. Problem Definition/Background 6](#section-1)

[Rationale for initiating the project.
6](#rationale-for-initiating-the-project.)

[Objectives of the project. 6](#objectives-of-the-project.)

[Regulatory information, applicable criteria and action limits.
7](#regulatory-information-applicable-criteria-and-action-limits.)

[A6. Project/Task Description 7](#a6.-projecttask-description)

[Project overview. 7](#project-overview.)

[Project summary and work schedule.
8](#project-summary-and-work-schedule.)

[Geographic focus. 8](#geographic-focus.)

[Resource and time constraints. 8](#resource-and-time-constraints.)

[A7. Quality Objectives and Criteria
8](#a7.-quality-objectives-and-criteria)

[Detailed performance measures. 8](#detailed-performance-measures.)

[Quality objectives. 11](#data-quality-objectives.)

[A8. Special Training/Certification
12](#a8.-special-trainingcertification)

[A9. Documents and Records 13](#a9.-documents-and-records)

[Report format/information. 13](#report-formatinformation.)

[Document/record control. 13](#documentrecord-control.)

[Other records/documents.
13](#other-recordsdocuments-maintained-by-the-project-manager.)

[Storage of project information. 13](#storage-of-project-information.)

[Backup of electronic files. 13](#backup-of-electronic-files.)

[B DATA GENERATION AND ACQUISITION
13](#b-data-generation-and-acquisition)

[B1. Sampling Process Design (Experimental Design)
14](#b1.-sampling-process-design-experimental-design)

[B2. Sampling Methods 14](#b2.-sampling-methods)

[Preparation of data collection instruments.
15](#preparation-of-data-collection-instruments.)

[B3. Sample Handling and Custody 15](#b3.-sample-handling-and-custody)

[Data entry QA procedures. 15](#data-entry-qa-procedures.)

[B4. Analytical Methods 15](#b4.-analytical-methods)

[B5. Quality Control 15](#b5.-quality-control)

[Crosschecking data. 15](#crosschecking-data.)

[Data anomalies. 16](#data-anomalies.)

[Quality control statistics. 16](#quality-control-statistics.)

[B6. Instrument/Equipment Testing, Inspection and Maintenance
16](#b6.-instrumentequipment-testing-inspection-and-maintenance)

[B7. Instrument/Equipment Calibration and Frequency
16](#b7.-instrumentequipment-calibration-and-frequency)

[B8. Inspection/Acceptance for Supplies and Consumables
16](#b8.-inspectionacceptance-for-supplies-and-consumables)

[B9. Non-Direct Measurements (I.e., Secondary Data)
16](#b9.-non-direct-measurements-i.e.-secondary-data)

[Key resources/support facilities needed.
17](#key-resourcessupport-facilities-needed.)

[Determining limits to validity and operating conditions.
17](#determining-limits-to-validity-and-operating-conditions.)

[B10. Data Management 17](#b10.-data-management)

[C ASSESSMENT/OVERSIGHT 17](#c-assessmentoversight)

[C1. Assessment and Response Actions
17](#c1.-assessment-and-response-actions)

[C2. Reports to Management 18](#c2.-reports-to-management)

[D DATA REVIEW AND EVALUATION 18](#d-data-review-and-evaluation)

[D1. Data Review, Verification and Validation
18](#d1.-data-review-verification-and-validation)

[D2. Verification and Validation Methods
19](#d2.-verification-and-validation-methods)

[D3. Evaluating Data in Terms of User Needs
19](#d3.-evaluating-data-in-terms-of-user-needs)

[Meeting and reporting needs of your project.
19](#meeting-and-reporting-the-needs-of-the-project.)

[Mathematical and statistical formulae.
19](#mathematical-and-statistical-formulae.)

[Approach to managing unusable data.
19](#approach-to-managing-unusable-data.)

[]{#__RefHeading___Toc75848716 .anchor}**A3. Distribution List**

Each person listed on the approval sheet and each person listed under
Project/Task Organization will receive a copy of this Quality Assurance
Project Plan (QAPP). Individuals taking part in the project may request
additional copies of the QAPP from personnel listed under Section A4.

This document has been prepared according to the United States
Environmental Protection Agency publication *EPA Requirements for
Quality Assurance Project Plans* dated March 2001 (QA/R-5).

## A4. Project/Task Organization

Personnel involved in project implementation are listed in Table 1, and
shown as an organization chart in Figure 1.

  ----------------------- -------------------- ---------------------------
  **Table 1: Project                           
  Implementation                               
  Personnel**                                  

  **Individual**          **Role in Project**  **Organizational
                                               Affiliation**

  Kate Brigman            Project Manager      MPCA Regional Division

  Luke Charpentier        QA Manager/Officer   MPCA Environmental Analysis
                                               and Outcomes Division

  Bob Finley              Principal            MPCA Regional Division
                          Investigator         

  Al Innes                EPA Agreement        MPCA Technology, Education,
                          Manager              and Assistance Division

  Don Adams               Partner              Stearns County Feedlot
                                               Program

  Mark Gernes             Partner              Winona County Feedlot
                                               Program

  Bob Lefebvre            Partner              MN Milk Producers
                          Representative       Association
  ----------------------- -------------------- ---------------------------

The MPCA Project Manager will be responsible for the following
activities:

-   Conduct outreach with feedlots and internal/external stakeholders;

-   Identify project participants;

-   Develop audit and inspection procedures;

-   With partners, develop compliance/P2 assistance documents in
    consultation with stakeholders;

-   Develop enforcement policy and procedures for ERP participating
    feedlots in consultation with stakeholders;

-   Utilize regulatory flexibility;

-   Maintain official, approved QAPP;

-   Overall QA for the project.

The MPCA Principal Investigator will be responsible for the following
activities:

-   Evaluate grant for work done to meet outcome accountability and
    ensures that resources are dedicated to work commitments;

-   Communicate with relevant MPCA Supervisors to limit competing
    resources (give time to closeouts, final reports, etc.)

-   Act as primary EPA contact for big picture program-area issues

-   Ensure program and fiscal requirements being met

The MPCA EPA Agreement Manager will be responsible for the following
activities:

-   Produce amended QAPP, as necessary;

-   Issue quarterly and annual reports to U.S. EPA with support of
    Project Manager;

-   Support project in other ways as necessary and available.

Contractor (to be determined) will be responsible for the following
activities:

-   Assist MPCA in developing a statistically sound measurement
    strategy.

# 

The partner association of regulated facilities, (Minnesota Milk
Producers Association - MMPA) through its representative above, will be
responsible for the following activities:

-   Assist in the identification of the facility universe;

-   Review of compliance/P2 assistance materials;

-   Marketing;

-   Assist with workshops and distribution of compliance/P2 assistance
    materials;

-   On-farm assistance in support of the self-certification process.

Counties anticipated to be partnering will be responsible for:

-   Providing the appropriate staff for training and inspections;

-   Assisting MPCA in securing host facilities for trainings and
    educational workshops;

-   Participating in stakeholder meetings and in development of
    materials, deliverables, and reports.

The participating facilities will be responsible for submitting
self-certification materials and, if applicable, returning to
compliance.

**Figure 1: Project Organizational Chart**

##  

## A5. Problem Definition/Background

The MPCA has the responsibility to regulate the collection,
transportation, storage, processing, and land application of animal
manure for the prevention and abatement of water, air, and land
pollution. As of August 2004, MPCA estimates that over 30,000 regulated
animal feedlot and manure storage sites exist in Minnesota. Those sites
regulated are feedlots over 10 animal units (AU) if within shoreland and
over 50 AU if outside shoreland. Regulated storage areas store manure
for 100 or more animal units. Minnesota also regulates the rate,
location, and timing of manure application to soil. Current data shows
that 11,460 feedlots are located within 1000 feet of surface water (an
approximation of "within shoreland"). Beyond the issues present at the
animal holding area or manure storage site, manure application
activities have the potential to release nutrients into surface and
ground water supplies. It is important to prevent the contaminants in
manure from moving from the animal holding areas and manure storage
areas as well as from land application areas.

In manure, the constituents most impacting water quality include
phosphorus, nitrogen, biological oxygen demand, and disease-causing
organisms (pathogens). Human health and the environment are put at risk
from these water quality impact factors. The problems caused by the
contaminants or the results of environmental contamination have
different pathways of entry and source areas. Various types of gaseous
compounds emanating from manure are an additional human health and
environmental concern.

Watershed projects conducted through the Minnesota Clean Water
Partnership Program have diagnosed water quality problems in 37 project
sites throughout the state. Sixteen projects identified feedlots as
significant contributors of contaminants to lakes and streams. While the
statewide effects of contaminants from manure have not been completely
separated from other sources of pollution, it is clear that surface
water quality is being impacted from agricultural sources in general,
which includes discharges and runoff from feedlots and manure
application sites.

To address these issues, MPCA implemented its updated feedlot rules in
October 2000, which included a requirement that owners register their
feedlots and manure storage areas. Over 30,000 sites are now registered.
Of the registered sites, approximately 600 are currently regulated
through NPDES permits; an additional estimated 200 will require NPDES
permits under EPA's new CAFO rules; the remaining sites are regulated
through state rules and permits. In addition, the MPCA has focused its
staff on inspection and compliance. Even with that focus, however, at
existing rates of inspection (given tight state and county budgets), the
estimated time to reach all state feedlots would be greater than 10
years.

### Rationale for initiating the project. 

This project will test and implement an Environmental Results Program
(ERP) approach modeled by other states for feedlots that fall below the
federal definition (roughly 30,000 entities in Minnesota), seeking
improved environmental performance and compliance. The EPA funding
covered by this QAPP will be used to develop and implement a pilot ERP
within a segment of the feedlot population: dairies.

Self-certification, assistance in partnership with the Minnesota Milk
Producers Association, and other ERP tools will help feedlots improve
environmental performance and compliance in a more timely way, and
without the typical level of MPCA and county effort. Partnership with
the producer association, counties, the Board of Water and Soil
Resources, Soil and Water Conservation Districts and Clean Water
Partnerships will aid development and distribution of the program. In
the long run, it is anticipated that statistical approaches will help
MPCA draw conclusions about compliance rates in dairy and possibly other
feedlot sectors, and will guide targeting of compliance assistance and
inspections.

The project also aims to extend the basic water quality protection and
land application management aspects of the feedlot rule to deal with
odors, dusts, pests, ancillary and maintenance operations, related feed
crop production, and many other aspects of operations, pushing toward a
multi-media, "whole-farm" management system.

### Objectives of the project. 

MPCA's goals and objectives for the feedlot ERP project are outlined
below.

**GOAL 1**: Understand and communicate the universe of state and federal
feedlots in Minnesota and their compliance/performance vectors over time

[OBJECTIVE 1]{.underline}: Maximize quality and completeness of MPCA's
feedlot database

[OBJECTIVE 2]{.underline}: Develop hard copy and electronic
self-certification annual submittal system and integrate inspection data

**GOAL 2**: Maximize leverage through partnerships

[OBJECTIVE 1]{.underline}: Strengthen producer (association)
partnerships through project planning and implementation

**GOAL 3**: Test the ERP methodology in the field to determine
effectiveness of self-certification process

[OBJECTIVE 1]{.underline}: Determine baseline conditions

[OBJECTIVE 2]{.underline}: Operate facility self-certification cycle

[OBJECTIVE 3]{.underline}: Follow-up inspections/assessments with all
baseline facilities to gauge effectiveness of process, program materials
and services in attaining compliance and multi-media performance goals

[OBJECTIVE 4]{.underline}: Explore the following research questions:

-   Does self-certification with Minnesota Milk Producers Association
    (MMPA) or state or county assistance yield equal, better, or earlier
    compliance results and better overall performance than that
    accomplished by dairies that do not undergo a self-certification
    process? What are the relative costs?

-   Does self-certification without significant MMPA or state or county
    assistance yield equal, better, or earlier compliance results and
    better overall performance than that accomplished by dairies that do
    not undergo a self-certification process? What are the relative
    costs?

-   Does self-certification with MMPA or state or county assistance
    yield equal, better, or earlier compliance results and better
    overall performance than that accomplished under the state interim
    permit or Open Lot Agreement programs? What are the relative costs?

Interim permits, issued by counties and the MPCA, are essentially
return-to-compliance schedules of 24-27 months for non-CAFO feedlots
(larger than 300 AU) inspected and found to be out of compliance. Open
Lot Agreements allow feedlots under 300 AU to attain 100% compliance by
2010.

### Regulatory information, applicable criteria and action limits. 

All feedlots are subject to state rule requirements for controlling or
preventing air, surface and ground water contamination. The feedlot
operators are required to register with the MPCA once every four years
in order to operate. MPCA will leverage the registration and open lot
certification process and partner with the MMPA to recruit participation
in this voluntary program.

## A6. Project/Task Description

### Project overview. 

This project will allow MPCA to explore whether an approach modeled upon
the Environmental Results Program (ERP) can help achieve these goals,
while improving regulatory cost-effectiveness. ERP is an innovative
approach to solving high-priority environmental problems in industry
sectors largely comprised of small businesses or operations. The ERP
concept combines technical assistance, self-certification, inspections,
and statistically based performance measurement in order to reduce
environmental impacts of business.

The promise of ERP is that it will cost-effectively reduce environmental
impacts of small businesses that may present a substantial cumulative
environmental risk. Businesses targeted so far by ERP include gas
stations, auto salvage yards, auto body and mechanical repair shops, dry
cleaners, and printers. ERP can help environmental agencies identify
previously unknown facilities, measure performance, increase regulatory
efficiency, and help improve overall environmental performance. ERP is
in part designed to help facilities that want to comply but don't
understand their requirements, and evidence suggests that ERP can
motivate operators to comprehensively review their environmental
performance and take needed action to come into compliance and adopt
best practices.

### Project summary and work schedule. 

The main tasks and timeline are outlined in Table 2 below. The project
period is 1/1/05 -- 12/31/06.

+------------------------------------------------+----------+-----+----+
| **Table 2: Tasks and Timeline**                |          |     |    |
+------------------------------------------------+----------+-----+----+
| **Task**                                       | *        | *   |    |
|                                                | *Start** | *En |    |
|                                                |          | d** |    |
+------------------------------------------------+----------+-----+----+
| (a) Operate and maintain stakeholder process   | Mon      | Fri |    |
|                                                | 8/9/04   | 12  |    |
|                                                |          | /29 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+
| (b) Develop EPA cooperative agreement          | Thu      | Mon |    |
|                                                | 6/17/04  | 2   |    |
|                                                |          | /28 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (c) Contract(s) for support                    | Wed      | Mon |    |
|                                                | 9/1/04   | 1/3 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (d) Develop facility ERP database              | Tue      | Mon |    |
|                                                | 9/1/04   | 6   |    |
|                                                |          | /20 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (e) Develop electronic interchange (if         | Mon      | Fri |    |
|     feasible)                                  | 9/6/04   | 7   |    |
|                                                |          | /28 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+
| (f) Develop statistical design/sampling (amend | Wed      | Mon |    |
|     QAPP)                                      | 9/1/04   | 6   |    |
|                                                |          | /20 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (g) Inspector checklist                        | Mon      | Fri |    |
|                                                | 9/1/04   | 4   |    |
|                                                |          | /29 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (h) ERP outreach/participation                 | Mon      | Fri |    |
|                                                | 10/18/04 | 4   |    |
|                                                |          | /29 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (i) Training inspectors                        | Tue      | Tue |    |
|                                                | 6/21/05  | 7   |    |
|                                                |          | /12 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (j) Baseline inspections and analysis          | Wed      | Tue |    |
|     (includes most data entry)                 | 7/13/05  | 11  |    |
|                                                |          | /15 |    |
|                                                |          | /05 |    |
+------------------------------------------------+----------+-----+----+
| (k) Assistance                                 | Tue      | Wed |    |
|                                                | 12/21/04 | 4   |    |
|                                                |          | /19 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+
| (l) Certification (mailed after baseline       | Tue      | Wed |    |
|     inspections in (j) are completed           | 8/23/05  | 5   |    |
|                                                |          | /17 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+
| (m) Post certification inspections             | Wed      | Mon |    |
|                                                | 5/3/06   | 10  |    |
|                                                |          | /16 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+
| (n) Reporting                                  | Thu      | Fri |    |
|                                                | 12/2/04  | 12  |    |
|                                                |          | /29 |    |
|                                                |          | /06 |    |
+------------------------------------------------+----------+-----+----+

### Geographic focus. 

Initially, MPCA will pilot the ERP approach in 2 dairy-rich counties
along the Mississippi River: Stearns in central Minnesota and Winona in
southeastern Minnesota. Depending on results, MPCA may then extend the
experiment statewide or test the approach in other sectors such as swine
or turkeys (or both).

### Resource and time constraints. 

[Coordination and development requirements of the database and
electronic reporting capabilities are unknown until a full analysis of
the current status of the program is evaluated. The evaluation will be
performed by an external contractor or internal IS professionals who
will then provide a cost estimate based upon the results of the
evaluation. *This section of the QAPP will be amended upon completion of
the evaluation.*]{.mark}

## A7. Quality Objectives and Criteria

### Detailed performance measures. 

This project is primarily interested in the following list of possible
performance measures for each task. Note that task #6 of this project
involves revisiting and reaffirming/revising these draft performance
measures. The final list will be submitted in a QAPP amendment in June
2005.

1)  Stakeholder Process Measures:

    a.  Partners understand project goals -- indicated by consensus and
        satisfaction coming out of external partner meetings.

    b.  Partners' willingness to market on MPCA's behalf -- indicated by
        consensus and partners delivering the product and level of
        effort identified in marketing strategy.

2)  EPA Cooperative Agreement Measures:

    a.  Completed at MPCA by 9/3/04

    b.  Executed by EPA by 12/31/04

    c.  Final project workplan by 2/28/05

3)  Contract for Support Measures:

    a.  On-time execution (1/3/05)

4)  Facility ERP Database Measures:

    a.  Universe of dairies well-documented statewide and in
        participating counties -- indicated by number of dairies missed
        and later located by county feedlot officers (CFOs) or MPCA
        staff.

    b.  MPCA users able to input, use, analyze -- indicated by training
        post-test, longer-term satisfaction, and documented error rate.

5)  Electronic Interchange Measures:

    a.  Replicate stormwater interchange at lower cost -- time to
        completion versus stormwater or other web exchange projects
        (from time-tracking records)

    b.  Dairies able and willing to use electronic data interchange --
        (short-term) evaluations from user workshop/beta test,
        (long-term) electronic versus paper submittals

    c.  Decreased paper use and cost of handling -- based on average
        sheets/envelopes/mailing and mailing costs

6)  Statistical Design/Sampling Measures:

    a.  Study design maximizes insights into research questions (see
        Objective #4 under Goal 3), within policy and resource
        constraints.

    b.  Feasibility overlay -- representative samples can be processed
        without drawing significantly on program staff not already
        assigned to the project.

7)  Inspector Checklist Measures:

    a.  Combines multiple programs and issues in maximum half-day visit
        -- results of dry runs

    b.  Inspectors & technicians accept the checklist -- attitudinal
        survey following completion

    c.  Checklist helps communicate project issues to dairies -- survey
        of dry run dairies

8)  ERP Outreach Measures:

    a.  Number of mailings (minus number of returns)

    b.  Web site hits

    c.  Requests for further information (either MPCA or MMPA)

    d.  Number of brochures and information requests managed by MN Dept.
        of Agriculture dairy inspectors

    e.  Sign-ups

    f.  MMPA membership increased

9)  Training Inspectors/Technicians Measures:

    a.  Number of county and MPCA staff, MMPA technicians trained

    b.  Hours in development - timetracking records

    c.  Inspector satisfaction -- training evaluations

    d.  MMPA techs work effectively with dairies -- MPCA survey calls to
        dairies

    e.  MPCA, county inspectors' effectiveness in working with dairies
        -- MPCA (third-party) calls to dairies

    f.  MMPA technician performance (complete checklists, operator
        satisfaction) -- MPCA survey calls to dairies

    g.  High-quality data collection (longer-term) -- accuracy of
        certifications, data reported

10) Baseline Inspections Measures:

    a.  Time investment relative to standard dairy inspections -- from
        timetracking records, relative to historical average for this
        type of dairy

    b.  Inspectors able (not able) to cover material projected in
        checklist -- inspector records

    c.  Response of operators during inspectors -- qualitative, from
        inspector records

    d.  Number of operators requesting assistance from MMPA technicians
        during inspections -- inspector records

    e.  Compliance rates -- inspector records

    f.  Best management practices (BMPs) in place -- inspector records

    g.  Data sufficient to run baseline FLEvals

    h.  Number of enforcement actions required (after any amnesty
        period) -- longer term

    i.  Percentage of facilities that reregister every year -- long term

11) Assistance Measures:

    a.  Workshop attendance -- operators in attendance

    b.  Workbooks distributed

    c.  Technician requests and deliveries -- MMPA records

    d.  Number of facilities assisted vs. self-certified -- MMPA records

    e.  Accuracy of self-certification vs. technician-assisted --
        longer-term, involves data from post-certification inspections
        (requires flags in database)

    f.  Number (percentage) of facilities claiming to have used
        assistance materials, but did not certify -- MPCA survey of
        those not completing certification, others

    g.  Environmental gains (using the same measurement methods as #13
        below) by producers receiving assistance but not certifying

12) Certification Measures:

    a.  Time spent in managing data electronically vs. what it would
        have been had the entire process been manual -- from
        time-tracking records, based on data submittals similar to
        certifications.

    b.  Number (percentage) of baseline operations completing
        certification

    c.  Number of return to compliance submittals

    d.  Number reporting compliance vs. non-compliance

    e.  Overall accuracy of certification forms -- (longer-term) waits
        for post-certification inspector records

    f.  Return to compliance forms submitted and return to compliance
        (RTC) plans implemented -- requires post-certification or other
        inspection records

13) Post-Certification Inspections Measures

    a.  Number of inspections completed vs. number needed for sample

    b.  Time elapsed to complete all planned inspections

    c.  Time/cost per post-cert inspection vs. standard and vs. baseline
        -- timetracking records

    d.  Accuracy of self-certifications --certification forms and
        post-certification inspector records

    e.  Consideration of (progress on) whole farm issues -- inspection
        records and longer-term MPCA followup

    f.  Number (percentage) of farmers adopting recommended nutrient
        management practices -- inspection records and longer-term MPCA
        followup

    g.  Performance of certifying facilities vs. others (including
        non-certifying facilities) -- longer-term (site visits)

    h.  (Increased) use of soil testing and other site assessment tools
        -- longer-term (site visits)

    i.  Use of P2 and BMPs reported -- annual reports, baseline and
        post\--certification inspection or survey data

    j.  Number (percentage) of farmers with approved manure management
        plans -- followup submittals, inspections

    k.  Estimated pollutant reductions based on Feedlot Evaluation
        (FLEval) and Board of Water and Soil Resources (BWSR) models

    l.  Any of the possible long-term measures under #15 which have been
        quickly implemented

14) Reporting Measures

    a.  Reports delivered on time

    b.  Interest from other states -- program staff phone logs and
        e-mail records

    c.  Interest from other MPCA program managers -- program records
        (staff training)

    d.  Interest from other livestock sectors

    e.  Adaptation by other states (long term) -- EPA records

15) Possible Long-term Measures:

    a.  Changes in surface water ambient conditions

    b.  Diversions of pollutant runoff based on modeling

    c.  Activity measures (e.g. structures receiving milkhouse wastes,
        open lots addressed and how, surface tile inlets closed, etc.)

    d.  Environmental gains not directly associated with feedlot
        management, such as solid waste, chemical storage, habit
        restoration, etc.

MPCA has discussed long-term measurement of project impacts. The costs
and difficulties of such efforts are well-known. Nevertheless, MPCA
commits to continue exploration of feasible means of measuring the
project's environmental impacts within the project period, with the goal
of producing a plan for long-term measurement, which can be feasibly
implemented, either within or after the project period. Some of the
possible tools or approaches that MPCA will consider in establishing its
long-term results measurement plan are:

-   Report form for 50 participants to insert in existing annual report
    or submit separately on which they indicate progress towards best
    management practices (BMPs) and needed corrections;

-   Phone survey of 20-50 participants - product: list of BMPs
    implemented, compliance correction;

-   On-farm survey (not an inspection) of 20-50 participants - product:
    list of BMPs implemented, compliance correction, information to
    complete FLEval;

-   Follow-up inspections of a sample or of all 50 participants after 2
    years (or other meaningful interval), looking at all or most of the
    vectors in the baseline and post-cert inspections -- product: same
    as #2 but direct verification plus credit for a formal inspection;

-   Work with state and local water monitoring staff to determine if
    existing sampling sites can be used to draw reliable conclusions
    about project impacts on water quality parameters;

-   Work with state and local water monitoring staff to determine if in
    the project timeline the partners could set up any new sampling
    sites, particularly in priority TMDL watersheds or if a number of
    participants are conveniently clustered. If so, then project staff
    could work with monitoring experts to determine proper site setup
    for data collection. Other sources of support for the monitoring or
    for the participating site's implementation would be necessary in
    order to pursue this option.

### Data Quality Objectives. 

[These measures as a whole will help MPCA determine whether to expand
this pilot ERP approach beyond the initial participants and, if so, how
to modify the approach. Quality objectives for these performance
measures will be developed as part of the Stakeholder Process and
Statistical Design tasks. Specific quality objectives for these measures
as a group (and, if necessary, individually) will be developed
considering EPA QA/G-4
\<<http://www.epa.gov/quality/qs-docs/g4-final.pdf>\>, and will then be
provided in the amendment to the QAPP.]{.mark}

The amendment to the QAPP will ensure that the quality objectives for
these performance measures are appropriate for the regulatory and
non-regulatory decisions to be made based upon those measures. This
determination will take into account both the best practices for similar
projects and the resources available for this project. In part, the
Project Manager will rely upon EPA\'s *Generic Guide to Statistical
Aspects of Developing an Environmental Results Program* (2003) for
advice in making decisions related to the optimizing the following
aspects of data quality for this project:

-   Bias

-   Representativeness

-   Completeness

-   Comparability

-   Sensitivity (if applicable)

[MPCA will set numeric objectives which will include specific
improvements over specific timeframes, for example, X% improvement in
compliance and overall environmental performance in Y years. Progress
towards these objectives will inform MPCA's decision whether to extend
the pilot to other feedlot sectors, or whether to consider making ERP
mandatory. Data availability and quality must be sufficient to support
such objectives. Further, MPCA understands that the participant sample
size is unlikely to be large enough to support conclusions based on
smaller increments of improvement. The QAPP amendment will address
specific data quality objectives and issues of data availability and
quality.]{.mark}

## 

## A8. Special Training/Certification

MPCA will develop and deliver mandatory and voluntary training sessions
to key parties to ensure quality data collection, to the extent
practicable.

Mandatory intensive in-person training sessions will be delivered to the
following individuals to ensure quality data collection:

-   Inspectors who will be collecting baseline and post-certification
    data;

-   Data-entry personnel who will be processing data from inspections
    and self-certification responses;

-   The project manager who will be cross-checking data and any
    additional QA/QC personnel (if any additional training is needed to
    familiarize them with the project);

-   Individuals who will be compiling the database containing the
    universe of facilities.

Each session will cover proper data collection and QA procedures.
Training will be augmented by debriefing personnel shortly after their
tasks have begun, to correct and clarify appropriate practices.

Voluntary intensive in-person training sessions will be offered to the
self-certifying facilities. Facilities will also be provided with clear
written instructions on how to prepare and submit data.

The Project Manager is responsible for ensuring that all personnel
involved with data generation (including state personnel, contractors,
and partners) have the necessary QA training to successfully complete
their tasks and functions. The Project Manager will document attendance
at all training sessions.

The Project Manager is also responsible for ensuring the
self-certification materials sent to facilities clearly document how
facilities should properly prepare and submit their data.

## A9. Documents and Records

[]{.mark}

### Report format/information. 

The format for all data reporting packages will be consistent with the
requirements and procedures used for data validation and data assessment
described in this QAPP.

### Document/record control.

The recording media for the project will be both paper and electronic.
The project will implement proper document control procedures for both.
For instance, hand-recorded data records will be taken with indelible
ink, and changes to such data records will be made by drawing a single
line through the error with an initial by the responsible person. The
Project Manager will have ultimate responsibility for any and all
changes to records and documents. Similar controls will be put in place
for electronic records.

The MPCA Quality Assurance Officer shall retain all updated versions of
the QAPP and be responsible for distribution of the current version of
the QAPP. The MPCA Quality Assurance Officer and the MPCA Project
Manager will approve annual updates. The Project Manager shall retain
copies of all management reports, memoranda, and all correspondence
between the MPCA and all project personnel identified in A4.

### Other records/documents maintained by the Project Manager. 

Other records and documents that will be produced in conjunction with
this project include:

-   Inspection checklists and reports

-   Self-certification forms

-   Return-to-compliance forms

-   Non-applicability forms

-   Enforcement documentation

-   Facility outreach materials, including workbook, fact sheets,
    brochures, etc.

-   Amended QAPP

-   Readiness reviews (see below)

-   Data handling reports

-   Quarterly and annual progress reports to EPA

-   Project final report (to include discussion of QA issues
    encountered, and how they were resolved)

-   Photographs

-   Data and reports from third parties (facilities, consultants,
    laboratories)

### Storage of project information. 

[After the completion of the project period (12/31/06), project
information will be stored in the MPCA archives for a period to be
determined in the QAPP amendment in accordance with Feedlot Program
standards and/or EPA standards.]{.mark}

### Backup of electronic files. 

[Project information contained in electronic files will be backed up on
disk and stored at off-site MPCA archives as well as at MPCA
offices.]{.mark}

# B DATA GENERATION AND ACQUISITION

## B1. Sampling Process Design (Experimental Design)

A key task in this project will be to develop a sound statistical
methodology for collecting and analyzing facility data, in order to draw
inferences related to the selected performance measures. The major
quality objective will be to collect representative data that truly
reflect the conditions of the universe of facilities upon which this ERP
focuses. Facility data is of four possible types:

1.  Inspection data, which will be collected by trained MPCA and county
    inspectors from sample facilities;

2.  Self-certification data[^1], which will be collected from facilities
    through a mail or electronic-submittal survey process. Participating
    facilities are expected (but not required) to respond. This suggests
    that project results will indicate the self-certification accuracy
    of those motivated to volunteer (versus the dairy population in
    general), but will not yield definitive data on how compulsory
    self-certifications would work. While the precise methods are not
    known at this point, they are expected to be built upon the advice
    given in EPA's *Generic Guide to Statistical Aspects of Developing
    and Environmental Results Program* (2003);

3.  Survey data, collected by MPCA or county staff either by phone or
    on-site surveys, or through inspections. The purpose of this data
    collection will be to determine what implementation of compliance
    measures or other P2 improvements have taken place and to get enough
    indirect (if by phone or otherwise self-reported by facilities) or
    direct (if on-farm) site-specific to run accurate FLEval models as
    to reduced pollutant impact;

4.  Modeled data from FLEval or BWSR pollutant runoff models specific to
    participating sites. The final QAPP will indicate how validation of
    the FLEval and BWSR models has taken place in the past, and whether
    further validation will be necessary as part of this project.

A fifth data type relates to the possibility of using existing or new
sampling programs to judge (in the long-term, after the project's end
date) the effect of a limited number of participants' site improvements
on water or soil quality:

5.  Surface water or groundwater samples collected as determined to be
    feasible in the course of the project. If such sampling is
    determined to be feasible, experimental and sampling design will be
    developed during the project in accordance with MPCA's existing QAPP
    for such sampling programs and with the MPCA's overall Quality
    Management Plan. In addition, MPCA's review of Nutrient Management
    Plans will include review of the results of soil testing the
    producer is required to conduct.

[This section of the QAPP will be amended upon completion of the
project-specific statistical methodology (task #6, section A7).]{.mark}

## B2. Sampling Methods

As described above, the primary data collected and used by this ERP will
come from a survey data collection process. Data will be collected from
3 distinct groups: (1) operations that volunteer to participate in the
MPCA ERP self-certification process; (2) operations that have , of their
own accord, participated in the MMPA\'s five-star certification program;
and (3) a control group consisting of operations similar to the kinds of
operations in group #1, but which will not participate in
self-certification. This section of the QAPP will be amended upon
completion of the project-specific statistical methodology, which will
detail the measurement approach to be used. The measurement approach
cannot be finalized until the number and nature of volunteer facilities
is known; that information will be established as one of the
implementation tasks before the development of the statistical
methodology. As mentioned elsewhere, that methodology will be prepared
consistent with the principles identified in the EPA\'s *Generic Guide
to Statistical Aspects of Developing an Environmental Results Program*
(2003), and standard statistical references.

### Preparation of data collection instruments. 

All data collection instruments will be subject to multiple rounds of
review by relevant internal and external stakeholders to help assure the
collection of high-quality and representative data. Data collection
instruments will be prepared in accordance with the guidance on data
collection instruments provided in EPA\'s *Generic Guide to Statistical
Aspects of Developing an Environmental Results Program* (2003).
Specifically, preparation will follow the checklist for data collection
instruments provided in an appendix of that guide.

## B3. Sample Handling and Custody

Upon completion of paper checklists, inspectors will sign the
checklists. Phone or on-farm survey staff will also use worksheets
provided for their followup data collection. Inspectors and survey staff
will enter data from paper checklists into the electronic database.
Inspectors and survey staff will be issued procedural guidelines for
data input. Facilities will mail or submit electronically signed forms
into MPCA, where data-entry staff will input hard copy data into the
electronic database.

Chain of custody is relevant to this project only if long-term water
and/or soil sampling programs are feasible enough to incorporate. Chain
of custody, when used, will be maintained by the sampler. The sampler
will keep samples in their custody until the samples are shipped or
turned over to the lab. Upon receipt the lab will verify that samples
match the chain of custody, that temperature and holding time criteria
are met, and then process the samples. [Reference applicable lab QA
manual.]{.mark}

### 

### Data entry QA procedures.

[Procedures for entering hand-written data into the database will follow
standard quality assurance procedures (peer review 100% of data and
overview 10% by the Project Manager.) Detailed quality assurance
procedures for data entry and acceptance will be prepared during the
development and implementation of a data management strategy, including
mirrored data entry training and QA procedures if both MPCA and county
staff are entering data. The final QAPP will reflect the
strategy.]{.mark}

## B4. Analytical Methods

This project will follow well-recognized statistical analytical methods
for survey samples. This section will be amended upon completion of the
detailed statistical methodology. Physical tests or chemical analyses
are anticipated for this project only if long-term water and/or soil
sampling programs are feasible enough to incorporate. If this is the
case, analytical methods used by MPCA and any participating analytical
laboratories will conform with those established in existing QAPPs for
relevant programs, and in the agency's overall Quality Management Plan.
The samples will be run by Minnesota Department of Health
(MDH)-certified labs only for approved EPA or Standard Methods
applicable. [Specific standard operating procedures will be submitted
with QAPP updates for analytical methods used.]{.mark}

## B5. Quality Control

Standard MPCA quality control requirements will be followed for all
analytical methods which includes 5% duplicate rate, 10% spike rate,
check standards and calibration requirements per the analytical method
used. See MPCA QAPP guidance and lab checklist at
[www.pca.state.mn.us/programs/qa_p.html](http://www.pca.state.mn.us/programs/qa_p.html)
for further information. This project will undertake the following
specific steps to measure/estimate the effect of data errors.

### Crosschecking data.

Primary data collection forms will be designed in such a way to allow
internal crosschecking of data by comparing answers of different
questions to each other, and such crosschecking will be automatic for
electronically entered data. Further, post-certification inspections
will offer the opportunity to compare inspection results with
self-certification results, if the facilities sampled have submitted
self-certification forms. [Data collected will be 100% peer reviewed,
and 10% will be overviewed by the Project Manager]{.mark}.

### Data anomalies.

[Procedures for handling data anomalies (such as outliers and missing
data) will be handled based on guidance prepared in the project-specific
statistical methodology (Z-test?).]{.mark}

### Quality control statistics.

The quality control statistics to be used in this project are described
in more detail in section D3.

## B6. Instrument/Equipment Testing, Inspection and Maintenance

[This section is relevant to this project only insofar as long-term
water and/or soil sampling programs are determined to be feasible to
incorporate. If this becomes the case, the MDH lab QA manual and
specific standard operating procedures will be referenced in QAPP
updates as the project develops.]{.mark}

[]{.mark}

## B7. Instrument/Equipment Calibration and Frequency

[This section is relevant to this project only insofar as long-term
water and/or soil sampling programs are determined to be feasible to
incorporate. If this becomes the case, calibration will be carried out
as required by MDH-certified methods. Further information on calibration
procedures will be described in QAPP updates as the project
develops.]{.mark}

[]{.mark}

## B8. Inspection/Acceptance for Supplies and Consumables

[This section is relevant to this project only insofar as long-term
water and/or soil sampling programs are determined to be feasible to
incorporate. If this becomes the case, all supplies and consumable
chemicals and equipment will be inspected and checked for quality by
project staff prior to use. Further detail will be given in QAPP updates
as the project develops.]{.mark}

[]{.mark}

## B9. Non-Direct Measurements (i.e., Secondary Data) 

This project will rely upon secondary data:

-   to identify the facilities in the target population;

-   if existing water monitoring data is used in the project;

-   to model pollutant runoff changes as a result of the project.

  -------------- -------------------- ---------------- ----------------------
  **Table 3:                                           
  Non-Direct                                           
  Measurements                                         
  (I.e.,                                               
  Secondary                                            
  Data)**                                              

  **Data         **Intended Use**     **Rationale for  **Acceptance
  Sources**                           Use**            Criteria**

  MPCA database  Identifying the      Commonly         All records will be
  of facilities  target population,   accepted source  accepted unless sample
                 for the sample       of facility list response indicates
                                                       facility should not be
                                                       part of target
                                                       population. MPCA will
                                                       crosscheck any
                                                       facility that
                                                       self-identifies as
                                                       non-applicable to this
                                                       project.

  MMPA database  Identifying          Only available   Records describing
  of Five-Star   facilities for       database of such facilities will be
  dairies        additional data      facilities       cross-checked and
                 collection, such as                   accepted. MMPA
                 the success of the                    technician-reported
                 existing                              and self-reported data
                 self-certification                    will be accepted if
                 program                               verified at the site
                                                       by county or MPCA
                                                       staff.

  Feedlot        Model changes in     Validated and    The FLEval model is
  Evaluation     pollutant runoff     commonly         currently accepted and
  Model (FLEval) from participating   accepted         used by MPCA. Modeled
                 sites                predictor of     results will be
                                      runoff based on  labeled according to
                                      specific site    the source of the
                                      conditions and   input data: MPCA
                                      practices        staff, county staff,
                                      implemented      or self-reported by
                                                       facilities.
  -------------- -------------------- ---------------- ----------------------

### Key resources/support facilities needed.

MPCA will require access to the data sources mentioned above, and this
information will be managed within the database created/utilized for the
overall project. MPCA does not anticipate any obstacles to this
approach.

### Determining limits to validity and operating conditions. 

Database containing the list of targeted facilities will be designed
such that the original source for all facility data is marked, and
procedures will be in place such that only the Project Manager can
officially remove a facility entry from the target population. In such
cases, facility entry will not be deleted from the database but will be
marked as non-applicable, and corrective data will be provided in fields
parallel to the original data.

## B10. Data Management

As part of this project, MPCA will develop a data management strategy,
and amend the QAPP based upon the strategy. The Project Manager is
responsible for ensuring that that strategy is developed and that the
QAPP is amended to reflect that strategy. Once amended, this QAPP
section on data management will provide information on the following
issues:

-   Data management scheme, from field to final use and storage;

-   Standard recordkeeping and tracking practices, and document control
    system (citing relevant agency documentation);

-   Data handling equipment/procedures that will be used to process,
    compile, analyze, and transmit data reliably and accurately;

-   Individuals responsible for elements of the data management scheme;

-   Process for data archival and retrieval.

# C ASSESSMENT/OVERSIGHT

## C1. Assessment and Response Actions

The Quality Assurance Officer will conduct a Readiness Review with the
Project Manager and their Supervisor immediately prior to the five major
data collection tasks: identifying targeted facilities, baseline
inspections, self-certification, targeted follow-up (if such follow-up
occurs), and post-certification inspections. The group will report
findings to the Section Manager (in this case the Principal
Investigator), who will take corrective action (if any is necessary)
before the data collection task begins. Further, the Project Manager and
QA Officer will thoroughly debrief project implementation staff a short
time after beginning their respective implementation tasks, to identify
emerging/unanticipated problems and take corrective action, if
necessary. This will be done to verify the Data Quality Objectives will
be met by the data being collected and to produce a valid data set that
is as representative as possible of the population sampled.

## C2. Reports to Management

Three kinds of reports will be prepared: readiness reviews (described
above), regular quarterly and annual progress reports, and project final
report. Progress reports will note the status of project activities and
identify whether any QA problems were encountered (and, if so, how they
were handled). The project final report will analyze and interpret data,
present observations, draw conclusions, identify data gaps, and describe
any limitations in the way the data should be used.

**Project QA** Status Reports

  ----------------- ----------------- ----------------- ------------------
  **Type of         **Frequency**     **Preparer**      **Recipients**
  Report**                                              

  Amended QAPP      Once, before      MPCA Project      All recipients of
                    primary data      Manager           original QAPP
                    collection begins                   

  Readiness Review  Before each major MPCA QA Officer   MPCA Principal
                    data collection   and Project       Investigator
                    task              Manager           

  Progress Report   Quarterly         MPCA              EPA Project
                                                        Officer (Copying
                                                        EPA OPEI)

  Progress Report   Annually          MPCA              EPA Project
                                                        Officer (Copying
                                                        EPA OPEI),
                                                        stakeholders

  Final Project     Once              MPCA              EPA Project
  Report                                                Officer (Copying
                                                        EPA OPEI),
                                                        stakeholders
  ----------------- ----------------- ----------------- ------------------

# 

# D DATA REVIEW AND EVALUATION

## D1. Data Review, Verification and Validation

This QAPP shall govern the operation of the project at all times. Each
responsible party listed in Section A4 shall adhere to the procedural
requirements of the QAPP and ensure that subordinate personnel do
likewise.

All data generated for this project will be reviewed by the Project
Manager prior to entry into the database. Where applicable, 10% of all
data will be checked against lab analytical reports to include
applicable quality control parameters. It is anticipated that full data
validation will not be done for this program (per the National
Functional Guidelines).

This QAPP shall be reviewed at least annually to ensure that the project
will achieve all intended purposes. All the responsible persons listed
in Section A4 shall participate in the review of the QAPP. The Project
Manager and the Quality Assurance Officer are responsible for
determining that data are of adequate quality to support this project.
The project will be modified as directed by the Project Manager. The
Project Manager shall be responsible for the implementation of changes
to the project and shall document the effective date of all changes
made.

It is expected that from time to time ongoing and perhaps unexpected
changes will need to be made to the project. The Project Manager shall
authorize all changes or deviations in the operation of the project. Any
significant changes will be noted in the next report to EPA, and shall
be considered an amendment to the QAPP. All verification and validation
methods will be noted in the analysis provided in the final project
report. All written reports will be 100% peer reviewed at MPCA prior to
submission to EPA or release to the public.

[]{.mark}

## D2. Verification and Validation Methods

To confirm that QA/QC steps have been handled in accordance with the
QAPP, a readiness review will be conducted before key data
collection/analysis steps, and data handling reports will be prepared
after each step. Standard statistical tests (described below in Section
D3) will be used to determine the extent to which inferences can be
drawn from the sample data. Specific quality control limits placed upon
analytical data will be verified based upon the applicable EPA or
Standard Method used and accepted laboratory standard operating
procedure. All secondary data will be checked and the source of the data
known if possible. If questionable data is found it will be flagged and
decisions made on data usability. If required, an audit will be done on
labs or facilities.

## D3. Evaluating Data in Terms of User Needs

This section will be written and finalized after completion of the
project-specific statistical methodology identified in EPA\'s *Generic
Guide to Statistical Aspects of Developing an Environmental Results
Program* (2003). This section will present the following information:

### [Meeting and reporting the needs of the project.]{.mark}

[This section shall contain a description of how the results of the
study will be analyzed and evaluated to determine whether the needs of
the project were met and then reported.]{.mark}

[]{.mark}

### [Mathematical and statistical formulae.]{.mark}

[This section shall contain details of formulae that will be used to
calculate precision, accuracy/bias, completeness, comparability and
sensitivity (if applicable) of the project data.]{.mark}

[]{.mark}

### [Approach to managing unusable data.]{.mark}

[This section shall contain a description of what will happen if data
are unusable, with particular emphasis on the impact of such unusability
on data representativeness.]{.mark}

**[Quality assurance.]{.mark}**

[This section will include data usability, verification that the Data
Quality Objectives are met, specific quality control needed or missed,
descriptions of corrective actions for data issues, changes or
modification to the QAPP or personnel on the project. An overall
assessment of data will then be made considering EPA QA/G-9.
[\<http://www.epa.gov/quality/qs-docs/g9-final.pdf\>](http://www.epa.gov/quality/qs-docs/g9-final.pdf)]{.mark}

[^1]: Includes data from self-certification forms, return-to-compliance
    forms, and non-applicability forms.