![](media/image1.png){width="1.20625in" height="1.20625in"}**Quality
Management Procedure**

***OETI-PMP-09***

**Environmental Protection Agency**

**Office of Enterprise Technology and Innovation (OETI)**

December 19, 2006 -- Version 1.0

Document Change History

  --------- ------------------ ---------------- -----------------------------
  Version   Date               Author           Description of Changes

                                                

                                                

                                                

                                                

                                                

                                                

                                                

                                                

                                                
  --------- ------------------ ---------------- -----------------------------

Contents

[**1.** **Introduction 1**](#introduction)

[1.1 Purpose 1](#purpose)

[1.2 Background 1](#background)

[2. Approach 3](#approach)

[2.1 Assumptions 3](#assumptions)

[2.2 Scalability 3](#scalability)

[2.3 Best Practices 4](#best-practices)

[3. Roles and Responsibilities 6](#roles-and-responsibilities)

[4. Procedure 7](#procedure)

[4.1 Process Flow Diagram 7](#process-flow-diagram)

[4.2 Steps 7](#steps)

[4.2.1 Define Quality Standards 7](#define-quality-standards)

[4.2.2 Perform Quality Planning 8](#perform-quality-planning)

[4.2.3 Create/Update Quality Assurance Plan
8](#createupdate-quality-assurance-plan)

[4.2.4 Perform Quality Assurance Activities
9](#perform-quality-assurance-activities)

[4.2.5 Evaluate Quality Results 9](#evaluate-quality-results)

[4.2.6 Determine Process Improvements
9](#determine-process-improvements)

[4.2.7 Change Request Needed? 10](#change-request-needed)

[4.2.8 Implement Process Improvements
10](#implement-process-improvements)

[5. Considerations 11](#considerations)

[Appendix A Acronyms A-1](#acronyms)

[Appendix B Checklist for Quality Management
B-1](#checklist-for-quality-management)

[Appendix C Additional Resources C-1](#additional-resources)

[Appendix D Interface Requirements D-1](#interface-requirements)

[Appendix E Quality Assurance Plan Template
E-1](#quality-assurance-plan-template)

[Appendix F Quality Assurance Techniques
F-1](#quality-assurance-techniques)

# Introduction

This document defines the process by which the staff within the
Environmental Protection Agency's (EPA's) Office of Enterprise
Technology and Innovation (OETI) performs quality management for
projects. This document incorporates industry best practices, EPA
policies, and standards from the Project Management Institute (PMI) and
the Software Engineering Institute's (SEI) Capability Maturity Model
Integration (CMMI).

## Purpose

This document defines the methodology, process flow, and relevant
standards by which OETI project staff performs quality management
activities and identifies participants and their responsibilities.

## Background

Quality is defined as "the degree to which a set of inherent
characteristics fulfill requirements."[^1] Quality continues to be a top
priority for senior executives in virtually all organizations and is
consistently tied to stakeholder value and customer satisfaction. The
intent of quality management is to reduce risk (cost and schedule) by
building quality processes into the project to prevent an error from
occurring rather than identifying an error or defect after the fact.

This procedure defines the approach for performing project quality
management activities, including both process quality and product
quality. Process quality can be applied to any project whereas product
quality is typically more specific to an actual product or deliverable.
In addition, quality assurance generally applies to the quality
management of processes whereas quality control refers to the quality
management of a product. However, the approach to developing a quality
management process for either is the same. The Quality Manager is the
individual responsible for quality activities and for developing the
Quality Assurance Plan with input from the Project Manager, Project Team
Leads, and stakeholders. The Quality Assurance Plan is defined during
quality planning activities and can be formal or informal in its
approach. It addresses the dimensions of quality applicable to the
project.

The following are examples of quality dimensions:

-   Time (e.g., samples evaluated over time yield results that
    consistently meet quality standards)

-   Timeliness (e.g., stated deadlines are consistently met)

-   Completeness (e.g., the project or resulting product addresses all
    stated requirements)

-   Consistency (e.g., processes are defined and repeatable)

-   Accessibility and convenience (e.g., product or service is available
    for use as stated in any requirements or service-level agreements
    \[SLAs\])

-   Accuracy (e.g., random samples of output are consistent and correct)

    -   Responsiveness (e.g., stated response, customer service, or
        issue resolution times are consistently met or exceeded)

These dimensions can be evaluated using audits, review procedures, or
quality metrics. The number and type of quality metrics that the Quality
Manager collects and evaluates for the project is determined during the
planning process. Metrics should correlate to any performance measures
stated for the project. These measures can be defined by the quality
standards to which the project is adhering and other organizational
requirements (e.g., internal SLA). In addition, metrics may be both
internal and external in nature. Internal metrics are those that the
Quality Manager uses to evaluate the project's adherence to stated
quality goals and are often outputs of project processes (e.g., number
of new issues identified and resolved per reporting period), while
external metrics may be used to evaluate a third party or contractor's
performance. If contractors are involved with the project, the Quality
Manager may monitor the contractor to determine compliance with the
required quality measures just as they audit and monitor the project
staff. Additional detail is included in Appendix D of this procedure.
External quality measures may be documented in such items as a Quality
Assurance Surveillance Plan (QASP), an SLA, performance metrics in the
contract, or the Quality Assurance Plan itself.

The quality management activities defined in the Quality Assurance Plan
for systems projects follow the requirements of the Office of
Environmental Information's (OEI) System Life Cycle Management (SLCM)
Policy and procedures. The OETI quality management procedure adds to the
agency's minimum requirements articulated in the SLCM Policy and
procedures. The SLCM Policy applies only for system projects and not for
non-system projects.

# Approach 

This section explains the approach used to develop the quality
management procedure. It details the assumptions, the degree of
scalability of the procedure, and the industry standards, best
practices, and EPA current practices consulted in creating this
procedure.

## Assumptions

The following assumptions guide this quality management procedure:

-   Quality management is a responsibility shared by project team
    members on OETI projects.

-   A Quality Manager will be appointed by the Project Manager who has
    familiarity with quality management practices. Quality may or may
    not be the Quality Manager's only responsibility, depending on the
    size, scope, and complexity of the project.

-   The Quality Manager will maintain documents using the document
    management procedures and tools defined for the project. (See
    *PMP-12 Document Management Procedure.*)

    -   The Quality Assurance Plan is a living document and its users
        continually update it over the life cycle of the project as new
        requirements, change requests, and re-planning activities occur.

## Scalability

The Project Manager and the Quality Manager make the decision whether
and to what extent to implement quality management at a high level
during project planning. The timing of project planning activities is
discussed in *PMP-02 Project Initiation and Planning Procedure*. Table
2-1 below provides basic guidelines for determining the extent to which
this quality management procedure is implemented for a given project.

Table 2.1. Quality Management Procedure Scalability Guidelines

+-------------+--------------------+-----------------------------------+
| Procedure   | Does the Procedure | Determining Procedure Scalability |
|             | Apply?             |                                   |
+-------------+--------------------+-----------------------------------+
| OETI-PMP-09 | Applies to all     | -   All projects require at least |
| Quality     | projects           |     basic quality assurance       |
| Management  |                    |     practices such as peer review |
| Procedure   |                    |     of a deliverable              |
|             |                    |                                   |
|             |                    | -   Development of a formal       |
|             |                    |     Quality Assurance Plan is     |
|             |                    |     required for projects with a  |
|             |                    |     Project Complexity Model      |
|             |                    |     rating of "High" and for      |
|             |                    |     system projects               |
|             |                    |                                   |
|             |                    | -   Quality assurance activities  |
|             |                    |     are performed for projects    |
|             |                    |     that incorporate performance  |
|             |                    |     based contracts (as dictated  |
|             |                    |     in the contract)              |
|             |                    |                                   |
|             |                    | -   More detailed quality         |
|             |                    |     assurance practices are       |
|             |                    |     required if:                  |
|             |                    |                                   |
|             |                    | -   There are significant risks   |
|             |                    |     associated with the project   |
|             |                    |     that require quality          |
|             |                    |     assurance as a mitigating     |
|             |                    |     action                        |
|             |                    |                                   |
|             |                    | -   The quality of the product    |
|             |                    |     plays a significant role in   |
|             |                    |     the success of the project    |
|             |                    |                                   |
|             |                    | -   The project must meet         |
|             |                    |     internal or external quality  |
|             |                    |     standards                     |
+-------------+--------------------+-----------------------------------+

The project team and Quality Manager define and tailor the quality
management activities during the planning process based upon critical
planning items such as complexity and risk, resource allocation and
responsibilities, project budget and scope, duration of the project, and
the number of organizations that make up the project team. Smaller
projects may require a simple quality review or a peer review of a
deliverable at a designated project milestone. There may be no metrics
or a smaller number of quality metrics to be collected, and fewer
activities on the project schedule for quality management and process
improvement activities. Large, complex projects typically require more
extensive quality activities and metrics.

Quality scope is the extent of quality activities considered necessary
for a project and is affected by the project purpose, the stakeholders,
project duration, and risk. The higher the level of risk documented for
a project, the greater the need for quality activities to mitigate the
known and potential unknown risk(s). Projects involving new technologies
or project types new to the organization are good candidates for quality
management activities. Additional guidance on understanding risks and
the management of risks is provided in *PMP-05 Risk Management
Procedure.*

If a contractor is used to support the project, metrics may be
established to evaluate the quality of the work performed or products
delivered, in accordance with the contract. See both *PMP-10 Procurement
Management Procedure* and Appendix D of this procedure for more
information about evaluating contractor support activities.

The project team is responsible for evaluating all these factors to
determine the type and extent of quality activities to implement for the
project.

## Best Practices

The OETI vision includes the employment of best practices from both
industry and the EPA. This procedure incorporates the following best
practices and existing regulations and policies:

-   **EPA regulations and standards**

```{=html}
<!-- -->
```
-   EPA Directive 2100.5, System Life Cycle Management Policy. Available
    at <http://intranet.epa.gov/oei/imitpolicy/>.

-   The EPA Interim Agency System Life Cycle Management Procedures.
    Available at:
    <http://intranet.epa.gov/otop/policies/Extended_InterimProcedures.pdf>.

-   Quality Assurance guidance for financial management can be found in
    the Quality Assurance Guide (2005_qa_guides.pdf) prepared by the
    Office of the Chief Financial Officer (OCFO). Available at:
    [http://intranet.epa.gov/ocfo/policies/2006_qa_guides.pdf](https://webmail.bah.com/exchweb/bin/redir.asp?URL=http://intranet.epa.gov/ocfo/policies/2006_qa_guides.pdf).

-   The EPA Quality Management and CMM information can be found in
    Section 12 of the EPA Interim Agency System Life Cycle Management
    Procedures. Available at
    [http://intranet.epa.gov/otop/policies/Extended_InterimProcedures.pdf](https://webmail.bah.com/exchweb/bin/redir.asp?URL=http://intranet.epa.gov/otop/policies/Extended_InterimProcedures.pdf).

```{=html}
<!-- -->
```
-   **Federal regulations, industry standards and best practices**

```{=html}
<!-- -->
```
-   International Standards Organization (ISO) ISO 9000, 9001. Available
    at: <http://www.iso.org/iso/en/ISOOnline.frontpage>.

-   Software Engineering Institute (SEI) Capability Maturity Model
    Integration (CMMI), CMMI for Systems Engineering, Software
    Engineering, Integrated Product and Process Development, and
    Supplier Sourcing, Version 1.1, CMMI-SE/SW/IPPD/SS, dated March
    2002.

```{=html}
<!-- -->
```
-   Project Management Institute Project Management Body of Knowledge
    (PMBOK®), Third Edition, 2004 (Chapter 8 -- Project Quality
    Management).

```{=html}
<!-- -->
```
-   Process Improvement in accordance with ANSI-ASQ Q9004:2000 (American
    National Standards Institute-American Society for Quality, Quality
    Management Systems -- Guidelines for Performance Improvements).

-   ANSI/ISO/ASQC Q10011-1-1991, Guidelines for Auditing Quality Systems
    -- Part 1: Auditing.

-   ISO/International Electrotechnical Commission (IEC). ISO/IEC 21827
    Systems Security Engineering Capability Maturity Model (SSE-CMM).

# Roles and Responsibilities

Table 3-1 presents the roles and responsibilities for OETI project staff
involved in quality management activities. This table lists functions or
tasks that each project role performs. While each role will be assigned
to an individual staff member, an individual may perform multiple roles
for a project.

Table 3.2. Quality Management Roles and Responsibilities

+--------------------+-------------------------------------------------+
| Role Definition    | Responsibilities                                |
+--------------------+-------------------------------------------------+
| **Project          | -   Provides guidance for the selection or      |
| Manager**          |     definition of quality standards             |
|                    |                                                 |
|                    | -   Assigns the role of Quality Manager to the  |
|                    |     individual who will be responsible for      |
|                    |     quality activities                          |
|                    |                                                 |
|                    | -   Participates in quality scope and planning  |
|                    |     process                                     |
|                    |                                                 |
|                    | -   Ensures that the Quality Assurance Plan is  |
|                    |     consistent with the project scope statement |
|                    |     and EPA standards                           |
|                    |                                                 |
|                    | -   Reviews and approves the Quality Assurance  |
|                    |     Plan                                        |
|                    |                                                 |
|                    | -   Assigns action items resulting from quality |
|                    |     management activities to the appropriate    |
|                    |     Project Team Lead                           |
|                    |                                                 |
|                    | -   Oversees implementation of quality process  |
|                    |     improvement action items                    |
|                    |                                                 |
|                    | -   Evaluates quality assurance results and     |
|                    |     assists with selection and prioritization   |
|                    |     of quality process improvement initiatives  |
|                    |                                                 |
|                    | -   Creates or delegates the creation of        |
|                    |     Corrective Action Plans (CAPs) and          |
|                    |     evaluates CAPs for cost, schedule, and      |
|                    |     scope impacts                               |
+--------------------+-------------------------------------------------+
| **Planning and     | -   Participates in the quality management      |
| Evaluation Team    |     planning process                            |
| Lead**             |                                                 |
|                    | -   Assists with evaluation of quality          |
|                    |     assurance reviews                           |
|                    |                                                 |
|                    | -   Reviews quality reports and metrics and     |
|                    |     participates in regular quality reporting   |
|                    |     meetings                                    |
+--------------------+-------------------------------------------------+
| **Project Team     | -   Participates in quality assurance and       |
| Lead**             |     quality management planning process         |
|                    |                                                 |
|                    | -   Assists with the development of the Quality |
|                    |     Assurance Plan                              |
|                    |                                                 |
|                    | -   Participates in quality assurance reviews   |
|                    |     and assists with capture of quality metrics |
|                    |     (as applicable)                             |
|                    |                                                 |
|                    | -   Oversees and assists with the               |
|                    |     implementation of team specific quality     |
|                    |     process improvement action items            |
+--------------------+-------------------------------------------------+
| **Project Team     | -   Participates in quality management planning |
| Member**           |     process                                     |
|                    |                                                 |
|                    | -   Implements quality process improvement      |
|                    |     initiatives                                 |
|                    |                                                 |
|                    | -   Participates in quality assurance reviews   |
|                    |     and assists with capture of quality metrics |
|                    |     (as applicable)                             |
+--------------------+-------------------------------------------------+
| **Quality          | -   Interprets EPA requirements for quality     |
| Manager**          |     management programs                         |
|                    |                                                 |
|                    | -   Participates in quality management planning |
|                    |     process and makes recommendations for an    |
|                    |     actionable Quality Assurance Plan specific  |
|                    |     to the project                              |
|                    |                                                 |
|                    | -   Develops the Quality Assurance Plan         |
|                    |                                                 |
|                    | -   Ensures project participants receive        |
|                    |     appropriate quality training                |
|                    |                                                 |
|                    | -   Oversees the execution of quality           |
|                    |     management activities and adherence to the  |
|                    |     quality standards                           |
|                    |                                                 |
|                    | -   Performs quality assurance audits/reviews   |
|                    |     and collects quality metrics, as            |
|                    |     appropriate                                 |
|                    |                                                 |
|                    | -   Works with Team Leads to evaluate quality   |
|                    |     assurance results and makes recommendations |
|                    |     for process improvements                    |
|                    |                                                 |
|                    | -   Determines whether process improvement      |
|                    |     initiatives have been implemented and       |
|                    |     evaluates impact on overall quality         |
+--------------------+-------------------------------------------------+

# Procedure

This section presents the process flow for quality management and
describes each step of the process in detail.

## Process Flow Diagram

Figure 4-1 depicts the process flow for the quality management
procedure. This process originates with the definition of quality
standards defined during project planning activities (see *PMP-02
Project Initiation and Planning Procedure*). The procedure may also be
triggered by risk mitigation plans that impact existing or require new
quality assurance activities. Finally, if contractor support is used,
the contract terms and performance criteria are inputs to the quality
management process. The procedure continues through quality planning,
collection of quality metrics, evaluation, and implementation of
necessary process improvements.

Figure **4**‑**1**. Quality Management Process Flow

## Steps

The following sections describe the steps of the quality management
process shown in Figure 4-1 and the roles involved with its execution.

### Define Quality Standards

Early in the planning process, the Project Manager assigns the role of
Quality Manager. The Project Manager and applicable Project Team Members
work with the Quality Manager to determine the basis or standards for
the Quality Assurance Plan. The standards could be organization- or
industry-specific or a combination thereof. The quality standards are
not typically developed specifically for the project unless there is a
unique requirement to do so. The standards should be existing internal
or external criteria that can be used to determine the types of quality
activities that are required or desirable. The standards selected also
provide the basis for the planning sessions and the basic quality goals
and provide some context for quality scope discussions. The Quality
Manager documents the selected standards in the Quality Assurance Plan.
Standards are more applicable for large projects. Smaller projects may
not require selected standards.

### Perform Quality Planning

The Project Manager, the Project Team Lead, and Project Team Members
work with the Quality Manager to determine the details of the quality
management activities. The scope of quality management activities varies
by project but includes all quality assurance processes that the project
team will perform over the life of the project. The Project Manager and
Quality Manager ensure that the scope of the process is consistent with
the project scope statement and determine which aspects of quality
management apply to their project. Often, the activities selected depend
on the type of project and the scope, risk, and complexity of the
project. Activities are also selected based on what aspect of quality
needs to be managed (product, process, or output such as a deliverable).
These aspects may include:

-   Quality goals and objectives for the project

-   Timeframe and frequency for quality activities (weekly, monthly,
    quarterly)

-   Resource allocation (internal versus external)

-   Roles and responsibilities related to quality activities for project
    resources

-   Quality training required for project resources

-   Budget allocated for quality activities

-   Type of quality assurance audits and reviews (process and/or
    product) depending on the nature of the project

-   Quality metrics to be collected

-   Quality reporting frequency and method

-   Issue escalation process for quality issues (should adhere to the
    process defined through *PMP-06 Issue Management*)

-   Communication of quality objectives, activities, and results

-   Schedule implications of quality activities

    -   Whether or not an Independent Verification and Validation (IV&V)
        is required for the project. (Note: The decision to conduct an
        IV&V for a project may be determined anytime during the project
        lifecycle and usually initiates a separate project track based
        on the need for procurement or identification of an independent
        evaluator). Refer to *PMP-10 Procurement Management Procedure*
        for procurement guidance and *PMP-02 Project Initiation and
        Planning Procedure* for guidance on initiating new projects.

The outputs from the quality management planning process may have
schedule or cost considerations and may need to be evaluated for
feasibility prior to finalizing the details of the Quality Assurance
Plan. Refer to *PMP-02 Project Initiation and Planning Procedure* for
guidelines for estimating the costs and resources for quality activities
and to *PMP-03 Project Schedule and Cost Baseline Procedure* for
guidelines on incorporating these activities into the schedule and cost
baselines for the project.

### Create/Update Quality Assurance Plan

The Quality Manager creates the Quality Assurance Plan as part of the
Project Management Plan. For system projects, the Quality Assurance Plan
is developed in accordance with OEI's SLCM policy and procedures and is
incorporated in the System Management Plan (SMP), a document similar to
the Project Management Plan. The Quality Assurance Plan should address
how the project will implement and monitor the quality activities
defined in the quality management planning process. In addition, the
Quality Assurance Plan defines what quality activities will be measured
for the project and reflects all decisions made during the quality
management planning process. The plan captures the specific metrics that
will be collected and analyzed to evaluate the quality processes. The
Project Manager reviews and approves the Quality Assurance Plan prior to
its issuance to the Project Team Members participating in quality
activities.

The Quality Assurance Plan is a living document and its users update it
periodically to reflect changes in scope or approach. The plan should
address the elements defined in the Checklist for Quality Management in
Appendix B. Links to resources and sample Quality Assurance Plans can be
found in Appendix C. A sample Quality Assurance Plan framework is
provided in Appendix E. The Project Manager should review and approve
the Quality Assurance Plan each time it is updated.

### Perform Quality Assurance Activities

The Quality Manager works with the Project Team Lead and Project Team
Members to perform quality assurance activities that are appropriate for
the project as defined in the Quality Assurance Plan. The Quality
Manager objectively evaluates adherence to process descriptions,
standards, and procedures determined to be applicable for the project in
the Quality Assurance Plan and reports findings accordingly.
Additionally, the Quality Manager ensures that quality training is
delivered to project resources as required. Detailed descriptions of the
types of quality assurance activities that may be selected and
implemented for a project are included in Appendix F*.*

### Evaluate Quality Results

The Quality Manager works with the Project Team Lead(s) for the
process/product being evaluated to document the appropriate results from
the quality assurance activities. He or she evaluates the results in
relation to the stated quality assurance and/or quality control goals or
standards. Reporting the results of the quality assurance activities is
part of this step. The Quality Manager defines the content and format
for the Quality Reports based on the quality assurance activities
employed. Reporting activities ensure that the proper information is
communicated to the appropriate level of management.

Evaluation activities determine whether the quality assurance findings
and metrics provide evidence of process or product issues. Issues may
include an increase in the number of errors or reported incidents or
metrics that exceed defined thresholds for a process or product. Issues
are communicated and tracked in accordance with *PMP-06 Issue Management
Procedure*. Project Team Members may also look at findings over time to
determine if processes are improving or becoming less efficient and log
risks for potential problems. Risk identification, mitigation, and
tracking are described in *PMP-05 Risk Management Procedure*. The
Quality Manager, Project Manager, and Project Team Leads may attempt to
identify process improvements or other mitigating actions in an effort
to prevent quality issues.

### Determine Process Improvements

The results of the quality assurance reviews together with the quality
metrics are used by the Quality Manager to determine whether process
improvement initiatives are needed. The criteria used to make this
decision can vary but usually include the severity of any quality issues
identified, potential impact on cost, schedule and project risk, and
whether identified problems can be tied to a specific event (e.g.,
resource turnover) or seem to point to a process that is "broken" or not
working effectively. The scope of the initiatives can be as simple as an
action item or as involved as a separate project for building additional
quality into existing processes or for adding new processes. Process
improvement recommendations can apply to any process. Typically, a CAP
is created by the Project Manager (or delegated to the appropriate
project team member) to define the specific steps the project team will
take to address the issues identified during the quality audits and/or
reviews. The Project Manager should evaluate CAPs to consider the impact
on cost, schedule, scope and overall quality.

### Change Request Needed?

The Quality Manager works with the Project Manager to evaluate the
proposed process improvement(s) to determine which improvements require
routing through the Change Control Process defined in *PMP-08 Change
Control Procedure*. For those with measurable cost, schedule, or
resource requirements for implementation, the Project Manager or
designated Project Team Lead completes a change request (CR) for review
and approval.

### Implement Process Improvements

The Quality Manager communicates the process improvement recommendations
and action items to the Project Team Members during the project
statusing activities that are defined for the project. Approved process
improvement activities and/or CAP(s) should be reflected in the Project
Management Plan, Quality Assurance Plan, and any applicable subsidiary
plans. Guidance for developing and updating the Project Management Plan
and defining statusing activities can be found in *PMP-02 Project
Initiation and Planning Procedure* and *PMP-04 Project Status,
Reporting, and Forecasting Procedure*.

The Project Manager or a designated team member determines the level of
effort required for each approved recommendation/action item. If the
resources of the project team are constrained, prioritization of process
improvement initiatives may be required. The team decides how the
recommendations and action items will be implemented and assigns the
activities to the appropriate resources. The Quality Manager will
evaluate the effectiveness and impact of the new measures in subsequent
quality assurance reviews in accordance with the Quality Assurance Plan.

# Considerations

The following provides a list of general best practices that should be
considered when conducting quality management:

-   The Quality Manager should be experienced in implementing and
    maintaining quality management activities and also be familiar with
    EPA and industry standards. In addition, the Quality Manager should
    be as autonomous as possible and not report directly to the Project
    Manager but rather to the Project Sponsor. For large and/or
    technically complex projects, the quality assurance role is usually
    performed by an external party or contractor.

-   Most projects, regardless of size, benefit from some type of quality
    assurance activity, even if it is as simple as a peer review of the
    final product before submission.

-   Schedules and work plans should incorporate quality activities and
    dependencies should be noted as soon as the Quality Assurance
    Planning process is complete. As a result of quality assurance
    activities, changes to work plans and schedules may be required to
    take into account any new process improvement activities and/or
    requirements. The status of quality assurance activities should be
    updated and reported to the project team at regular intervals.

-   The greater the number and severity of risks on the project, the
    greater the need for a quality management program. Quality assurance
    and risk management activities should be integrated. Quality
    assurance processes can identify project risks. Through the risk
    management process the project team determines the risk response,
    which may include additional, more detailed or more frequent quality
    assurance activities.

-   The quality management process should be in effect for the duration
    of the project.

###### Acronyms

  -------------- -----------------------------------------------------------
  Abbreviation   Description

  **ANSI**       American National Standards Institute

  **ASQ**        American Society for Quality

  **CAP**        Corrective Action Plan

  **CMMI**       Capability Maturity Model Integration

  **CR**         Change Request

  **EPA**        Environmental Protection Agency

  **IEC**        International Electrotechnical Commission

  **IMS**        Integrated Master Schedule

  **IRM**        Information Resources Management

  **ISO**        International Standards Organization

  **IV&V**       Independent Verification and Validation

  **OCFO**       Office of the Chief Financial Officer

  **OEI**        Office of Environmental Information

  **OETI**       Office of Enterprise Technology and Innovation

  **PMBOK**      Project Management Body of Knowledge

  **PMI**        Project Management Institute

  **PMO**        Project Management Office

  **QASP**       Quality Assurance Surveillance Plan

  **SEI**        Software Engineering Institute

  **SLA**        Service Level Agreement

  **SLCM**       System Life Cycle Management

  **SMP**        System Management Plan

  **SSE-CMM**    Systems Security Engineering Capability Maturity Model
  -------------- -----------------------------------------------------------

######  Checklist for Quality Management

The following provides a checklist for the key activities associated
with each step of this quality management procedure.

+-------------------------------------------------+--------------------+
| Activities                                      | Responsible        |
|                                                 | Parties            |
+=================================================+====================+
| 4.2.1 Define Quality Standards                  |                    |
+-------------------------------------------------+--------------------+
| -   The role of Quality Manager is assigned     | Project Manager    |
+-------------------------------------------------+--------------------+
| -   External quality requirements for the       | Quality            |
|     project are reviewed                        | Manager/Project    |
|                                                 | Manager            |
| -   The quality standards for the project are   |                    |
|     defined                                     |                    |
|                                                 |                    |
| -   Implications of the quality standards on    |                    |
|     the type of quality assurance activities    |                    |
|     that should be implemented for the project  |                    |
|     are reviewed                                |                    |
+-------------------------------------------------+--------------------+
| 4.2.2 Perform Quality Planning                  |                    |
+-------------------------------------------------+--------------------+
| -   Roles and responsibilities for quality      | Quality            |
|     assurance activities are defined            | Manager/Project    |
|                                                 | Manager/Project    |
| -   A schedule for quality assurance activities | Team Lead/Project  |
|     is defined                                  | Team Members       |
|                                                 |                    |
| -   Quality assurance activities are defined    |                    |
|     for the project based on project size,      |                    |
|     scope and complexity, and defined quality   |                    |
|     standards                                   |                    |
|                                                 |                    |
| -   Quality metrics are defined specifically    |                    |
|     for the project                             |                    |
|                                                 |                    |
| -   A process for capturing and documenting     |                    |
|     quality metrics are defined                 |                    |
|                                                 |                    |
| -   Communication activities for providing      |                    |
|     quality assurance feedback and guidance     |                    |
|     within the project and across the           |                    |
|     organization are defined                    |                    |
|                                                 |                    |
| -   Reporting mechanisms for quality assurance  |                    |
|     activities are defined                      |                    |
|                                                 |                    |
| -   A process for creating and implementing     |                    |
|     Corrective Action Plans (CAPs) is defined   |                    |
|                                                 |                    |
| -   A process for escalating and resolving      |                    |
|     quality issues is defined                   |                    |
+-------------------------------------------------+--------------------+
| -   Quality training requirements for team      | Quality Manager    |
|     members are defined                         |                    |
+-------------------------------------------------+--------------------+
| -   Quality assurance activities are            | Quality            |
|     incorporated into the project schedule      | Manager/Project    |
|                                                 | Manager            |
+-------------------------------------------------+--------------------+
| 4.2.3 Create/Update Quality Assurance Plan      |                    |
+-------------------------------------------------+--------------------+
| -   Decisions made during quality planning are  | Quality Manager    |
|     documented and incorporated in the Quality  |                    |
|     Assurance Plan                              |                    |
|                                                 |                    |
| -   The Quality Assurance Plan is communicated  |                    |
|     to Project Team Leads and project team      |                    |
|     members (if applicable)                     |                    |
+-------------------------------------------------+--------------------+
| -   The Quality Assurance Plan is consistent    | Quality            |
|     with all other project plans in terms of    | Manager/Project    |
|     scope and approach                          | Manager            |
|                                                 |                    |
| -   The Quality Assurance Plan is completed,    |                    |
|     reviewed, and approved by project           |                    |
|     management                                  |                    |
+-------------------------------------------------+--------------------+
| 4.2.4 Perform Quality Assurance Activities      |                    |
+-------------------------------------------------+--------------------+
| -   Quality training is has been delivered to   | Project Team Leads |
|     project resources as required               |                    |
+-------------------------------------------------+--------------------+
| -   Quality Assurance activities are conducted  | Quality Manager    |
|     as defined in Quality Assurance Plan and at |                    |
|     the intervals and/or scheduled dates        |                    |
|     defined in the project schedule             |                    |
|                                                 |                    |
| -   The results of the quality assurance        |                    |
|     activities are documented as defined in the |                    |
|     Quality Assurance Plan                      |                    |
+-------------------------------------------------+--------------------+
| 4.2.5 Evaluate Quality Results                  |                    |
+-------------------------------------------------+--------------------+
| -   Report formats are defined                  | Quality Manager    |
|                                                 |                    |
| -   Quality Assurance results are communicated  |                    |
|     to the appropriate Project Team Leads or    |                    |
|     team members                                |                    |
+-------------------------------------------------+--------------------+
| -   Quality Assurance results are reviewed as   | Quality            |
|     defined in Quality Assurance Plan and at    | Manager/Project    |
|     the intervals and/or scheduled dates        | Manager            |
|     defined in the project schedule             |                    |
+-------------------------------------------------+--------------------+
| -   Evidence of quality issues is analyzed to   | Quality            |
|     determine impact and scope of the           | Manager/Project    |
|     problem(s)                                  | Manager/Project    |
|                                                 | Team Leads         |
+-------------------------------------------------+--------------------+
| 4.2.6 Determine Process Improvements            |                    |
+-------------------------------------------------+--------------------+
| -   Quality results are reviewed by appropriate | Quality            |
|     team members and necessary process          | Manager/Project    |
|     improvement options are evaluated           | Manager/Project    |
|                                                 | Team Leads         |
| -   Process improvement recommendations are     |                    |
|     reviewed and prioritized as needed          |                    |
+-------------------------------------------------+--------------------+
| -   Corrective Action Plans (CAP)s are          | Project Manager    |
|     developed for implementation of process     |                    |
|     improvements                                |                    |
+-------------------------------------------------+--------------------+
| -   Impact (cost, schedule, resources) of each  | Quality Manager    |
|     process improvement is analyzed and         |                    |
|     documented                                  |                    |
+-------------------------------------------------+--------------------+
| 4.2.7 Change Request Needed?                    |                    |
+-------------------------------------------------+--------------------+
| -   The proposed process improvement is         | Quality Manager    |
|     submitted as a change request if there are  |                    |
|     measurable cost, schedule and resource      |                    |
|     impacts                                     |                    |
|                                                 |                    |
| -   Process improvements not requiring          |                    |
|     measurable cost, schedule, or resources     |                    |
|     change are scheduled for implementation     |                    |
+-------------------------------------------------+--------------------+
| 4.2.8 Implement Process Improvements            |                    |
+-------------------------------------------------+--------------------+
| -   Process improvements not requiring          | Quality            |
|     measurable cost, change, and resources are  | Manager/Project    |
|     implemented by working with appropriate     | Manager/Project    |
|     Project Team Leads and/or team members      | Team Leads         |
+-------------------------------------------------+--------------------+
| -   Approved change requests for process        | Quality            |
|     improvements are planned and scheduled for  | Manager/Project    |
|     implementation                              | Manager            |
|                                                 |                    |
| -   The Project Management Plan, Quality        |                    |
|     Assurance Plan, and any other impacted      |                    |
|     subsidiary plans are updated to reflect the |                    |
|     changes resulting from implementation of    |                    |
|     process improvement(s)                      |                    |
|                                                 |                    |
| -   Implemented process improvement actions are |                    |
|     measured and evaluated over time to         |                    |
|     determine success of the improvement(s)     |                    |
+-------------------------------------------------+--------------------+

######  Additional Resources

The links below provide additional background, guidance and examples for
the development of a quality plan and quality checklist for the project:

 

+---+-----------------+-------------------------+---------------------+
|   | Form/Guidance   | Source                  | Website             |
+---+-----------------+-------------------------+---------------------+
| 1 | Guidance for    | State of New Jersey     | <h                  |
| \ | non-software    |                         | ttp://www.state.nj. |
| . | Quality         |                         | us/transportation/e |
|   | Assurance       |                         | ng/documents/miscre |
|   |                 |                         | f/qaplanguide.shtm> |
+---+-----------------+-------------------------+---------------------+
| 2 | Guidance for    | Washington State        | <http://isb         |
| \ | Software        | Department of           | .wa.gov/tools/pmfra |
| . | Quality         | Information Services    | mework/planning/qua |
|   | Assurance and   |                         | lityassurance.aspx> |
|   | Plan Template   | NASA Goddard Space      |                     |
|   |                 | Flight Center           | http:/              |
|   |                 |                         | /sw-assurance.gsfc. |
|   |                 | NASA Marshall Space     | nasa.gov/discipline |
|   |                 | Flight Center           | s/quality/index.php |
|   |                 |                         |                     |
|   |                 | Tantara Inc. (private   | <http://www.cio.    |
|   |                 | sector company offering | energy.gov/document |
|   |                 | advice for software     | s/csr_sqa_plan.doc> |
|   |                 | process improvement and |                     |
|   |                 | software quality        | <                   |
|   |                 | assurance)              | http://www.tantara. |
|   |                 |                         | ab.ca/ja_qaiee.htm> |
+---+-----------------+-------------------------+---------------------+
| 3 | Sample Quality  | Department of Energy    | <http://cio.ene     |
| \ | Assurance       |                         | rgy.gov/it-project- |
| . | Checklists      | State of Texas          | management/510.htm> |
|   |                 |                         |                     |
|   |                 |                         | <http://www.        |
|   |                 |                         | dir.state.tx.us/eod |
|   |                 |                         | /qa/monitor/qa.htm> |
+---+-----------------+-------------------------+---------------------+
| 4 | ISO Quality     | International Standards | [ht                 |
|   | Standards,      | Organization            | tp://www.iso.org](h |
|   | Section ISO     |                         | ttp://www.iso.org/) |
|   | 9000-14000,     |                         |                     |
|   | Understand the  |                         |                     |
|   | Basics, ISO     |                         |                     |
|   | 9000:2000       |                         |                     |
|   | Selection and   |                         |                     |
|   | Use             |                         |                     |
+---+-----------------+-------------------------+---------------------+
| 5 | CMMI for        | Software Engineering    | <http:              |
|   | Development     | Institute               | //www.sei.cmu.edu/> |
|   | Version 1.2,    |                         |                     |
|   | Process and     |                         |                     |
|   | Product Quality |                         |                     |
|   | Assurance, page |                         |                     |
|   | 353             |                         |                     |
+---+-----------------+-------------------------+---------------------+

###### Interface Requirements

The purpose of this appendix is to provide general guidelines for
collecting the appropriate information from contractors to ensure
seamless integration of project quality data and promote efficient
monitoring of the overall project. Frequently, data is needed by support
contractors to enable the Project Manager to assess real-time status
accurately against overall performance, schedule, and cost objectives.
In addition, defined interface points ensure that both the government
and contractor understand their specific roles and responsibilities in
quality management and reporting and that the information can be
efficiently captured utilizing the project's established management
processes and tools. As a result, the data, reporting, and interface
requirements need to be well defined early in the process in order to
ensure that they are fully described in the awarded contract. For
quality management, these requirements may include a Quality Assurance
Surveillance Plan (QASP), Service Level Agreements (SLAs), performance
metrics in the contract, or the Quality Assurance Plan itself.

The following series of questions is provided to help determine the
quality interface requirements appropriate for a specific project.
Requirements may vary significantly depending on the scope, complexity,
size, duration of the project and type of contracts awarded. Overall,
the questions are designed to help refine what kind of information will
be needed to ensure effective quality management of the project and the
correlating responsibilities of the contractor.

-   Does the contractor have explicit quality standards or requirements
    stated in the contract?

```{=html}
<!-- -->
```
-   Will the contractor be responsible for providing the metrics or data
    on which it will be evaluated?

-   What format is required for submission of quality data?

-   How often must the data be submitted?

-   Who from the project team is responsible for assisting with the
    evaluation of the data?

-   Is anyone from the project team responsible for analyzing contractor
    deliverables to determine if they meet specified requirements?

```{=html}
<!-- -->
```
-   Are the implications of bad data or unacceptable metrics clearly
    stated in the contract?

```{=html}
<!-- -->
```
-   What data are needed for EPA to access contractor performance in the
    manner described by the QASP? How and when should this data be
    submitted?

-   Will the contractor be required to identify and track risks and
    issues and provide updates to EPA? If so, in what format and at what
    frequency?

-   Will the contractor be required to report errors for certain
    activities (such as software development) and provide updates to
    EPA? If so, in what format and at what frequency?

-   Will quality activities performed by the contractor be reflected in
    the Integrated Master Schedule (IMS)?

-   Will the contractor be responsible for participating in quality
    assurance activities? If so, are its responsibilities clearly
    defined in the contract or task order?

    -   Is the contractor required to meet certain quality standards for
        project processes? If so, how will adherence be evaluated?

###### Quality Assurance Plan Template

This appendix provides a sample Quality Assurance Plan template that may
be used and/or tailored as appropriate according to project needs.
Appendix C provides internet links to other examples of quality
assurance plans and templates.

**Quality Assurance Plan Template**

Acceptance / Approval Page

**DOCUMENT CHANGE HISTORY** -- *Complete the version, date, author and
description column to accurately describe the modifications made to this
document.*

  --------- ------------------ ---------------- -----------------------------
  Version   Date               Author           Description of Changes

  V X.X                                         
  --------- ------------------ ---------------- -----------------------------

**1.0 INTRODUCTION**

**1.1 PURPOSE**

*\[This section defines the purpose of the plan. Example: The Quality
Assurance Plan defines the guidelines and procedures for managing,
executing, tracking and reporting the quality assurance activities for
the \<project name\> project.\]*

**1.2 SCOPE**

*\[This section defines the parameters of the quality assurance plan.
Add any text necessary to define what is included and what is excluded
as part of the scope of the plan. This text may:*

-   Name the specific entity, product, project team, and/ or component
    for which quality assurance activities are performed and those that
    are expressly excluded

-   Define whether the plan addresses quality assurance/control
    activities for processes and/or products

-   Identify if contractor support is used on the project and if they
    will be the subject of quality assurance activities

    -   Describe other boundaries for the plan\]

**1.3 REFERENCE DOCUMENTATION**

*\[This section identifies the references, standards, procedures and
other documents used to develop the QA activities for the project. This
list may include:*

-   \<project name\> Project Management Plan

-   \<project name\> Project Schedule

-   \<project name\> Configuration Management Plan

-   \<project name\> Configuration Item/Configuration Data List

-   \<project name\> Risk Management Plan

-   \<project name\> Statement of Work and/or Contract and/or Applicable
    Contractual Requirements

    -   Other project documentation, as necessary\]

    -   Internal or external quality management resources or standards,
        as applicable. For system projects, the Quality Manager should
        refer to the Office of Environmental Information's System Life
        Cycle Management Policy and corresponding procedures when
        developing the quality assurance plan.

**1.4 PLAN MAINTENANCE**

*\[This section identifies the office or group responsible for
developing, maintaining, and distributing this plan. It also establishes
how often the plan is reviewed, typically by organizational directives.
Updates are prepared as required.\]*

**2.0 QUALITY GOALS AND OBJECTIVES**

*\[This section identifies the goals and objectives of the QA plan
followed by a brief sentence explaining how they are achieved.\]*

**3.0 ROLES AND RESPONSIBILITIES**

*\[This section provides a description of the roles and responsibilities
for key participants involved in implementing the QA plan. It also
provides the hierarchy of roles within EPA.\]*

**4.0 QUALITY ASSURANCE ACTIVITIES**

**4.1 QUALITY ASSURANCE RESOURCES**

*\[Describe the composition of the resources that support the project's
quality assurance activities. All quality assurance-related costs should
be planned, monitored and controlled in accordance with PMP-02 Project
Initiation and Planning Procedure, PMP-03 Project Schedule and Cost
Baselining, and PMP-04 Project Status, Forecasting and Reporting.\]*

**4.2 PROGRAM ACTIVITIES**

*\[This section describes the quality assurance activities for the
project, how often they occur, and the procedures that apply to them.
The information may be presented in narrative or table format. The
Quality Manager recommends the type of activities needed to objectively
evaluate both process and product quality. Quality activities may
include:*

-   Audits - The Quality Manager is primarily responsible for conducting
    product and process audits with the support and assistance of
    project team members and contractors as appropriate. The purpose of
    audits is to identify deviations in process performance or work
    products, identify noncompliance items that cannot be resolved at
    the technical support or project management level, validate process
    improvement/corrective action achievements, and provide relevant
    reports to all management levels.

-   Process Audits - The Quality Manager conducts process audits in
    accordance with the audit process documented in PMP-09 Quality
    Management Procedure. Process audits occur throughout the life of
    the project and are planned and documented in the project schedule.
    The Quality Manager typically has the option of conducting *ad hoc*
    process audits as deemed necessary.

-   Product Audits - A product audit is an independent examination of a
    deliverable and/or work product(s) to assess compliance with
    specifications, standards, client requirements, or other criteria.
    Product audits occur throughout the life of the project and are
    planned and documented in the project schedule.

-   Reviews - The Quality Manager participates, observes, and/or
    conducts reviews, in accordance with the process documented in
    *PMP-09 Quality Management Procedure*. Additionally, the Quality
    Manager conducts independent reviews of the project's process
    adherence and product standards. These reviews are usually *ad hoc*
    and coordinated with the Project Manager.

*Example:*

*The project's Quality Manager performs the activities in Table 1 below
in support of the quality goals and objectives in Section 2. The
activities in Table 1 are not one-time events, but occur regularly,
periodically, or on an* ad hoc *basis. The Quality Manager and the
Project Manager make the appropriate determinations about the frequency
of the activities and reflects those decisions in the project QA
Schedule. For each activity that is repeated, the focus may be adjusted
to specific areas of concern to ensure that all aspects of the quality
objectives are addressed.*

*Procedures for performing the following activities are reflected in the
right hand column of the table and are explained in the corresponding
procedure. For example, quality assurance activities as determined by
the Quality Manager and the Project Manager are documented in the
project's schedule in accordance with guidance provided in* PMP-03
Project Schedule and Cost Baseline Procedure*. This schedule should be
reviewed and updated as appropriate within the life cycle of the project
and statused as described in* PMP-04 Project Status, Reporting, and
Forecasting Procedure*.\]*

***Table 1. QA Program Activities***

+--------------------------------------------------+-------------------+
| QA Activity                                      | Applicable QA     |
|                                                  | Procedures        |
+==================================================+===================+
| **QA Planning**---Quality Manager performs the   | QA Planning       |
| appropriate planning for all QA activities and   |                   |
| reflects the results in the QA Plan and          |                   |
| Schedule. The QA Schedule is reviewed at least   |                   |
| monthly and updated as appropriate. The plan is  |                   |
| reviewed at least annually and updated as        |                   |
| appropriate.                                     |                   |
+--------------------------------------------------+-------------------+
| **QA/QM Plan Reviews**---Quality Manager reviews | QA Product Review |
| the QA/QM plans developed by contractors.        |                   |
| Comments are provided on non-compliance issues.  | QA Document       |
| The Quality Manager approves the plans when      | Management        |
| corrections are made based on the comments       |                   |
| provided.                                        | QA Status         |
|                                                  | Reporting         |
+--------------------------------------------------+-------------------+
| **Attend Selected Reviews**---Quality Manager    | QA Document       |
| attends selected reviews (contractor status      | Management        |
| meetings, Integrated Baseline Reviews,           |                   |
| Integrated Program Reviews, and other executive  | QA Status         |
| reviews) to evaluate from a QA perspective the   | Reporting         |
| program/project objectives, activities, and      |                   |
| issues.                                          |                   |
+--------------------------------------------------+-------------------+

**4.3 REPORTING**

*\[This section establishes the reporting channels for quality
assurance. It describes the requisite report format(s) to include the
scope of the audit/review, findings, and quality improvement
recommendations. This section also defines how findings are tracked and
reported as well as any tools that are used in the quality assurance
process. The project's Quality Manager is responsible for monitoring,
tracking, and updating the status of findings through closure. Formal
reporting of quality assurance activities and findings should be
performed in accordance with* PMP-04 Project Status, Forecasting and
Reporting Procedure*.\]*

**4.4 ESCALATION (If applicable)**

*\[This section defines the escalation procedure for issues not
resolvable by the Project Manager and the Quality Manager. Reference*
PMP-06 Issue Management Procedure *for specific guidance on identifying
and tracking issues.\]*

**5.0 QUALITY ASSURANCE METRICS (if applicable)**

*\[This section describes the use of measurement and analytic techniques
to determine the effectiveness of quality assurance activities. The
number of metrics defined is proportional to the size, complexity, risk,
and duration of the project. Example:*

*Metrics are a means of tracking and reporting on the progress of
quality assurance program. Measurements are data collected while
conducting quality assurance activities. These data can be analyzed to
derive indicators (metrics), which can be depicted in charts, graphs,
and plots to support the following:*

-   Determination of QA process/program status.

-   Identification of trends.

-   Identification of process and product deficiencies.

    -   Identification of process improvement opportunities.

*Define the measurements that will be collected during quality assurance
activities:*

-   Number of full and partial audits planned for a period of time.

-   Number of full audits, partial audits, and monitoring activities
    conducted:

```{=html}
<!-- -->
```
-   By month

-   By year

```{=html}
<!-- -->
```
-   Number of CAPs opened from audits and monitoring.

-   Number of CAPs closed over a period of time.

    -   Number of CAPs that missed their due date.

*The Quality Manager's analysis is conducted with the Project Manager to
determine trends and support continuous process improvement.\]*

**6.0 CONSTRAINTS AND RISKS**

**6.1 CONSTRAINTS**

*\[This section defines the constraints associated with the quality
assurance plan. Example:*

*Project quality assurance activities are limited by the number of staff
assigned, including contractor. This is often due to availability of
funding for QA and the procedural and timeframe restrictions of the
budget process.\]*

**6.2 RISKS**

*\[This section describes the risks and mitigation strategies for those
risks associated with the quality assurance plan. Example:*

*The consequences of not having enough staff to maintain the operation
will jeopardize the Project Management Office (PMO)'s ability to mature
into an effective organization, and will adversely affect the PMO's
credibility with Congress, the public, and other stakeholders. To
mitigate this risk, the PMO should continue to implement its quality
assurance activities in a phased approach using management input and
risk-based prioritization as the basis for identifying the processes on
which quality assurance is to be performed. In addition, funding should
be provided to increase the quality assurance resources to a level that
will support workload effectiveness. Ultimately, a QA staff equivalent
to three to five percent of total project staff (industry standard)
should be the target for having a viable QA function.\]*

**APPENDIX A** - **ACRONYMS**

**APPENDIX B** - **DEFINITIONS**

###### Quality Assurance Techniques

The types of quality assurance activities are defined as follows:

**Product/service and process audits** -- The purpose of audits is to
identify deviations in process performance, identify findings, escalate
findings that cannot be resolved at the technical support or project
management level, validate process improvement/corrective action
achievements, and provide relevant reports to all management levels. The
audit process is essentially the same for either a process or product
audit.

-   Process audits ensure compliance with the applicable organizational
    and project plans, processes, and standards. Process audits occur
    throughout the life of the project and are identified in the project
    schedule. Unscheduled process audits may also be conducted as deemed
    necessary by the Quality Manager or Project Manager.

    -   A product audit is an independent examination of a deliverable
        and/or work product(s) to assess compliance with specifications,
        standards, customer/stakeholder requirements, or other criteria,
        preferably conducted prior to delivery to the
        customer/stakeholder. Product audits occur as products are being
        developed, prior to delivery, and/or at project milestones. The
        Quality Manager identifies, documents, and tracks
        deviations/non-compliances to closure. Product audits are
        conducted in accordance with the applicable audit processes and
        checklists as tailored for the projects and are also identified
        in the project schedule.

The Quality Manager performs the following activities (as appropriate)
when conducting either audit:

-   Define the scope and purpose of the audit within the audit
    notification

-   Prepare for the audit. (Preparation activities can include preparing
    audit procedures, checklists, and questionnaire)

-   Conduct preparation meetings in advance of the audit and/or send
    notification of pending audit:

```{=html}
<!-- -->
```
-   Define areas to be reviewed

-   Define review criteria

```{=html}
<!-- -->
```
-   Examine evidence of implementation and controls

-   Interview personnel to learn the status and functions of the
    processes and the status of the products (when applicable)

-   Identify findings

-   Discuss findings with the individual(s) being audited

-   Prepare and submit an audit report to Project Manager and
    individual(s) responsible for the process or product being audited

-   Refer unresolved findings to individual(s) responsible for the
    process or product audited and to the Project Manager

    -   Follow up on corrective action/process improvement.

An audit is considered complete when:

-   Each element within the scope of the audit has been examined

-   Findings have been presented to the audited organization

-   Response to findings have been received and evaluated

    -   Audit findings and recommendations are documented (Audit Report)
        when the Project Team has developed and submitted an approved
        CAP.

**Quality Reviews** - Reviews allow the evaluation of documentation
(processes, procedures, and artifacts) and other activities in a less
rigorous manner than audits. They allow the Quality Manager to observe
and identify problems, issues, and areas in need of improvement with a
minimum of impact on the operational aspects of a project. Reviews seek
to identify problems and issues early, and help simplify the monitoring
and management of problem areas throughout the project. The Quality
Manager may also conduct independent reviews of the project's process
adherence and product standards. While the results may indicate findings
like an audit, the process is different.

A typical review includes the following steps:

-   Coordinate review with Project Manager, as these are normally
    conducted on an *ad hoc* basis

-   Verify that correct review procedures are in place

-   Document review results against quality factors (correctness,
    timeliness, reliability, productivity, consistency)

-   Verify product/service traceability, if applicable

-   Verify product/service against contractual requirements, if
    applicable

-   Verify product/service against standards and procedures, if
    applicable

-   Validate process adherence as necessary

-   Document review results/findings against product validation/process
    verification information

-   Report results to affected groups and Project Manager

-   Review CAPs as appropriate.

[^1]: Project Management Institute Project Management Body of Knowledge
    (PMBOK®), Third Edition, 2004, Chapter 8.