> **9/15/03**

[Code 580 Rev H]{.underline}

Software Engineering Division

*Controlled documentation is available on-line at:*
[http://isc.gsfc.nasa.gov/ISO9k/pdh/PDH.pdf](http://isc.gsfc.nasa.gov/Iso9k/iso9001.htm)

*It is the user's responsibility to verify that the version of any
printed documentation matches the on-line version. This document is
referenced by 580-PG-8730.3.1.*

Product Development

Handbook

Signature on File

**Approved by: Joseph Hennessy**

Chief, Information Systems Division

# Preface {#preface .PDHHeading2}

The Information Systems Division (ISD) Product Development Handbook is
under ISD configuration control. Approved changes to this document
should be listed on this page as follows:

[Revision]{.underline} [Section]{.underline} [Description of
Change]{.underline}

A (3/26/99) 1.1 GSFC Quality Policy, scope now listed by reference to
GSFC QMS web page

A (3/26/99) 1.4 Deleted

A (3/26/99) 2.0 Roles & Responsibilities updated to respond to changes
in GPGs

A (3/26/99) 3.0 Roles & Responsibilities updated to respond to changes
in GPGs

A 3/26/99) 3.1 Clarification of Product Plan scopes

Updated definition of Quality Planning Document

A (3/26/99) 6.1 Deleted

A (3/26/99) 6.2 Roles & Responsibilities updated to respond to changes
in GPGs

A (3/26/99) 6.2 Library Modification Process updated

A (3/26/99) Appendix B References to appropriate GPGs added

A (3/26/99) Appendix C Team Organization Chart added to Product Plan
Table of Contents; GPG references added

A (3/26/99) Appendix D Updated Quality Records list and Instructions to
respond to GPG changes

A (3/26/99) Appendix E Mandatory metrics simplified

A (3/26/99) Appendix G Roles & Responsibilities updated to respond to
changes in GPGs

A (3/26/99) Appendix I Web addresses updated

B (5/12/99) 3.0 Roles & Responsibilities updated to add Statistical
Techniques requirements and clarify NCR reporting requirements

B (5/12/99) Appendix B Aligned Required Team Process Criteria names with
the names used in the Library of Approved Team Processes

B (5/12/99) Appendix C Updated to document statistical techniques needs
of ISC (5/12/99) Appendix E Mandatory Metrics revised

B(5/12/99) Appendix G Added Statistical Techniques Requirements;
Clarified NCR reporting Requirements

[Revision]{.underline} [Section]{.underline} [Description of
Change]{.underline}

C (7/27/99) All, I Updated URL references

C (7/27/99) Title, 1.1 Removed PG number, replaced with revision letter

C (7/27/99) All Updated GPG titles and reference numbers

C (7/27/99) 6.1 Updated configuration management process

C (7/27/99) Appendix A Added mapping of Product Plan to ISO 9000-3

C (7/27/99) Appendix C Updated Product Plan instructions

C(7/27/99) 3.1 Updated customer agreement requirements

D (9/01/99) All Updated reference GPGs

D (9/01/99) Title page Updated reference PG, added document traceability
diagram

D (9/01/99) All Updated training requirements for management and Team
Lead

D (9/01/99) Appendices Reordered all appendices and deleted Appendix F
(Customer Involvement Considerations)

E (2/23/00) Appendix A, Major revisions

B, E

E (2/23/00) Appendix F Appendix F deleted, Appendix G, H re-lettered as
Appendix F, G, respectively

E (2/23/00) All Editorial changes to improve clarity

F (6/15/02) Cover Sheet Changed signature approval, corrected GPG
references in Document Hierarchy

F (6/15/02) Change Sheet Added dates of Revisions

F (6/15/02) 1 Removed Yearly Action Planning Process, System Support
Process and change name of Strategic Implementation Process to Strategic
Planning Process, changed all references to them, and update figure 1.1

F (6/15/02) 2 Changed wording to add location of "official" and
"unofficial" personnel folders

F (6/15/02) 5.4 Update Section 5.4 ( new 4.4) to remove GPG 1410.2
reference

F (6/15/02) 4/5/6 Deleted Section 4, Changes sections 5/6 to 4/5 and
updated all references to them accordingly

F (6/15/02) Appendix A Changed reference in Section 4.2 from GPG 8700.2
to GPG 1410.2

G(2/23/03) All Change Information Systems Center to Information Systems
Division

G(2/23/03) Appendices Relettered Appendices A through G as Appendices C
through I, respectively. Added Appendices A and B (Software Development
Process and Software Maintenance Process, respectively)

G(2/23/03) 1.2 Added definitions for Appendices A and B

G(2/23/03) 4.2 Changed responsibility for Software Methodology
Assessment from Software Engineering Laboratory to GSFC Engineering
Process Group (EPG)

H(9/15/03) Appendices Add requirement to follow Electrical Engineering
Division policies and practices for connecting to flight hardware

Changes or questions concerning this document should be addressed to the
ISD/Quality Management System Representative.

Table of Contents

Section Title Page

Preface........... 2

1.0 Scope of Document 7

1.1 Purpose 7

1.2
Definitions.......................................................................7

1.3 Business Development Process Overview 11

1.4 Operating Structure Overview 12

2.0 Business Selection Process 13

2.1 Record Request 14

2.2 Scope Request 15

2.3 Select Work 15

2.4 Notify Customer 15

2.5 Select Team Lead 15

2.6 Document Order 15

2.7 Select Team 16

3.0 Order Fulfillment Process 17

3.1 Understand Customer Needs and Establish Agreement 21

3.2 Assemble/Build Product 21

3.3 Field Product 23

3.4 Deliver the Product 24

4.0 Assessment Process 25

4.1 Business Process Assessment 25

4.2 Software Methodology Assessment 25

4.3 Product Line Assessment 26

4.4 Product Development Handbook Assessment 26

5.0 ISD Crosscutting Processes 27

5.1 Configuration Control Process 27

5.2 Process for Adding, Deleting or Modifying Approved Team Processes in
the Library for Use in Product Plans 28

Appendix A: Software Development
Process\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\.....29

Appendix B: Software Maintenance
Process\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\....45

Appendix C: Product Plan Instructions 56

Appendix D: List of Quality Record Types 75

Appendix E: ISD Roles and Responsibilities 76

Appendix F: Required Team Processes & Functions 82

Appendix G: Mandatory Team Metrics 88

Appendix H: References 91

Appendix I: Acronym List 92

List of Figures

Figure Title Page

1.1 Information Systems Division Business Development Process
[11](#refheading___toc462565016)

2.1 Business Selection Process [14](#__RefHeading___Toc462565019)

3.0 The Iterative Process for ISD Product Development
[17](#__RefHeading___Toc462565028)

3.1 The Assemble/Build Process [22](#__RefHeading___Toc462565031)

3.2 Fielding the Product [23](#__RefHeading___Toc462565033)

3.3 Deliver the Product Process [24](#__RefHeading___Toc462565035)

ISD Product Development Handbook

# 1.0 Scope of Document  {#scope-of-document .PDHHeading1}

This document is intended for use by ISD personnel working with or
without contract support to produce a software product for a customer.
It does not include services such as consultation or management
oversight of contractor development activities. When working for
projects being directed outside ISD, ISD personnel and support
contractors shall follow that project's processes except as delegated to
ISD by the project manager.

# 1.1 Purpose {#purpose .PDHHeading2}

This document is intended to serve as a single reference guide which
describes the ISD end-to-end processes for providing software products
to our customers. These processes are consistent with the GSFC Quality
Manual (QM) directive (GPG 8730.3) and Directives and Documentation
Management (GPG 1410.1). References to the QM will be shown as "(QM
4-x)" where relevant. Detailed documentation associated with the GSFC
Quality Management System (QMS) (for ISO 9001) may be found on the Web
at:

[http://arioch.gsfc.nasa.gov/ISO9000/index.htm](http://arioch.gsfc.nasa.gov/iso9000/index.htm)

The current versions of this handbook may be found on the Web at the
following URL:

[http://isc.gsfc.nasa.gov/ISO9k/pdh/PDH.pdf](http://isc.gsfc.nasa.gov/Iso9k/iso9001.htm)

Each version will be dated and will contain a revision letter.
Notification of updates will be made via e-mail to all Code 580
personnel and Document Configuration Manager.

# 1.2 Definitions {#definitions .PDHHeading2}

-   **Build**: A version of a system or component that incorporates a
    specified subset of the capabilities that the final product will
    provide.

-   **Configuration Management (CM**): A discipline applying technical
    and administrative direction and surveillance to: identify and
    document the functional and physical characteristics of a
    configuration item; control changes to those characteristics; and
    record and report change processing and implementation status.

-   **Cross-Cutting Activities**: Activities that are performed
    throughout the life of the project and apply to all phases of a
    development or maintenance project.

-   **Customer**: Any individual, group, or organization that receives
    and/or pays for a product or service, or arranges to have the
    product or service provided

-   **Design/ Development:** For software, this encompasses all the
    activities that occur during the design, implementation, and testing
    of the product prior to validation.

-   **Engineering Peer Review (EPR**): A focused, in-depth technical
    review that supports the evolving design and development of a
    product subsystem or lower level of assembly. The purpose of an EPR
    is to add value and reduce risk through expert knowledge infusion,
    confirmation of approach, and specific recommendations. An EPR
    provides a penetrating examination of design, analysis,
    manufacturing, integration, test and operational details, drawings,
    processes and data.

-   **Functional Specification**: A document that specifies a function
    that a system or system component must be able to perform.

-   **Glue-ware**: Software designed to implement integration of
    government or commercial off the shelf components into a software
    system.

-   **Integrated Independent Review (IIR)**: One of a series of system
    level reviews conducted at critical project/product milestones in
    accordance with GPG 8700.4. IIR\'s build upon the results of a
    robust set of EPR\'s.

-   **ISD Management Team**: Includes the ISD Chief, and Associate
    Chiefs, Assistant for Technology, Institutional Support Manager,
    Senior Staff Engineers, Senior Administrative Officer, Senior
    Resource Analyst, Branch Heads and Associate Branch Heads

-   **Object Code**: "The output of a code generator (e.g.,
    compiler)\... This code may take a variety of forms: an absolute
    machine-language program, a relocatable machine-language program, an
    assembly-language program, or perhaps a program in some other
    programming language." (Aho and Ullman, [Principles of Compiler
    Design]{.underline}, p. 518.)

-   **Order**: The description of the product that ISD agrees to deliver
    to the customer

-   **Product**: The output of a project that fills a customer's order

-   **Product Development Lead (PDL)**: The manager or leader with
    overall responsibility for managing the software development
    activities, including managing the technical and organizational
    interfaces, and forming and leading the product development team
    (PDT). For a maintenance project the lead is known as the Product
    Maintenance Lead (PML).

-   **Product Development Team (PDT)**: The team with overall
    responsibility for the software development activities. This team is
    equivalent to the QMS Product Design Team. For a maintenance project
    the team is called a Product Maintenance Team (PMT).

-   **Product Manager (PM**): The individual designated as having
    management responsibility for a product. A Product Manager may be
    assigned to any directorate and have a title such as project
    manager, principal investigator, or RTOP manager

-   **Product Plan**: A description of the work to be performed and the
    resources needed to accomplish the goals and objectives established
    in the customer agreement (Appendix C)

```{=html}
<!-- -->
```
-   **Project**: The set of management, administrative, and technical
    activities leading to the delivery of a Center product in response
    to customer requirements and in accordance with Agency and GSFC
    requirements (see NPG 7120.5A). A Team Lead and Team are formed to
    assemble and deliver a product that satisfies an order.

```{=html}
<!-- -->
```
-   **Qualified Individual**: One who is capable of performing assigned
    tasks on the basis of appropriate education, training, and/or
    experience.

-   **Release**: A particular version of a configuration item that is
    made available for use outside the development team.

-   **Resources**: Supplies needed to fill the order such as skilled
    people, money, facilities, etc.

-   **Software Library**: A controlled collection of software and
    related documentation designed to aid in software development, use,
    or maintenance. (IEEE Std 610.12-1990)

-   **Software Management Plan**: A description of the work to be
    performed and the resources needed to meet a customer's
    requirements. The Software Management Plan includes the *design
    planning information* and the *process management information*. It
    may be gathered together as a single document, consist of multiple
    documents, or be a portion of a more comprehensive document, such as
    a Project, Product, Implementation, or equivalent Plans. In the case
    where the product is the result of a software development or
    software maintenance, the Software Management Plan is the Product
    Plan.

-   **Team**: One or more qualified individuals assigned to a project,
    who work at the direction of the Team Lead in filling a customer's
    order. (This is equivalent to a Product Development Team )

-   **Team Lead**: A qualified individual assigned responsibility and
    given authority to ensure that the Team fills an order (This is
    equivalent to the Product Development Lead)

-   **Validation**: In design and development, validation is the process
    of examining the product to determine conformity with the user's
    functional requirements. This includes product inspections,
    functional and operational tests, and environmental simulations.

-   **Verification:** In design and development, verification is the
    process of examining the design to determine conformity with the
    documented design requirements. This includes reviewing
    documentation prior to release, performing alternate evaluations to
    verify the original analysis, and performing physical tests of
    hardware and operational tests of software.

ISO 9001 in Brief

An ISO 9001 compliant organization has in place a quality management
system that ensures quality is built into each of the processes
throughout the organization. This requires that we must:

![](media/image1.wmf)

**1.3 Business Development Process Overview**

The overall process, known as the **Business Development Process**
(Figure 1.1) is composed of a number of subprocesses, the first of which
is the Strategic Implementation Process. The output from the Strategic
Planning Process is the Information Systems Division (ISD) Strategic
Plan, which is, aligned with the AETD, GSFC and NASA strategic plans.
The high level ISD Business process consists of four main subprocesses.

-   Strategic Planning Process

-   Business Selection Process

-   Order Fulfillment Process

-   Assessment Process

[]{#refheading___toc462565016 .anchor}Figure 1.1 Information Systems
Division Business Development Process

This document focuses on the processes directly associated with
fulfilling a customer's product order.

The **Strategic Planning Process** continuously and systematically
reviews and evaluates ISD's mission, vision, and strategic goals; and
ISD's success in meeting organizational objectives and customer
feedback. Based on analysis, the ISD Management Team determines the
organization's future and focus for time, energy and resources. Also,
within this process, the ISD Management Team evaluates the
organization's progress in meeting the objectives, and assessing the
state of ISD's alignment with the NASA and GSFC's strategic objectives.
The output from this process is the updated ISD Strategic Plan that may
periodically validate the existing plan, or may revise the plan to
reflect changes and refinements in the strategic direction. The
Strategic Plan may be found on the web at
[http://isc.gsfc.nasa.gov/Intro/Intro.htm]{.underline}.

The **Business Selection Process** (Section 2) describes the processing
of a customer's request, and if accepted, selecting the Team and Team
Lead to complete the order. When a customer requests a specific product
of ISD, the ISD Management Team determines whether to accept the request
by using the Business Selection Process or to present alternatives to
the customer. The Business Selection Process also requires logging or
recording all customer requests, whether accepted or rejected. Recording
the request closes the process loop and serves as feedback into the
Strategic Planning Process. If the order is accepted, a Team Lead and
Team are selected to work with the customer to provide the requested
product using the Order Fulfillment Process.

The **Order Fulfillment Process** (Section 3) describes the processes to
produce the requested product, beginning with the logging of an order
and continuing through to delivery. This section also details the
contents of the Team Product Plan.

The **Assessment Process** (Section 4) is used to evaluate the product
development process and the products in order to improve both. The Team
provides assessments for Business Process, Software Methodology, and the
product. In addition, the customer supplies an independent assessment of
the process and product. Appropriate information from this process is
used to update the Product Development Handbook and is also used as
input to the Strategic Planning Process.

# 1.4 Operating Structure Overview {#operating-structure-overview .PDHHeading2}

-   ISD will form a Team to fill ALL customer orders (Note: A Team may
    consist of one person).

-   Product Plans are written for all products covered by this handbook.

-   Metrics for both the processes and products shall be collected and
    analyzed for project management, process improvement, and quality
    assurance.

-   Team progress is evaluated using metrics.

-   The customer and the Team agree upon content, formality, and number
    of reviews.

-   ISD Management Team will work with the Team Leads to ensure the
    product meets the customer's needs.

-   The Team Lead has responsibility, accountability, and authority for
    the quality of the product delivered, and the responsibility to
    identify issues and concerns to the customer and appropriate ISD
    Management Team member.

# 2.0 Business Selection Process  {#business-selection-process .PDHHeading1}

Overview

The ISD Management Team has the primary responsibility for initiating
and completing this process which begins with the reception of a request
for software from a customer and ends with the selection of a Team Lead
and Team to fill the order for those requests accepted (Reference Figure
2.1).

Roles and Responsibilities

Below are listed the roles and responsibilities associated with this
process. Team and Team Lead roles have been expanded upon in Appendix E.

ISD Chief

-   Ensure a Team Lead is identified.

-   Ensure the Team Lead and Team consists of qualified individuals.

ISD Management Team

-   Develop individual skills, Team skills and centers of expertise.

-   Select and provide qualified Team Leads and Team members.

-   Define the Team and Team Leads responsibility, authority and
    accountability for delivering quality products.

-   Document any "required" training needs or On the Job Training (OJT)
    of employees in accordance with GPG 3410.2.

-   Ensures that training received using funds other than GSFC training
    funds is documented either in the employee's official or unofficial
    personnel folder.

Team Lead

-   Meet with the customer to understand and document the product
    requested.

-   Present to the ISD Management an estimate of the staffing and skill
    levels, and approximate time commitments required to meet the
    customer's request.

-   Assess the skills and expertise needed. Work with the ISD Management
    Team to select an appropriate qualified Team to ensure that the
    individuals are qualified to perform their assigned
    responsibilities.

-   Maintain a list of any "required" training needed and received by
    team members

-   Assign work to Team members.

Location of Employee Qualification Records

-   Team Lead and Team members basic qualifications to perform their job
    derive from their meeting the criteria specified in their Position
    Description (PD) that are kept in their personnel folders.

-   Team Lead and Team members training achievements requested via GSFC
    training forms are kept in their training records in Office of Human
    Resources (OHR). All others are kept either in the employee's
    official (with the Office of Human Resources ) or unofficial (with
    supervisor) personnel folder.

-   Team Lead also maintains records of "required" training for Team
    members.

-   For those Team members or Team Leads who are new hires or have been
    reassigned after 1/4/99, On the Job Training will be documented by
    the supervisor per GPG 3410.2.

-   Specific QMS management and Team Lead requirements related to
    employee training and qualifications may be found in GPG 3410.2.

Ideal Output of Process Is:

-   Clearly defined communication channels

-   Full alignment with ISD/GSFC strategic objectives

-   Customer satisfaction

-   Quick response to request

-   Clear, mutual understanding of customer's goals and schedule

-   Flexible solution options with clear, unbiased tradeoffs

-   Timely, accurate customer profile and needs information for ISD
    planning purposes

![](media/image2.png){width="4.961805555555555in"
height="3.2465277777777777in"}

[]{#__RefHeading___Toc462565019 .anchor}Figure 2.1 Business Selection
Process

# 2.1 Record Request {#record-request .PDHHeading2}

-   Anyone, internal or external to the ISD, may request ISD products.
    Requests may originate formally, such as through the issuance of
    Statements Of Work, or informally through conversation and phone
    calls. Any ISD employee may act as this initial Point of Contact in
    such requests for products.

-   Initial product request documentation and subsequent initial ISD
    Management notification of such product requests is the
    responsibility of this initial Point of Contact employee.
    Documentation may take a range of forms, but generally an e-mail
    shall be formulated and forwarded to Branch management or higher.
    Such documentation shall include the name and organization of the
    individual requesting products, the date of the contact requesting
    these products, a description of the products requested, any
    information on schedule, staffing, funding, etc., and any other
    related information thought useful. In initial discussions the ISD
    employee shall outline the ISD process and set a target date for ISD
    follow-up contact back to the product request originator.

-   Following Branch/ISD management notification, the ISD Management
    Team shall select an ISD Point of Contact to iterate and clarify
    this product request with the originator.

# 2.2 Scope Request  {#scope-request .PDHHeading2}

-   The ISD Point of Contact shall further explore the product request
    with the customer and scope the work being proposed. This may
    involve the Point of Contact arranging meetings involving other ISD
    civil servants and/or contractors as appropriate. The Point of
    Contact shall document customer discussions and agreements in
    electronic form to serve as a basis for customer review, update, and
    concurrence.

-   The ISD Point of Contact shall work with the customer and the ISD to
    establish a top-level schedule and to provide a rough estimate of
    resources (people, money, facilities, etc.) needed to provide the
    requested products.

-   The ISD Point of Contact shall actively involve the ISD Management
    Team in the resolution of any identified issues, such as questions
    of feasibility, work appropriateness, resources, etc.

# 2.3 Select Work {#select-work .PDHHeading2}

-   The ISD Management Team holds the responsibility for agreements to
    provide customer products.

-   The ISD Management Team, supported by the designated ISD Point of
    Contact, shall decide whether to proceed with the work or reject the
    work, based on a range of factors, including ISD Strategic
    Implementation Plan & Yearly Action Plan criteria, request
    prioritization against other product needs, GSFC mission priorities,
    available resources, etc. (long sentence)

-   The ISD Management Team shall work with the ISD Point of Contact to
    briefly document the basis for the ISD "Go" or "No Go" decision.

# 2.4 Notify Customer {#notify-customer .PDHHeading2}

-   The Point of Contact, and when appropriate, an ISD Management Team
    representative shall meet with the customer to discuss and affirm
    the ISD "Go" or "No Go" decision.

-   When limitations within the ISD preclude the support desired to
    develop the requested product, the ISD shall work with the customer
    at exploring other alternative solutions.

# 2.5 Select Team Lead {#select-team-lead .PDHHeading2}

-   ISD Management Team shall be responsible for selecting the Team Lead
    to provide the product requested.

-   Depending upon the nature of the support needed, selection may be
    made through a variety of processes, including Branch level
    assignment, ISD wide e-mail interest solicitations, or formal
    personnel actions.

-   Customer participation shall be encouraged in the Team Lead
    selection process.

# 2.6 Document Order  {#document-order .PDHHeading2}

-   The Team Lead shall be responsible for documenting the details of
    the product request and the general plan for meeting this request in
    the framework of the Product Plan. Specific Product Plan sections
    required are: 1.3, 1.4, 1.5, 1.6, 1.11 and 1.12 listed in
    Appendix C. (The remaining sections of the Product Plan are
    completed and the plan signed in the Order Fulfillment Process.)

-   The order must be a signed Customer Agreement, which should either
    specify the requirements directly or refer to where the requirements
    are documented. This agreement could be accomplished by having the
    customer sign the Product Plan. In formulating this documentation,
    the Team Lead shall make maximum use of the ISD Point of Contact
    documentation and knowledge. Efforts in this Order agreement
    elaboration and negotiation shall involve other ISD employees and
    contractors as appropriate.

# 2.7 Select Team {#select-team .PDHHeading2}

-   The Team Lead and ISD Management Team share responsibility in Team
    staffing.

-   Team staffing, as for the Team Lead selection, may be made through a
    variety of processes, including Branch assignment, ISD-wide e-mail
    interest solicitations, or formal personnel actions.

-   Customer involvement in Team selection is generally more a matter of
    concurrence than active participation.

#  {#section .PDHHeading1}

# 3.0 Order Fulfillment Process  {#order-fulfillment-process .PDHHeading1}

Overview

The Team now has the responsibility for completing the Order Fulfillment
Process. This process begins with the continuation of detailed
communication with the customer to further refine the customer's needs
and expectations. These refinements are documented in the Product Plan.
This process is followed when the customer desires a tangible product.
The quality of the product and the effectiveness of the process is
dependent upon the iteration between assembling, building, fielding the
product and customer feedback. If the product is a software product or
the maintenance of a software product, use the processes for software
development or software maintenance in Appendices A and B, respectively.
This iterative process is illustrated in Figure 3.0.

![](media/image3.png){width="5.997222222222222in"
height="3.529861111111111in"}

[]{#__RefHeading___Toc462565028 .anchor}Figure 3.0 The Iterative Process
for ISD Product Development

Roles and Responsibilities

Below are listed the roles and responsibilities associated with this
process. Team and Team Lead roles have been expanded upon in Appendix E.

ISD Chief

-   Designates the ISD Management Team Member responsible for the
    approval of any Product Plan covered by this handbook. Unless
    specified otherwise in the Product Plan, this will be the supervisor
    of the Team Lead.

ISD Management Team

-   Ensure that approaches taken by the Team stay aligned with the ISD
    Strategic Implementation Plan

-   Ensure that the Team obtains necessary resources in a timely and
    effective manner

-   Reviews and approves the Team's Product Plan

-   Effectively remove barriers which impede a Team's progress

-   Ensure and defend the Team's empowerment rights throughout the
    process

-   Document the process for reviewing and changing Product Plans
    (customer agreements)

-   Facilitate support that the Team cannot provide themselves

-   Ensure customer inputs for the Team Lead's performance plans and
    evaluations are made and maintained (appropriate supervisor)

-   Ensure Team Lead's inputs for the Team performance plans and
    evaluations are made and maintained (appropriate supervisor)

-   Provide training to Team members as requested by the Team lead

-   Document any identified training needs of employees on Performance
    Plan

-   Document any required OJT and "required" training per GPG 3410.2

-   Allocate resources as requested by the Team and approved

Available Contracts Data Base Administrator

-   Maintain a data base of current contracting mechanisms (hardware,
    software, manpower) that have demonstrated acceptable performance

-   Details of the information and format for this database may be found
    in the Supplier Performance Records, GPG 5100.2

Team Lead

-   Manage the entire customer order fulfillment process from receipt of
    the initial customer objectives to the final fielding of quality
    products to the customer

-   Document each Team members roles and responsibilities for inclusion
    in the Product Plan

-   Provide performance assessment on Team members as requested by the
    ISD Management Team

-   Report appropriate information to customers and the designated ISD
    Management Team member in a timely and accessible manner

-   Determine and document any training required for Team members and
    work with the ISD Management Team to ensure that training needs are
    met and that only qualified people work in producing the product

-   Maintain a list of "required" training received by Team members

-   Work with the customer and the ISD Management Team to resolve issues
    and conflicts that are beyond control of the Team

-   Identify any internal or external consultants that may be needed

-   Identify and request resources that are not directly under the
    Team's control

-   Ensure that the Team maintains a customer focus throughout the
    process and aligns the Teams goals with the customer's objectives

-   Ensure the Team works with the customer to define project guidelines
    and reflects these guidelines in the Product Plan

-   Ensures the success of the Team in meeting customer requirements
    according to cost and schedule goals

-   Provide leadership in developing a cohesive, focused, motivated Team

-   Ensure Team personnel are implementing and following through on
    their responsibilities

-   Maintain documentation of key issues and decisions, and, optionally,
    supporting rationale or information

-   Produce and maintain a master list of documentation/information for
    the project and ensure it remains under Configuration Control. This
    is to include a:

```{=html}
<!-- -->
```
-   Product Plan (see Appendix C)

-   List Of All Processes Used

-   List Of All Quality Record Types (in format listed in GPG 1440.7,
    see App. B)

```{=html}
<!-- -->
```
-   Identify a Quality Records Custodian whose name will be recorded in
    the Product Plan and who has responsibility for control of the
    quality records associated with the Product Plan

-   Ensure Team personnel are implementing and following all of their
    responsibilities

-   In addition to the work processes and processes identified above,
    documents that appear on the master list must be controlled

-   Have a design plan and process management plan for your product
    development. Include these plans in the Product Development Plan.

```{=html}
<!-- -->
```
-   Identify design verification and validation activities and assure
    that they are executed and documented

-   Establish a Team review plan and ensure that all planned reviews are
    conducted

-   Document action items as Requests for Action (RFAs) and document
    responses to RFAs

```{=html}
<!-- -->
```
-   Ensure that the Team evaluates the need for statistical techniques
    required for developing or testing the product as well as the need
    for any statistical techniques required to analyze the product
    development process

```{=html}
<!-- -->
```
-   Document any processes used to implement or control the application
    of any statistical techniques identified above

```{=html}
<!-- -->
```
-   Identify and use a corrective action process

```{=html}
<!-- -->
```
-   The project corrective action process must address the following:
    customer complaints, actions resulting from audit findings, and
    major nonconformances (see GPG 1710.1)

-   Ensure that non-conformances meeting the criteria listed in GPG
    1710.1 are documented in the Center on-line NCR/CA database. This
    shall occur [after]{.underline} an initial acceptance tested version
    release of a system or subsystem to the customer or representative
    has occurred (accompanied by a release letter). For flight software,
    this is after initial acceptance testing completion on the flight
    software test bed.

-   The process must be written and controlled. It must be referenced or
    detailed in the Product Plan.

```{=html}
<!-- -->
```
-   Identify and use a preventive action process

```{=html}
<!-- -->
```
-   Maintain a Preventive Action List

```{=html}
<!-- -->
```
-   Define and use a Configuration Management process

```{=html}
<!-- -->
```
-   The project must have an explicit, written Configuration Management
    process. It may be by reference, but it must be explicit and define
    what items will be controlled, who has authority to make changes,
    how the status of changes will be handled, and what configuration
    checking (audit) processes will be used.

```{=html}
<!-- -->
```
-   Have a process for reviewing and approving documents

```{=html}
<!-- -->
```
-   All controlled documents and deliverables must be reviewed and
    approved. Projects must have a written process for completing this
    activity for documents controlled by the project.

-   All controlled documents and data must have evidence that they were
    reviewed and approved by authorized personnel. Further, changes to
    these must be reviewed and approved by the same
    functions/organizations that initially performed the activities.

```{=html}
<!-- -->
```
-   Have a written process for uniquely identifying and controlling all
    products

```{=html}
<!-- -->
```
-   Develop a system to identify product and track status of work being
    performed on product and provide a method for s/w that tracks the
    status of tests planned/test results/and test status by unique s/w
    version number or build identification and date.

-   All products must have a unique product identification name or
    number so that erroneous products or versions are not used by
    project personnel in work performance

-   Ensure that tests and inspections to be conducted are documented in
    test plans

-   Ensure that status of tests and inspections are documented in test
    database

```{=html}
<!-- -->
```
-   Ensure that test results are documented and that testing does not
    proceed past a planned test/inspection point until non-conformances
    have been documented

-   Every formal release shall be documented in a written release notice
    containing the version/release changes, including database changes,
    the requirements and the nonconformance information

-   Have a process for controlling incoming products

```{=html}
<!-- -->
```
-   Ensure that Receiving Inspection Instructions are prepared for
    incoming products where applicable and document on Work Order
    Authorization (WOA) (see GPG 4520.2) using GSFC Receiving Inspection
    & Test System (RITS)

-   Ensure that Receiving Inspection Instructions are executed and that
    any non-conformances are documented on WOA and on the appropriate
    non-conformance form (see GPG 5100.1)

```{=html}
<!-- -->
```
-   Provide input for evaluation for Supplier Performance Records (see
    GPG 5100.1)

-   Ensure that test equipment and test software is working properly and
    maintain records of this verification

-   Have a process for controlling and safeguarding any
    customer-supplied product

```{=html}
<!-- -->
```
-   Inform the customer and obtain authorization if repair or rework is
    required of the customer supplied product

-   Document any damage or malfunction of customer supplied product and
    report to the customer for disposition of the product

```{=html}
<!-- -->
```
-   Ensure that any product requiring handling, storage or shipping is
    processed according to the GPG 6400.1. Retain shipping records on
    Form 20-4.

Team

-   Defines their mission, vision and objectives based on their
    customer's objectives and ISD strategic implementation goals

-   Identify customer expectations and develop metrics to ensure that
    those expectations are met quantitatively and qualitatively

-   Establish effective communications mechanisms to facilitate the
    change process and to avoid misunderstandings

-   Identify customer supplied resources and information that will be
    needed for the successful completion of the project

-   Record effort and progress in the form of project metrics. Specific
    metrics are listed in Appendix G.

-   Know the GSFC Quality Policy and how it affects products and
    processes, including Team efficiency

-   Know where to find a master document list identifying the acceptable
    processes for performing the job

-   Have an immediate access to all current process required for the
    job. Relevant process for the job must be written, reviewed,
    approved (signed by an authorized person), available (you must have
    access), current (up to date), and followed

-   Know how processes used are improved or changed

-   Discard or appropriately mark all outdated controlled documents

-   Know where project schedules are maintained and what they are

-   Know where Team quality records are maintained and be able to
    produce them

-   Know how each unique version of a system or product is identified
    and controlled

-   Know how you are qualified to do your work, how training is offered,
    and how work assignments are made

-   Follow the processes that pertain to your job and maintain
    appropriate records

# 3.1 Understand Customer Needs and Establish Agreement {#understand-customer-needs-and-establish-agreement .PDHHeading2}

To deliver a quality product the Team must understand the needs and
requirements of the customer. To do this, the Team communicates and
builds consensus and agrees to the requirements. This agreement must be
established, written and updated in the Product Plan. In QMS
terminology, the Product Plan *IS THE QUALITY PLANNING DOCUMENT* for
producing the product. The table of contents for a Product Plan and
salient points are in Appendix C and include the customer agreement, the
management approach, the technical approach, and product assurance. The
length and level of detail of the Product Plan should be commensurate
with the scope and complexity of the project. Updates to the Product
Plan follow the same approval cycle as for the original agreement. For
software development and maintenance, the Product Plan is the Software
Management Plan. See Appendix A, Section 1.1.1.1 and Appendix B, Section
2.1.1.1.

There must be a signed Customer Agreement, which should either specify
the requirements directly or refer to where the requirements are
documented. This agreement could be accomplished by having the customer
sign the Product Plan. The Product Plan should also specify the elements
and extent to which the customer's processes are to be used, especially
in the areas such as reviews, non-conformance reporting, configuration
management, inspection and testing, etc.

# 3.2 Assemble/Build Product {#assemblebuild-product .PDHHeading2}

After producing the Product Plan, the Team begins the process of
producing the product according to the Product Plan. The process begins
with the Team to determine what organizational assets and information
are available to be utilized on the Project. This might include
technology assessments, Off-the-Shelf capabilities, lessons learned,
etc. The process ends with the release of a product to the customer,
whether it is a final delivery or an agreed-upon interim delivery with
partial capabilities. For software and software maintenance products,
refer to the processes in Appendix A and Appendix B, beginning with
Sections 1.2 and 2.2, respectively. The steps for assembling/building a
product are illustrated in Figure 3.1.

![](media/image4.png){width="5.993055555555555in"
height="5.329166666666667in"}

[]{#__RefHeading___Toc462565031 .anchor}Figure 3.1 The Assemble/Build
Process

# 3.3 Field Product {#field-product .PDHHeading2}

After building/assembling a release of the product the Team delivers the
product according to the Product Plan. Test support is provided, as
requested by customer, and might include such things as operational
scenarios, special test cases, end-to-end testing, spacecraft
integration and test, concurrent release testing, mission readiness
testing. After fielding, customer feedback is evaluated. This includes
both product and process assessments and metrics collection to be used
in improving Team performance. Following this assessment, new agreements
are reached as needed, and the process repeats again, beginning with the
Understand Customer Needs and Establish Agreement step. The steps for
fielding a product are illustrated in Figure 3.2.

![](media/image5.png){width="5.985416666666667in"
height="5.107638888888889in"}

[]{#__RefHeading___Toc462565033 .anchor}Figure 3.2 Fielding the Product

# 3.4 Deliver the Product {#deliver-the-product .PDHHeading2}

Once the customer has accepted the final product, any ancillary products
specified in the Product Plan are to be delivered to the customer. This
also triggers an orderly shut down of the project. Typical elements of
the Deliver the Product Process include:

-   Delivering final source code, users guide, test materials, and
    documentation, etc., to the customer

-   Packaging appropriate information, quality records, and lessons
    learned, as available, and moving them to a central storage site

-   Closing out contracts

-   Closing out resources (money, facilities, and people)

-   Transitioning the system to maintenance, as specified in the Product
    Plan

-   Transferring licenses and agreements, as required

-   Closing out Team Performance appraisal evaluations

-   Celebrating Team success

    For software products, see Appendix A, Sections 1.1.1.4-1.1.1.6 and
    Sections 1.6.1-1.6.2.3.

    For maintenance products, see Appendix B, Sections 2.1.1.4 and
    Sections 2.6.1-2.6.2.3.

The steps for delivering the product are illustrated in Figure 3.3.

![](media/image6.png){width="5.985416666666667in"
height="1.8416666666666666in"}

[]{#__RefHeading___Toc462565035 .anchor}Figure 3.3 Deliver the Product
Process

#   {#section-1 .PDHHeading1}

# 4.0 Assessment Process  {#assessment-process .PDHHeading1}

Assessments are done continuously throughout the lifecycle of the
project, from initial request to closeout. Assessments are made of the
Business Process, software process and products, and the product line.
These assessments are also used to update the ISD Product Development
Handbook as required, and to provide inputs to the Strategic
Implementation Planning Process.

Roles and Responsibilities

Below are listed the roles and responsibilities associated with this
process.

ISD Chief

-   Establish a continuous improvement process for improving the
    Business Development Process and the Product Development Handbook

ISD Management Team

-   Ensure the ISD Quality Manual (i.e., the Product Development
    Handbook) remains aligned with the GSFC Quality Manual

-   Review GSFC Quality Management System for recommended improvements

-   Ensure alignment of organization to the ISD Quality Manual

-   Define responsibilities and authority for delivering quality
    products

# 4.1 Business Process Assessment {#business-process-assessment .PDHHeading2}

ISD Management Team is responsible for assembling a team consisting of
selected ISD management and Team Leads, the ISD/Quality Management
System Representative, a Software Engineering Laboratory representative.
This team:

-   Collects feedback routinely from

```{=html}
<!-- -->
```
-   Teams

-   Customers

-   ISD Management Team

-   ISD/QMS Representative

```{=html}
<!-- -->
```
-   Assesses the feedback

-   Makes recommendations for improvement

-   Solicits comments from all ISD personnel and supporting contractors

```{=html}
<!-- -->
```
-   via the Web

-   via e-mail

```{=html}
<!-- -->
```
-   Finalizes recommendations to ISD Management Team

-   Incorporates approved recommendations into Product Development
    Handbook and Strategic Implementation Process

# 4.2 Software Methodology Assessment {#software-methodology-assessment .PDHHeading2}

The Goddard Space Flight Center Engineering Process Group (EPG), using
Team assessments and metrics collected during the project, provides
recommendations for improvements to the software development process.
The Software Methodology Process uses the following approach:

-   Understand, characterize, and document ISD's software processes and
    products

-   Identify candidate process changes, design studies based on ISD's
    business goals, and understand impacts and changes to the baselined
    process resulting from the proposed study

-   Identify appropriate metrics required for s/w process improvement
    activities

-   Collect, store, quality assure, summarize, and export the ISD's
    project data

-   Analyze the project data to develop and maintain organizational
    models (e.g., cost estimation models, effort profiles, error
    profiles)

-   Develop and update software engineering standards based on the
    experiences of the ISD Teams

-   Package results and provide derived information in useful forms such
    as guidebooks, tools, and training courses

-   Maintain the ISD's projects database, which contains historical data
    from the organization's various projects

-   Maintain the ISD's library of experience packages, including
    guidebooks, standards, handbooks, and reports

# 4.3 Product Line Assessment  {#product-line-assessment .PDHHeading2}

The ISD Management Team is responsible for assessing the product line:

-   With respect to:

```{=html}
<!-- -->
```
-   What customers want

-   The ISD strategic plan

-   The commercial market

```{=html}
<!-- -->
```
-   With respect to the minimum requirements of:

```{=html}
<!-- -->
```
-   Customer satisfaction

-   Cost

-   Schedule

-   Burden of maintenance

-   Commercial product availability

# 4.4 Product Development Handbook Assessment {#product-development-handbook-assessment .PDHHeading2}

-   Update handbook as required using ISD Configuration Management
    process

#  {#section-2 .PDHHeading1}

# 5.0 ISD Crosscutting Processes {#isd-crosscutting-processes .PDHHeading1}

This section provides information on three additional processes which
are used and controlled at the ISD level (versus Team level). They are
not part of the six basic ISD Business Processes, but are used in
support of them in a crosscutting manner.

# 5.1 Configuration Control Process (Items include Quality Records, Documentation, and Data) {#configuration-control-process-items-include-quality-records-documentation-and-data .PDHHeading2}

Roles and Responsibilities

ISD/QMS Representative

-   Represent ISD at the Directorate level in all matters concerning the
    GSFC Quality Management System and ISO 9001 certification,
    implementation, and maintenance

-   Direct all activities pertaining to the Quality Management System or
    ISO 9001 within ISD

-   Overall responsibility for maintaining the ISD Product Development
    Handbook

-   Review newly developed Team Processes specified in the Product Plan
    for ISO 9001 compliance

-   Overall responsibility for adding/deleting/maintaining the ISD
    Library of Approved Team Processes

ISD Document Configuration Manager

-   Maintains a list of ISD level quality records and configuration
    controlled items, date or identification number/letter, their
    locations and their owners

-   Obtains approval for deletion of items from controlled list from the
    ISD Management Team

Process Description

-   Quality record, documents, and data which are held and controlled at
    the ISD Center level are managed by the Document Configuration
    Manager

-   When an individual desires to place a new document under CM that
    individual request permission via email from the ISD Chief

-   Once approved, the ISD Management Team assigns an owner who is
    responsible for maintaining the history and current version of the
    item

-   A list of all controlled items, their owners, their latest version
    date and their locations is maintained by the Document Configuration
    Manager

-   The owner is responsible for ensuring that all information under his
    control is current, legible, readily retrievable and safe from
    damage/loss

-   Deletion of items from the ISD level of control can be made by
    anyone through the owner to the Document Configuration Manager. The
    Document Configuration Manager will then request permission from the
    ISD Management Team and will proceed accordingly.

-   The Configuration Control Board for a document is determined by the
    ISD Management Team, and consists of those people identified on the
    document signature page and stated accordingly in the document under
    change control procedures

-   Change control authority is determined by the ISD Management Team
    and is indicated by those people/positions on the signature page of
    each document

-   For items under configuration control requiring updates, the
    following change control process will be used:

```{=html}
<!-- -->
```
-   Changes to any item may be submitted by anyone to the owner of the
    configured item. This shall include an assessment of the cost and
    schedule impact, and changes required to other configuration
    controlled items.

-   The owner shall log the change request and prepare the appropriate
    document updates

-   The owner will communicate with other owners affected by the item
    and any other personnel deemed appropriate in order to negotiate the
    acceptability of the change

-   The owner will document the negotiated changes required by any other
    owners and distribute them to the owners and Document Configuration
    Manager

-   Individual owners affected by this change are responsible for
    updating their respective configuration controlled items to reflect
    negotiated changes

-   The owner shall make the appropriate changes in the configuration
    controlled copy of the document and update the document date

-   Notification of items that have been updated are to be sent via
    email to the Document Configuration Manager and to all affected
    personnel by each owner. The Document Configuration Manager shall
    retain the email notifications of the document updates.

# 5.2 Process for Adding, Deleting or Modifying Approved Team Processes in the Library for Use in Product Plans {#process-for-adding-deleting-or-modifying-approved-team-processes-in-the-library-for-use-in-product-plans .PDHHeading2}

The Library of Approved Team Processes is maintained on the web at
[http://isc.gsfc.nasa.gov/ISO9k/ISO9001.htm](http://isc.gsfc.nasa.gov/Iso9k/ISO9001.htm)
by the ISD/QMS representative, and is under ISD configuration control.

Roles and Responsibilities

ISD/QMS Representative

-   Review newly developed Team Processes specified in the Product Plan

-   Overall responsibility for maintaining the ISD Library of Approved
    Team Processes

Library Modification

-   Anyone can make a recommendation for adding, deleting or modifying a
    reference

-   The recommendation should be written up and should include the
    process to be added, deleted or modified, the name of the process
    for which it is an alternate (relative to the Product Plan Table of
    Contents), and a short description of the merits of the approach

-   The recommendation should be submitted to the ISD/QMS Representative
    for an assessment of compliance with QMS (ISO 9001)

-   Once approved, the ISD/QMS Representative is responsible for
    ensuring that the new process is included in the reference library
    for approved use

-   The ISD/QMS Representative shall notify the ISD Document
    Configuration Manager to update the documentation in the ISD
    Configuration Controlled Items database

-   The ISD/QMS Representative shall notify all of Code 580 and its
    support contractors of the update via e-mail

# Appendix A: Software Development Process {#appendix-a-software-development-process .PDHHeading1}

This process description is intended to be used as a starting point in
defining the project\'s technical approach. The activities are not
intended to be strictly time sequenced. This allows them to be used to
define activities in a variety of project lifecycles. For example, a
project that uses an incremental build lifecycle may use the
Implementation and Integration activities many times, while a small
project with well-known requirements may use the waterfall lifecycle and
execute each applicable activity only once. There may be activities that
are unique to the project to mitigate special risks or to handle special
requirements. These may be added to meet the project\'s unique needs.

For administrative or institutional software, the responsible Director
of may grant waivers to these requirements, using the process described
in GPG 8730.3. Note that any software used in any flight or ground
system is not considered institutional.

## 1.1 Cross-Cutting Activities 

### 1.1.1 Team Management 

#### **1.1.1.1 Develop a Software Management Plan**

The Product Development Lead (PDL) shall develop a Software Management
Plan (SMP) using the Software Management Plan Guide in the Attachment to
this GPG. [GSFC Form
19-21](http://gdms.gsfc.nasa.gov/gdms/plsql/masterlist.menu) provides a
template for the recommended SMP outline. (Note: When using [GSFC Form
19-21](http://gdms.gsfc.nasa.gov/gdms/plsql/masterlist.menu),
information can be rearranged or supplemented as long as required
information is included.) The completed SMP describes the customer
requirements and the resources (e.g., budget, schedule, and staffing)
allocated to execute the Plan. The Plan defines the approach that will
be used to track project status, for configuration management, product
assurance, risk management, and independent verification and validation.
It also summarizes the technical approach including the review plan,
documentation to be produced and the criteria used to tailor this
Process for Software Development. Tailoring is determined by the PDL
with the concurrence of line management and the customer (typically a
project manager or principal investigator).

Tailoring factors include project characteristics such as the
criticality of the application, the size of the PDT and user community,
the degree of reuse, and other project specific factors. Required
activities are designated as \"shall\" and may not be deleted. Best
practices from past experience are designated as "recommended" and
should be thoughtfully considered for adoption by the PDL.

GPG 8700.4 and GPG 8700.6 define the procedures and guidelines for
required mission-level reviews and their applicability. The software PDT
participates in, or contributes material to, required mission-level
Project or Program reviews (e.g. EPR\'s or IIR\'s), as required.

The PDL shall define, with the participation of line management, Product
Manager, and the customer, an appropriate set of reviews as a resource
to increase the probability of success. Reviews may be combined to
improve value or efficiency. However, when reviews are combined, review
objectives from each shall be addressed to the level of detail required
for the individual reviews. The following reviews for software
development projects should be considered:

-   Software System Concept Review (SCR)

-   Software System Requirements Review (SRR)

-   Software Specification Review (SSR)\*

-   Preliminary Design Review (PDR)\*

-   Critical Design Review (CDR)

-   Acceptance Test Readiness Review (ATRR)

-   Operational Readiness Review (ORR)\*

    **\* Required for all projects**

    All software PDT reviews, above, shall:

```{=html}
<!-- -->
```
-   Be scheduled with published agendas.

-   Have a review team chair and other members with appropriate
    expertise that are independent of the PDT, project and immediate
    line management

-   Include customer representatives

-   Record meeting notes

-   Collect Requests for Action (RFA\'s)

-   Record and track RFA's to resolution with the independent review
    team.

-   Report results to the Project Manager and the Code 301 Integrated
    Review (System Review) Team Chair in the case of an activity that is
    part of a larger system.

#### **1.1.1.2 Manage the Task**

The PDL negotiates project-staffing assignments, assigns technical work
to staff members, and helps assure that all PDT members are qualified to
perform their assigned duties. The PDL is also responsible for the
day-to-day management of the task according to the SMP. The PDL shall
periodically review, record and report the status of system design,
development, implementation, and testing using the methods defined in
the SMP. It is recommended that project status be an objective measure
of work products accomplished against planned resource allocation (e.g.,
schedule, cost and effort).

#### **1.1.1.3 Collect Metrics**

Project metrics, at a minimum, shall include schedule planned vs. actual
dates, budget (effort and cost), product size, and product error
information, such as open/closed nonconformances (i.e., discrepancy
reports), and shall be collected periodically (e.g., monthly or
quarterly) as defined in the SMP. Project metrics are analyzed and the
results used to initiate process improvement activities.

#### **1.1.1.4 Document Lessons Learned**

The PDL shall query the NASA Lessons Learned Information System (which
is maintained at [http://llis.nasa.gov](http://llis.nasa.gov/) ) and
other knowledge resources, as appropriate, to access relevant past
experiences and knowledge that can be leveraged to reduce risk, improve
quality and efficiency. These queries shall be conducted at the
beginning of, and then periodically throughout, the software development
lifecycle. The PDL shall also submit significant lessons learned to the
web-based LLIS throughout the product lifecycle, as appropriate.

It is recommended that lessons learned be documented at the end of each
phase in a Software Development History. The Software Development
History is updated to include lists of the products produced, milestones
and key events, phase duration, key decisions, problems encountered, and
summaries of the metrics collected during the phase. It describes the
specific lessons and recommendations that pinpoint the major strengths
and weaknesses of the process used and the product itself, with
particular attention to planning, requirements, development, testing,
CM, QA, and new technology.

#### **1.1.1.5 Prepare and Maintain Project Documentation**

The PDT shall design, prepare, and maintain the documents as specified
in the SMP and consistent with higher-level Project requirements. The
PDT ensures that each document is reviewed, that the changes identified
during the review are properly implemented and that appropriate approval
signatures are obtained. Well in advance of delivery, the PDT shall
complete GSFC Form 1679, "New Technology Report;" this form allows the
commercialization office to make an assessment of the software before
delivery. It is important not to wait until delivery to complete this
form, so that the software may be released in a timely fashion.

#### **1.1.1.6 Transition to Maintenance Organization**

At the conclusion of the project the completed system may be
transitioned to the internal or external customer, or another
organization, for ongoing operations and maintenance. To protect
Government-owned intellectual property, this transition should take
place in accordance with the provisions of NPD/NPG 2210.1 where
applicable (see section 1.6 below). If such a transition occurs, the PDT
shall perform training and maintenance activities as defined in the SMP
and shall prepare the system and documentation for final delivery.
Configuration management, quality assurance and delivery activities are
performed as for delivery of a build/release. Project records are to be
stored, transferred or disposed of according to the SMP and NPG 1441.1C.

### 1.1.2 Training

###  

#### **1.1.2.1 Identify and Document Required Training Needs** 

The PDL shall identify and document the QMS-required training needed by
PDT members. Required training includes familiarizing PDT members with
the SMP, the methodology, standards, and design process used,
maintenance of PDT records, and use of a nonconformance recording
system. It also includes training required when working in proximity to
instruments or spacecraft, such as Electrostatic Discharge Awareness
Training, Range or Launch Safety, Laser Safety, etc. The PDL works with
the appropriate supervisors to assure that the identified training is
provided. It is recommended that the PDL identify any special
developmental training such as language and tool training, and identify
on-the-job-training to build an adequate level of understanding in all
PDT members

#### **1.1.2.2 Record Training Received**

As PDT members obtain training, the PDL shall maintain records of the
training that has been received and by whom (see GPG 3410.2).

### 

### 1.1.3 Configuration Management (See GPG 1410.2)

#### **1.1.3.1 Perform Configuration Identification**

The PDT shall identify, in the SMP, the types of items to be placed
under configuration control and shall identify when they will be placed
under configuration control. Items that are to be controlled at the
higher-level Project shall be identified. The PDT defines baselines for
major stages in system development, e.g., requirements baseline, design
baseline. The PDT identifies the products comprising each baseline, down
to the level of the smallest controllable unit. The PDT shall develop a
unique identification for each planned system build/release and
delivery. The PDT defines naming and labeling conventions as
appropriate. It is recommended that the PDT clearly identify reused
software components and that the identification scheme distinguish
software components that have been modified from those that are being
reused without modification.

#### **1.1.3.2 Maintain Configuration Control**

The PDT shall preserve the integrity of all system baselines,
components, and products that are not under higher-level Project
control. The PDT shall track changes to controlled products to assure
that the configuration of all identified items is known at all times.
Controlled products include system baselines, project documents
(requirements, design, test plans, etc.), source code, object code,
released and developmental versions of the target system, critical test
software and any customer-provided product (e.g., government off the
shelf \[GOTS\]) used in the development of the software system. The use
of a commercial configuration management tool is recommended for
configuration control of the project's permanent source code libraries.
The PDT shall maintain records of all software configuration management
activities performed, including all changes made to software and
documents under configuration control.

#### **1.1.3.3 Monitor and Report Configuration Status**

It is recommended that the PDT periodically produce configuration status
reports. These reports typically include such information as the number
of changes made to date, the reason for each change, the number of
system releases to date, the functionality provided with each release,
and the latest version and revision identifiers. It is recommended that
the first configuration status report list configuration items for any
components that are to be reused. This report is reviewed against design
documentation and configuration records from the system that is the
source of the software to be reused to ensure that all components are
accounted for and of the proper version.

### 1.1.4 Quality Assurance

#### **1.1.4.1 Support Project Reviews, Walkthroughs, and Inspections**

Quality reviews address completeness, readability, traceability, and
conformance to PDT standards. They assess both the quality of the
product (the deliverable under review) and the quality of the process
(the way the review is conducted, including planning, announcement,
preparation, conduct, and follow-up). Problem areas are identified and
appropriate corrections suggested. Although quality is a shared
responsibility, it is recommended that the responsibility for advocating
and assessing the quality assurance process be centralized in a single
individual. It is recommended that product quality be addressed during
all major reviews (requirements reviews, design reviews, etc.). It is
recommended that quality also be addressed during in-process reviews
(walkthroughs, inspections, test reviews, etc.). It is recommended that
inspection criteria for any customer-provided product (e.g., GOTS) used
by the PDT be established in conjunction with the customer. Results of
all reviews and inspections, including those of incoming
customer-supplied products, shall be documented.

#### **1.1.4.2 Review Project Deliverables**

It is recommended that all major project deliverables (documents,
software releases, presentations, etc.) be reviewed for quality. It is
recommended that a sample of minor deliverables (unit designs, unit
code, test plans and results, etc.) be reviewed for quality.

#### **1.1.4.3 Monitor and Manage Risk**

The Risk Management process is identified in the SMP. Risks shall be
continuously identified, analyzed, planned, tracked, controlled,
communicated and documented. A detailed discussion of the NASA process
and resources for Continuous Risk Management is found in NPG 7120.5
"NASA Program and Project Management Processes and Requirements."

#### **1.1.4.4 Monitor and Support Problem Identification and Corrective Action**

The PDT shall employ and follow a problem identification and corrective
action process consistent with the Goddard requirements. If a minor
Nonconformance Reporting /Corrective Action (NCR/CA) system is used it
should include the version or release number where the problem was found
and, ideally, the version number that includes the corrections. The PDT
shall record the number and severity of defects and nonconformances and
the corresponding corrective and preventive actions; this information is
used for process improvement. The PDT shall identify and segregate any
nonconforming builds/releases, or deliveries. When, with customer
concurrence, software is delivered with known nonconformances, the
release letter clearly identifies the nonconformances. Software with
nonconformances that has not been approved for delivery is identified
and maintained separately so that it is not inadvertently included in a
delivered product. The PDT shall document any nonconformances for
customer-supplied products (e.g., GOTS). After product delivery major
nonconformances which meet the criteria specified in GPG 1710.1 shall be
recorded in the in the Center NCR/CA system.

#### **1.1.4.5 Control Documents and Records**

The PDT shall control all of the records identified in the SMP. The PDT
shall document and implement a process for the identification, review,
approval, distribution, retention, and disposition of documents and
records.

#### **1.1.4.6 Perform Configuration Audits**

Prior to each release or delivery, it is recommended that an audit of
the software be performed. The audit verifies that every delivery item
(e.g., program, input file, test software, or document) is as reported
in the delivery documentation and release letter. Each item on the
delivery list is checked to ensure that the item is present, is
complete, is the correct version, is in the specified delivery format,
and is correctly identified. (Additional requirements apply for external
releases or deliveries; refer to 1.6.1 for additional information.)

#### **1.1.4.7 Support IV&V Activities as Required**

The PDL shall provide support to the Project for the submission of the
IV&V criteria information listed at
[http://swmetrics.nasa.gov/ivvcrit/.](http://swmetrics.nasa.gov/ivvcrit/)
If it is determined that the project requires IV&V support, the PDT
shall provide the required support to the IV&V facility for these
activities throughout the life of the project.

**1.1.4.8** **Comply with Specific Electrical Engineering Division and
Project Requirements for Connecting to Flight Hardware**

The ISD PDL shall insure that any ISD non-flight device connecting to
flight hardware shall be developed and maintained in compliance with the
Electrical Engineering Division (EED) and any specific Project policies
and practices (MASIS for HST, for example). The PDL shall also insure
that all civil servants and support contract staff equipment interfacing
with flight elements for purposes of software/firmware debug and update
comply with EED and specific Project direction.

## 1.2 Requirements Activities

### 

### 1.2.1 Software/System Concept Definition

#### **1.2.1.1 Develop a System, Software and Operations Concept** 

It is recommended that the PDT develop system, software and operation
concepts if they do not already exist. In performing this activity the
PDT formulates overall concepts for the system and the software by
examining customer needs, looking for similarities in previous missions
or systems, and identifying existing software, including commercial off
the shelf (COTS) or GOTS that could be used or reused. The system and
software concepts take the form of a high-level conceptual architecture.
Operations concepts take the form of scenarios that show how users
operate the system for each major operational mode.

Where software is to be reused this activity makes use of existing
documentation. Where the architecture is reused and a significant
portion of the code is planned for reuse, a goal of this activity is to
understand the similarities and differences between the system to be
reused and the new system. The PDT validates the software concept of the
system selected for reuse against customer needs. COTS/GOTS in the
existing system are reviewed for continued suitability. Where gaps,
unneeded elements, and areas requiring significant modification are
identified the PDT looks for additional compatible existing software
from other sources, including COTS/GOTS that could be used or reused.
Effort is concentrated on functionality that is not met by the system to
be reused and areas where significant changes are anticipated.

#### **1.2.1.2 Hold a Software System Concept Review**

It is recommended that the PDT hold a SCR, or participate in a
higher-level Project SCR. The purpose of the SCR is to bring together
experts independent of the PDT, project and immediate line management,
customers and other interested parties to examine and influence the
proposed system and operations concept before detailed requirements are
written. If the SCR is held, the PDL shall collect RFA's and shall track
RFA responses and resolutions.

**SCR Topics:**

-   Top level requirements (e.g., customer needs, problem description,
    higher-level Project requirements, anticipated software safety
    requirements)

```{=html}
<!-- -->
```
-   High level, \"ideal,\" architecture, identification of components to
    be reused

-   Identification of major external interfaces

-   Operations concepts for major modes of operation

    For a system with significant reuse, and where both the PDT and the
    customers have experience with the system to be reused, it is
    recommended that the focus of the review be on similarities and
    differences between the system planned for reuse and the target
    system. The review helps to determine if the gaps have been properly
    identified and addressed.

### 1.2.2 Requirements Definition

#### **1.2.2.1 Define and Document High-Level Requirements**

If the customer has not provided high-level functional/operational
requirements, the PDT shall define and document the high-level
requirements based on the system concept. The PDT works with the
customer to define the system and high-level requirements and to derive
requirements down to the subsystem (or equivalent) level. The PDT
defines internal and external software interfaces, and determines
performance and reliability requirements.

#### **1.2.2.2 Conduct a Software System Requirements Review** 

It is recommended that the PDT hold a SRR or participate in or
contribute to a higher-level Project SRR**.** At the SRR the high-level
requirements and specifications are presented (subject to appropriate
non-disclosure agreements) to experts independent of the PDT, project
and immediate line management, customers, users, and other interested
parties (e.g., managers and subject area experts). If the SRR is held,
the PDL shall collect RFA's that result from the SRR and track RFA
responses and resolutions. The PDT incorporates feedback from the review
into the requirements document(s) and places the documents under
configuration management. If the high-level requirements have been
provided, the SRR serves as a forum for the PDT to demonstrate its
understanding of the requirements and to identify any TBD\'s.

**SRR Topics**

-   Detailed review of requirements, concentrating on critical
    requirements, including software safety requirements

-   Traceability of requirements to customer source documents

-   Performance, interface and derived requirements.

-   Identify risks, TBD\'s and required resolution dates

#### **1.2.2.3 Analyze Requirements for Omissions, Contradictions and Clarifications**

The PDT shall analyze the high-level requirements. Conflicting,
ambiguous and infeasible requirements are identified. The PDT classifies
each requirement into one or more categories such as, \"mandatory\",
\"needs clarification,\" \"information only,\" etc. Use of the Automated
Requirements Measurement (ARM) Tool developed by the Software Assurance
Technology Center is recommended. More information about the tool can be
found at <http://satc.gsfc.nasa.gov/tools/arm/index.html>.

#### **1.2.2.4 Derive and Document Detailed Requirements and Specifications** 

The PDT uses appropriate requirements analysis techniques (e.g.,
object-oriented or functional decomposition techniques) to analyze the
requirements. At this stage, the PDT focuses on what the software needs
to do rather than how to do it. Software operations concepts are refined
to a more detailed level and reports, displays, etc. are defined. The
PDT shall identify and document detailed requirements and specifications
by identifying the primary input types and formats, and output products
needed to satisfy the requirements. The PDT shall produce a detailed
specification of the technical requirements for the software product.
The form and media for the specification are at the discretion of the
PDL. Good requirements are clear, complete, consistent, testable, and
traceable to customer requirements. The PDT shall document
software/hardware interfaces and interfaces among subsystems developed
by different teams in interface control documents (ICD\'s) as needed
(e.g., telemetry formats).

#### **1.2.2.5 Conduct a Software Specifications Review (a.k.a. Software Requirements Review)** 

The PDT shall conduct a SSR to present the detailed software
requirements, specifications and operations concepts presented (subject
to appropriate non-disclosure agreements) to experts independent of the
PDT, project and immediate line management, customers, users, and other
interested parties. The PDL shall collect the RFA's that result from the
SSR and track RFA responses and resolutions. The PDT incorporates
feedback from the review into the requirements, interface and related
documents.

**SSR Topics:**

-   Detailed software requirements

-   Software operations concepts

-   Selection of COTS components

-   Allocation of requirements to components to be reused and/or COTS or
    GOTS components

-   Hardware that may need to be built

-   Traceability of detailed requirements to high-level requirements

-   Definition of data in external interfaces

-   Definition of data transformations

-   Initial allocation of requirements to builds (i.e., build plan)

-   Risks and prototype plans or results

-   Safety critical issues

-   External Dependencies

-   Identification of TBD's and required resolution dates

    For projects where there is extensive reuse, the SSR concentrates on
    the new and modified components. Details of requirements for the
    components to be reused without modification are presented in enough
    detail to provide context.

#### **1.2.2.6 Obtain Customer Concurrence on Requirements**

The PDT shall obtain customer concurrence on the requirements. This is
often achieved through a formal review, such as an SSR, and includes
resolution of RFA's generated by the review. Alternative means of
achieving concurrence include a document review and approval cycle or a
series of requirements walkthroughs where comments are recorded and
incorporated as appropriate. When customer concurrence is achieved the
resulting requirements shall be placed under configuration control.

### 1.2.3 COTS/GOTS Evaluation and Selection (where applicable)

#### **1.2.3.1 Validate Existing COTS/GOTS Selections**

For projects with extensive reuse, it is recommended that COTS or GOTS
components be validated to determine if they adequately meet the
requirements of the new system. In some cases only upgrades or new
versions are needed.

#### **1.2.3.2 Determine the Selection Criteria for COTS/GOTS** 

The PDT shall identify and document, as a matrix or checklist, the
attributes that will be used to evaluate potential COTS/GOTS for use in
the system. Appropriate criteria include: the degree to which the
COTS/GOTS provides required functionality; the estimated total cost of
using and maintaining the product vs. the cost of developing and
maintaining the same functionality over the full life cycle; the
familiarity of the PDT with the product and the cost of the learning
curve; and vendor considerations such as stability and customer support.

#### **1.2.3.3 Identify Potential COTS/GOTS Candidates**

It is recommended that the PDT select a small number of COTS/GOTS
candidates for in-depth evaluation. Identification is accomplished using
vendor documentation, industry review and ratings, and previous
experience. (Note that in most cases, open competitive procurement of
COTS products is required. See GPG 5100.1 for additional information on
procurements.)

#### **1.2.3.4 Obtain Demo Versions and Evaluate the Candidate COTS/GOTS**

It is recommended that the PDT obtain evaluation versions of candidate
COTS/GOTS**.** The PDT works with vendors to obtain demo versions of
candidate products. If demo versions are not available, the PDT works
with the vendor to understand how the product meets the project needs.
The PDT uses the selection criteria to evaluate the candidates.

#### **1.2.3.5 Document the Selection Criteria and Evaluation Results**

For each product that is evaluated, the PDT shall document the results
of the evaluation against the selection criteria. The results of the
evaluation and selection are reported at the SSR. The report provides a
record that shows the key factors in the evaluation and provides insight
into the decision-making.

#### **1.2.3.6 Document the Risks Associated With the Use of the Selected COTS/GOTS**

It is recommended that the COTS evaluation identify potential risks
(cost, schedule, reliability, etc.) posed by the selected COTS/GOTS
components. Acceptable risk thresholds and mitigation strategies are
identified.

## 1.3 Design Activities 

### 

### 1.3.1 Procurement 

#### **1.3.1.1 Procure and Install Hardware, Software, and Firmware in the Development Environment** 

The PDT procures and installs hardware and COTS/GOTS in the development
environment. It is recommended that any software that will be reused be
obtained and installed in the development environment. Software to be
reused shall be placed under configuration management when it is
installed in the development environment. These efforts often extend
into implementation.

### 1.3.2 Prototyping 

#### **1.3.2.1 Perform Prototyping to Reduce Risks** 

It is recommended that the PDT develop prototypes to mitigate risks such
as requirements uncertainty, new hardware, a new development language or
environment, or stringent performance requirements. Prototypes to
resolve requirements uncertainty often take the form of functional user
interface mockups and occur in parallel with requirements definition
through to implementation. Prototypes involving COTS/GOTS to evaluate
their ability to meet functional requirements may occur in parallel with
requirements analysis. Prototypes to investigate performance risks or to
evaluate new technology often occur in parallel with design activities.

#### **1.3.2.2 Document the Prototyping Effort**

If the prototype is large or lengthy or if the risks under investigation
are significant it is recommended that planning, development, and
management of the prototype be documented in a prototype plan. A
prototype plan describes the purpose and goals of the prototype, the
resources allocated to the prototype, and the prototyping method and
procedures. It is recommended that the results of the prototype be
documented in an update to the prototype plan.

### 

### 1.3.3 Preliminary Design

#### **1.3.3.1 Refine Operational Scenarios**

Operations concepts or operational scenarios are received from a
higher-level Project or are developed as part of system concept
definition. It is recommended that operational scenarios be refined or
developed from operations concepts. Additional detail and clarifications
are added. Scenarios are decomposed into lower levels of detail. Screen
layouts and report formats are defined, as are preliminary interface
dialogs for both human-computer interfaces (HCI) and systems interfaces.

#### **1.3.3.2 Perform Performance Modeling**

If the system has performance requirements that are anticipated to be
difficult to meet, it is recommended that performance
sizing/modeling/measurement be performed.

#### **1.3.3.3 Prepare High-level Architecture Diagrams**

The purpose of preliminary design is to define the high-level
architecture that best satisfies the requirements and specifications.
The PDT uses appropriate analysis techniques (e.g., object-oriented
techniques or functional decomposition) to analyze the requirements. The
PDT evaluates design options, weighing choices according to system
priorities, such as performance, usability, reliability, or
maintainability. The PDT shall generate high-level diagrams of the
selected architecture, including any COTS/GOTS and glue-ware. The PDT
pays special attention to interfaces among COTS/GOTS and custom software
elements.

For high-reuse projects the PDT starts with the high-level architecture
of the system to be reused. If there is not adequate high-level design
documentation for the system to be reused it is recommended that the PDT
create this documentation.

#### **1.3.3.4 Design and Document High-level Functions and Specifications**

The PDT shall design and document the principal, or critical, modules in
the system. The PDT selects an appropriate analysis and documentation
approach to accomplish this (e.g., use cases and package specifications,
or prologs and program design language). The PDT prepares specifications
for the principal components in the system. For projects with extensive
reuse the PDT starts with high-level functions and specifications of the
system to be reused, reviewing them to ensure that the requirements of
the new system are adequately met. Modifications are made as required
and new functions and specifications are developed to meet new
requirements.

#### **1.3.3.5 Verify the High-level Design**

The PDT shall verify the high-level design. The PDT shall document the
procedures it uses for this purpose. Appropriate verification activities
include walkthroughs and/or PDT level inspections of the high-level
design documentation (e.g., design diagrams, use cases, package
specifications, etc.). Follow up actions shall be documented and tracked
to closure.

#### **1.3.3.6 Conduct a Preliminary Design Review** 

The PDT shall hold a PDR to present the system design (subject to
appropriate non-disclosure agreements) to experts independent of the
PDT, project and immediate line management, customers, users and other
interested parties (e.g., managers and experts). The PDL shall collect
RFAs that result from the PDR and track RFA responses and resolutions.
The PDT incorporates feedback from the review into the preliminary
design documentation.

**PDR Topics:**

-   Design

-   Design alternatives and identified \"ideal\" design (hardware &
    software)

-   Design drivers

-   Safety critical issues

-   Software hazard reduction features

-   Risks and prototype results

-   Lessons learned (previous lessons applied and new ones learned)

-   Reuse/COTS tradeoffs

-   Testing strategy

-   Traceability of requirements to design elements

-   Size estimates and schedule

-   Staffing plans

-   Updated build plan

-   Issues and TBD's, along with resolution dates

    For projects where the architecture of another system is being
    reused, the PDR concentrates on new functions and modifications to
    the existing architecture.

### 

### 1.3.4 Detailed Design 

#### 

#### **1.3.4.1 Develop and Document the Low-level Design** 

The purpose of low-level design is to complete the design so that it
satisfies all the requirements and can be directly implemented in code.
It is recommended that the PDT prepare design documentation to the
lowest level of detail of the system to produce "code-to" specifications
for each module. The PDT selects an appropriate analysis and
documentation approach to accomplish this (e.g., use cases and package
specifications, or prologs and program design language). The PDT refines
the operations scenarios for the system and completes the design of
input and output formats, such as displays, reports, and databases.

For projects where there is extensive reuse it is recommended that the
PDT use the same analysis and documentation approach as was used for the
system that is being reused. This facilitates understanding of the
design of the new system as a whole. If there is not adequate low-level
design documentation for the system to be reused it is recommended that
the PDT create it to the level needed to understand the new system. It
is recommended that code-to specifications for modules to be modified be
produced. Code-to specifications for modules to be reused without
modification need not be produced.

#### **1.3.4.2 Verify the Low-level Design**

It is recommended that the PDT verify the low-level design. If PDT level
verification of the low-level design is performed, the PDT shall
document the procedures it uses for this purpose. Walkthroughs and/or
PDT level inspections of the emerging designs are appropriate
verification activities. Follow up actions shall be documented and
tracked to resolution. The low-level design of modules to be reused
without modification need not be verified.

#### **1.3.4.3 Conduct a Critical Design Review** 

It is strongly recommended that the PDT hold a CDR and participate in or
contribute to a higher-level Project CDR**.** The PDT presents the
completed design (subject to appropriate non-disclosure agreements) to
experts independent of the PDT, project and immediate line management,
customers, users, and other interested parties (e.g., managers and
experts). Participants evaluate the design to ensure that it satisfies
requirements and is correct, complete, robust, and testable. If the CDR
is held, the PDL shall collect the RFA's that result from the CDR and
shall track RFA responses and resolutions. The PDT incorporates feedback
from the review into the design documentation.

**CDR Topics:**

-   Final design (hardware & software)

-   Final Reuse/COTS decisions

-   Interface definitions and ICD status

-   Prototype results

-   Build/development dependencies

-   Final build plans

-   Updates to Test strategies

-   Updates to requirements and design traceabilty, including
    traceability to test plans

-   Updates to size estimates and schedule

-   Staffing plans

-   Issues, including software safety issues, risks and TBD's

    For projects with extensive reuse and where the review is held at
    the PDT level, the CDR concentrates on the new or changed functions
    and modifications to the existing design.

## 1.4 Implementation Activities

### 1.4.1 Code New Modules and Modify Reused Modules

PDT members shall code new modules and/or modify reused modules
according to the design, using the coding standards or conventions
specified in the SMP. This activity also includes implementing tailoring
and configuration of COTS/GOTS components.

### 1.4.2 Verify New and Revised Modules

The PDT shall verify new and revised modules. Code reading,
walkthroughs, inspections and unit tests are appropriate verification
activities. Records appropriate to the verification techniques shall be
kept. Errors are corrected and the module is certified as having been
verified. The PDT shall document the procedures it uses for verification
and certification.

### 1.4.3 Integrate Modules

The purpose of integration is to ensure that modules function correctly
together. It is recommended that the PDT plan and document the
integration process and integration testing activities. The PDT
integrates the modules (COTS/GOTS, glue-ware, and/or new modules), and
tests the integrated modules to ensure that they function together
correctly. The PDT reports and corrects defects and then delivers the
integrated system for build/release testing.

### 1.4.4 Draft System and User Documentation 

It is recommended that the PDT produce a draft of the system description
and user's guide. At a minimum, the draft user\'s guide could consist of
an outline of material to be incorporated into the operations manual.
The system description document is an \"as-built\" design document.

## 1.5 Test Activities

### 1.5.1 Build Testing 

#### **1.5.1.1. Prepare a Build/Release Test Plan**

The purpose of build/release testing is to verify that the software
provides the functionality required of the build/release and is a
correct implementation of the design; the emphasis is on functional and
performance requirements. A test team, consisting of PDT members or a
team independent of the PDT but supported by the PDT, shall prepare a
build/release test plan. The test plan documents the procedures to be
followed in testing each build/release to ensure that it satisfies
requirements. The test plan includes traceability between tests and
requirements. It includes regression tests to ensure that previously
tested functions are not adversely affected in each new build.

#### **1.5.1.2 Verify the Build/Release Test Plan**

The test team shall perform a team level verification of the build or
release test plan. The verification inspection ensures (1) that the test
procedures completely test the requirements to be satisfied in the
build/release and (2) that they are written to maximize reuse in system
and/or acceptance testing.

#### **1.5.1.3 Prepare or Procure Test Software**

If required for adequate test, the PDT shall build, or procure, special
software to generate data sets or simulate missing hardware or software
components. The PDT determines the degree to which test software must
follow PDT standards.

#### **1.5.1.4 Verify Critical Test Equipment or Software**

If test software or test equipment is critical to ensuring the success
and validity of software, the equipment or software shall be verified to
ensure that it functions correctly. The PDT shall document procedures
and tests used to verify test equipment and test software to ensure that
the tests can be easily and efficiently repeated.

#### **1.5.1.5 Conduct Build/Release Tests**

The test team executes the tests specified in the plan. The results of
build/release tests shall be recorded. The PDT analyzes and corrects
discrepancies found in build testing. It is recommended that the test
team produce a test report after the completion of testing for the
build/release.

### 1.5.2 System Testing 

System testing is designed to verify the functionality of the software
product against the requirements and specifications in a more realistic
operational environment, for example as a \"string test\" including
multiple systems. The test team shall develop a system test plan to
verify the end-to-end functionality of the system in satisfying the
requirements and specifications. The plan includes traceability of test
cases to requirements and specifies the environment for the system
tests. It is recommended that the tests be executed on the integrated
spacecraft/hardware system where this is feasible.

For a project with significant reuse the end-to-end system tests often
include system tests defined for the system that is being reused. It is
recommended that the end-to-end system tests include regression testing
of components that are being reused.

#### **1.5.2.1 Verify the System Test Plan**

The test team will verify the system test plan in order to ensure that
all requirements are adequately tested and that test data can reasonably
be made available.

#### **1.5.2.2 Develop Detailed Test Procedures**

It is recommended that detailed, step-by-step test procedures be
prepared. The test procedures identify each test step, the data to be
used, and expected results from each step. It is recommended that the
test team use draft system and user documentation in preparing test
procedures.

#### **1.5.2.3 Conduct System-Level or End-to-End Tests**

#### 

The test team executes the test procedures specified in the system test
plan. Test results shall be recorded. The test team issues documentation
of product and test problems that are identified. The PDT, according to
CM procedures, will correct these identified problems.

#### **1.5.2.4 Verify System and User Documentation**

During conduct of the system test, the draft of the users' guide is
verified, noting such flaws as missing or unclear instructions,
contradictions, etc. The PDT shall analyze and correct any problems
encountered in the document and shall complete the documentation.

#### **1.5.2.5 Report System-Level or End-to-End Test Results**

The test team shall produce a test report after the completion of system
testing. The test report identifies the tests that were executed and
their results, includes a summary of nonconformances found during
testing, and identifies remaining nonconformances.

### 1.5.3 Acceptance Testing 

#### **1.5.3.1 Conduct an Acceptance Test Readiness Review** 

It is recommended that an ATRR be held. The acceptance test team, which
is usually independent of the PDT, meets with the PDT, project and
immediate line management, customers, testers, users, and other
interested parties to review the results of the testing completed to
date, and to evaluate preparedness for acceptance testing. If any
external individuals (not Civil Servants) participate in this review,
appropriate non-disclosure agreements must be signed in advance of the
review. Outstanding problems that affect acceptance testing are
discussed. If an ATRR is held, the RFA's that result from the ATRR shall
be collected and tracked to resolution.

**ATRR Topics:**

-   Status of system readiness

-   Summary of test results; tests performed, successful tests, known
    problems, waivers and issues

-   Discussion of support to be provided for Acceptance Testing.

-   Acceptance test approach including ground rules, tools and reporting
    (input provided by the Acceptance Test Team)

-   Test schedule (input provided by the Acceptance Test Team)

#### **1.5.3.2 Support Acceptance Testing** 

It is recommended that members of the PDT be available to support
Acceptance Testing as conducted by the customer or an independent test
organization. This support consists of consulting activities, analysis
of test results, and correction of nonconformances as requested.

## 1.6 Delivery and Support Activities 

### 1.6.1 Delivery

When software is released or delivered to an external organization, it
is important that Government-owned intellectual property be protected.
Refer to NPD/NPG 2210.1, "External Release of NASA Software," for
further information concerning the domestic or foreign release of
software created by or for NASA.

#### **1.6.1.1 Prepare the Release Letter**

The PDL shall document the delivery of the accepted system release in a
release letter to the customer that details the release identification,
release contents, release capabilities, and any remaining
nonconformances.

#### **1.6.1.2 Prepare Shipping Records**

The PDT shall prepare shipping records that accompany any hardware or
other items transported. The records identify each item being delivered,
its delivery medium, and its destination.

#### **1.6.1.3 Deliver the Completed System or Release**

The PDT shall deliver the software and all documentation using the media
and methods identified in the SMP. (See NPD/NPG 2210.1 regarding
external releases or deliveries.)

### 1.6.2 Operations Support 

#### **1.6.2.1 Support the Operational Readiness Review**

####  

The PDT shall support the ORR held by the Acceptance Test Team, or shall
participate in, or contribute to, a higher-level Project ORR**.** The
ORR serves as an opportunity to review the ability of the system to
support operations. The status of the remaining nonconformances and the
status of system documentation are presented. If an ORR is held, the PDL
shall collect the RFA's that result from the ORR and shall track RFA
responses and resolutions.

**ORR Topics:**

-   Summary of test results including Acceptance Testing with
    nonconformances and their impact on operations

-   Status of system documentation including user and operations
    manuals.

-   Status of external interface agreements

-   Readiness of operational environment for installation

-   Operational support and maintenance support plans

-   Configuration control procedures

#### **1.6.2.2 Provide Customer Training**

The PDT shall provide training in the installation and execution of the
software to operations staff (or other users) to the extent specified in
the SMP. The PDT shall train maintenance personnel as specified in the
SMP.

#### **1.6.2.3 Provide Technical Assistance**

The PDT shall provide technical support and maintenance during the
agreed-upon support period as defined in the SMP. The PDT responds to
questions raised by operations staff and other users and assists in
identifying and analyzing defects or anomalies in the behavior or
performance of the system. Records of maintenance requests shall be
kept.

# PROCESS FOR SOFTWARE MAINTENANCE

## Appendix B: Software Maintenance Process

This process description is intended to be used as a starting point in
defining the project\'s technical approach. The activities are not
intended to be strictly time sequenced. This allows them to be used to
define activities in a variety of maintenance situations. There may be
activities that are unique to the project to mitigate special risks or
to handle special requirements. These may be added to meet the
project\'s unique needs.

## 2.1 Cross-Cutting Activities

### 2.1.1 Team Management

###  

#### **2.1.1.1 Develop a Software Management Plan**

The Product Maintenance Lead (PML) shall develop a SMP using the
Software Management Plan Guide in the Attachment of this GPG. [GSFC Form
19-21](http://gdms.gsfc.nasa.gov/gdms/plsql/masterlist.menu) provides a
template for the recommended SMP outline. (Note: When using [GSFC Form
19-21](http://gdms.gsfc.nasa.gov/gdms/plsql/masterlist.menu),
information can be rearranged or supplemented as long as required
information is included.) The completed SMP describes the customer
requirements and resources (e.g., budget and staffing) allocated to
execute the Plan. It documents release planning and the strategy for
prioritizing maintenance requests including criteria for identifying
emergency changes. It identifies a nominal maintenance release schedule,
reviews for nominal releases and their formality, and internal Product
Maintenance Team (PMT) level verification. It identifies exceptional
circumstances that may result in fewer or more reviews or more formal
reviews (e.g., response to a critical spacecraft failure).

For external releases or deliveries, additional requirements may apply.
See NPD/NPG 2210.1, "External Release of NASA Software," for further
information on the domestic or foreign release of software created by or
for NASA.

The SMP defines the approach that will be used to track project status,
for configuration management, product assurance, risk management, and
independent verification and validation. It also summarizes the
technical approach including the review plan, documentation to be
maintained and the criteria used to tailor this Process for Software
Maintenance.

Tailoring is determined by the PML in consultation with management and
the customer. Tailoring factors include project characteristics such as
the criticality of the application, the size of the Product Maintenance
Team (PMT) and user community, the degree of reuse, and other project
specific factors. Required activities (i.e., those designated as
\"shall\") may not be deleted.

The PML defines the review plan for nominal releases based on the
criticality, complexity, and the number of users affected by the
maintenance changes. For high priority or emergency patches the PML has
the authority to accomplish the goals of a review - to achieve customer
concurrence - through other means (e.g., telecon or email), followed by
more formal documentation of concurrence after the emergency is
resolved. The following reviews should be considered:

-   Maintenance Release Planning Review

-   Maintenance Release Design Review\*

-   Acceptance Test Readiness Review

-   Operations Readiness Review

**\***Required Reviews

Software project reviews shall:

-   Be scheduled with published agendas

-   Include customers or customer representatives

-   Record meeting notes

-   Collect RFA's

-   Record and track RFA's to resolution.

#### **2.1.1.2 Manage the Task**

####  

The PML negotiates project-staffing assignments, assigns technical work
to staff members, and helps assure that all PMT members are qualified to
perform their assigned duties. The PML is also responsible for the
day-to-day management of the task according to the SMP. The PML shall
periodically review and record the status of maintenance activities
using the methods defined in the SMP. It is recommended that project
status be an objective measure of work products accomplished against
planned resource allocation (e.g., schedule, cost and effort).

#### **2.1.1.3 Collect Metrics**

Project metrics, at a minimum, shall include schedule planned vs. actual
dates, budget (effort and cost), actual release content vs. planned
release content, change in product size, and product error information,
such as open/closed nonconformances (i.e., discrepancy reports), and
shall be collected periodically (e.g., monthly or quarterly). Project
metrics are analyzed and the results used to initiate process
improvement activities.

#### **2.1.1.4 Document Lessons Learned**

The PML shall review, at the beginning of, and then periodically
throughout the maintenance effort, the LLIS (which is maintained at
[http://llis.nasa.gov](http://llis.nasa.gov/) ) for relevant experiences
and knowledge that can be leveraged to reduce risk, improve quality,
an/or apply Best Practices. The PML shall also submit significant
lessons learned to the web-based LLIS system as appropriate.

It is recommended that lessons learned be documented in a Software
Development History at each maintenance release. The Software
Development History is updated to include lists of the products in the
delivery, milestones and key events, release duration, key decisions,
problems encountered, and summaries of the metrics collected during the
phase. It describes the specific lessons and recommendations that
pinpoint the major strengths and weaknesses of the process used and the
product itself, with particular attention to planning, requirements,
development, testing, CM, QA, and changes in technology.

#### **2.1.1.5 Prepare and Maintain Project Documentation**

#### 

The PMT shall maintain the project documents as specified in the SMP.
The PMT ensures that each change to project documentation is reviewed,
and that the changes identified during the review are properly
implemented and that appropriate approval signatures are obtained.

### 2.1.2 Training 

#### **2.1.2.1 Identify and Document Required Training Needs**

The PML shall identify and document the training for PMT members that
are required by the QMS. Required training includes familiarizing PMT
members with the SMP, the methodology, standards, and design process
used, maintenance of PMT records, and use of a nonconformance recording
system. It also includes training required when working in proximity to
instruments or spacecraft, such as Electrostatic Discharge Awareness
Training, Range or Launch Safety, Laser Safety, etc. The PML works with
the appropriate supervisors to assure that the identified training is
provided. It is recommended that the PML identify any special
developmental training such as language and tool training, and identify
on-the-job-training to build an adequate level of understanding in all
PMT members of software that will be maintained.

#### **2.1.2.2 Record Training Received**

As PMT members obtain training, the PML shall maintain records of the
training that has been received and by whom (see GPG 3410.2).

### 2.1.3 Configuration Management (See GPG 1410.2)

#### **2.1.3.1 Perform Configuration Identification**

The PMT shall identify, in the SMP, the types of items to be placed
under configuration control and shall identify when they will be placed
under configuration control. Items, if any, that are to be controlled at
the higher-level Project, shall be identified. For maintenance projects
each release constitutes a baseline. The PMT identifies the products
comprising each baseline, down to the level of the smallest controllable
unit. The PMT shall develop a unique identification for each planned
system release/version and delivery. The PMT defines naming and labeling
conventions as appropriate. It is recommended that the PMT follow naming
and labeling conventions (e.g., release/version, file and module naming
conventions) consistent with those established during the system
development.

#### **2.1.3.2 Maintain Configuration Control**

The PMT shall preserve the integrity of all system baselines,
components, and products. The PMT shall track changes to controlled
products to assure that the configuration of all identified items is
known at all times. Controlled products include system baselines,
project documents (requirements, design, test plans, etc.), source code,
object code, released and developmental versions of the target system,
critical test software and any customer-provided product (e.g., GOTS)
used in the development of the software system. The use of a commercial
configuration management tool is recommended for configuration control
of the project's permanent source code libraries. This also includes
maintaining change control over any critical test software. The PMT
shall maintain records of all software configuration management
activities performed, including all changes made to software and
documents under configuration control.

#### **2.1.3.3 Monitor and Report Configuration Status**

It is recommended that the PMT periodically produce configuration status
reports. These reports typically include such information as the number
of changes made to date, the reason for each change, the number of
system releases to date, the functionality provided with each
release/version, and the latest version and revision identifiers. It is
recommended that the first configuration status report list
configuration items for the system at the start of the maintenance
project and include identification of any known nonconformances that
exist at the time. This report is reviewed against design documentation
and configuration records received with the software at the start of the
project.

### 2.1.4 Quality Assurance

#### **2.1.4.1 Support Project Reviews, Walkthroughs, and Inspections**

Quality reviews address completeness, readability, traceability, and
conformance to PMT standards. They assess both the quality of the
product (the deliverable under review) and the quality of the process
(the way the review is conducted, including planning, announcement,
preparation, conduct, and follow-up). Problem areas are identified and
appropriate corrections suggested. Although quality is a shared
responsibility, it is recommended that the responsibility for advocating
and assessing the quality assurance process be centralized in a single
individual. It is recommended that product quality be addressed at each
major review. It is recommended that quality also be addressed during
in-process reviews (walkthroughs, inspections, test results etc.). It is
recommended that inspection criteria for any customer-provided product
(e.g., GOTS) used by the PMT be established in conjunction with the
customer. Results of all reviews and inspections, including those of
incoming customer-supplied products, shall be documented.

#### **2.1.4.2 Review Project Deliverables**

It is recommended that all major project deliverables (documents,
software releases, presentations, etc.) be reviewed for quality. It is
recommended that a sample of minor deliverables (unit designs, unit
code, test plans and results, etc.) be reviewed for quality.

#### **2.1.4.3 Monitor and Manage Risk**

The Risk Management approach is identified in the SMP. Risks shall be
continuously identified, analyzed, monitored, tracked, mitigated and
status reported at each review. A detailed discussion of risk management
is found in NPG 7120.5 "NASA Program and Project Management Processes
and Requirements."

#### **2.1.4.4 Monitor and Support Problem Identification and Corrective Action**

The PMT shall employ and follow a problem identification and corrective
action process consistent with the Goddard Center requirements. If a
minor NCR/CA system is used it should include the version or release
number where the problem was found and, ideally, the version number that
includes the corrections. The PMT shall record the number and severity
of defects and nonconformances and the corresponding corrective and
preventive actions; this information is used for process improvement.
The PMT shall identify and segregate any nonconforming builds/releases
or deliveries. When, with customer concurrence, software is delivered
with known nonconformances, the release letter clearly identifies the
nonconformances. Software with nonconformances that has not been
approved for delivery is identified and maintained separately so that it
is not inadvertently included in a delivered product. The PMT shall
document any nonconformances for customer-supplied products (e.g.,
GOTS). Major nonconformances that meet the criteria specified in GPG
1710.1 shall be recorded in the in the Center NCR/CA system.

#### **2.1.4.5 Control Documents and Records**

The PMT shall control all its own records as identified in the SMP. The
PMT shall document and implement a process for the identification,
review, approval, distribution, retention, and disposition of documents
and records.

#### **2.1.4.6 Perform Configuration Audits**

At the beginning of the maintenance project and prior to each release or
delivery, it is recommended that an audit of the software be performed.
The audit at the start of the project ensures that all the required
system elements and documentation are present. Audits performed prior to
a release or delivery verify that every delivery item (e.g., program,
input file, test software, or document) is as reported in the delivery
documentation and delivery letter. Each item on the delivery list is
checked to ensure that the item is present, is complete, is the correct
version, is in the specified delivery format, and is correctly
identified. (Additional requirements apply for external releases or
deliveries; refer to 2.6.1 for additional information.)

#### **2.1.4.7 Support IV&V Activities as Required**

The PML shall provide support to the Project for the submission of the
IV&V criteria information listed at
[http://swmetrics.nasa.gov/ivvcrit/.](http://swmetrics.nasa.gov/ivvcrit/)
If it is determined that the project requires IV&V support, the PMT
shall provide the required support to the IV&V facility for these
activities throughout the life of the project.

**2.1.4.8** **Comply with Specific Electrical Engineering Division and
Project Requirements for Connecting to Flight Hardware**

The ISD PML shall insure that any ISD non-flight device connecting to
flight hardware shall be developed and maintained in compliance with the
Electrical Engineering Division (EED) and any specific Project policies
and practices (MASIS for HST, for example). The PML shall also insure
that all civil servants and support contract staff equipment interfacing
with flight elements for purposes of software/firmware debug and update
comply with EED and specific Project direction.

## 2.2 Requirements Activities

### 2.2.1 Maintenance Request Evaluation

#### **2.2.1.1 Document Each Maintenance Request and Assign a Unique Identifier for Tracking Purposes**

Each maintenance request shall be assigned a tracking number and logged.
The PML assigns PMT members to evaluate the request. It is recommended
that the PML assign lead responsibility for investigating each
maintenance request to a specific PMT member.

#### **2.2.1.2 Perform Preliminary Assessment of Resources Required to Implement Each Maintenance Request**

It is recommended that a preliminary assessment of the resources
required to implement a solution to the maintenance request be performed
and documented. It is recommended that the PMT document procedures for
performing this assessment in the SMP.

#### **2.2.1.3 Classify the Maintenance Request**

The SMP identifies recommended classification categories for maintenance
requests, such as severity level and type (e.g., by subsystem, by error
type). It is recommended that the maintenance request be classified and
that classification metrics be collected.

#### **2.2.1.4 Prioritize the Maintenance Requests**

The PML shall periodically meet with the customer to establish and
accept or reject recommendations for all new change requests.
Notification of the rejected changes shall be sent to the originator
along with a brief rationale. Accepted maintenance requests shall be
assigned a resolution priority based on the user\'s priority, the PMT\'s
assessment of the impact on operations, and the effort to implement a
solution. It is recommended that the SMP include guidelines and
procedures for prioritization, including criteria for designating a
nonconformance as serious enough to require following emergency patch
procedures.

### 2.2.2 Maintenance Release Definition

#### **2.2.2.1 Derive New or Modified Requirements from the Maintenance Request**

If the request is for minor modifications to existing requirements, the
PMT identifies the required changes. If the request is for new
functionality or for complex changes, the analysis method specified in
the SMP is used to derive detailed requirements to fully define the
change. The focus of requirements definition and analysis activities is
on what the changed software needs to do rather than how to do it.
Modifications that affect operations, for example changes to system
parameters, reports, displays, etc. are defined. New reports and
displays are identified. Test requirements are also derived. If the
change is driven from needs external to the user community (e.g., a
change to an institutional or other interfacing system that requires a
corresponding change to the system being maintained), walkthroughs with
users and representatives of interfacing systems are recommended**.**
Draft updates to the requirements document are prepared using mark-ups
to show where text is to be inserted or deleted to make clear what is
being changed.

#### **2.2.2.2 Refine the Estimated Resources (e.g., Staff and Effort) Required to Implement the Requested Maintenance Change**

It is recommended that the assigned PMT member reevaluate the
preliminary impact assessment to make a detailed estimate of the
resources and schedule required to implement the change. The resource
estimate includes the effort to generate additional or changed test data
and to modify test software as required. It is recommended that the PMT
identify dependencies among components and determine if a phased
implementation is desirable and/or feasible.

#### **2.2.2.3 Plan Release Contents Using Request Priorities and Resource Estimates**

The PML shall work with customers to plan the content of maintenance
releases. The PML uses knowledge about the areas of the system that will
be affected by a change and about upcoming work, along with the
customer\'s priority input, to identify planned release contents. If a
change is identified as critical to continued operations, one or several
maintenance requests may be designated as an emergency patch and
implemented on an expedited schedule.

#### **2.2.2.4 Conduct a Maintenance Release Planning Review** 

It is recommended that a Maintenance Release Planning Review be
conducted for each Maintenance Release. The purpose of the review is to
gain customer concurrence on the release contents, the proposed changes
to requirements and the planned release schedule. Review participants
include customers, users or their representatives. If the Maintenance
Release Planning Review is held, meeting minutes shall be taken and
RFA's tracked to closure.

**Maintenance Release Planning Review Topics:**

-   Summary of maintenance requests in the release, including priority
    and resource estimates

-   New and modified requirements

-   Operational impacts

-   Release schedule

In most cases the review is an informal technical meeting. However, it
is recommended that the review be formal for major upgrade releases,
where there are multiple user organizations, if the changes are complex
or have a significant impact on operations, or if the changes impact
interfacing systems.

## 2.3 Design Activities 

### 2.3.1 Update the Maintenance Request Documentation

For each maintenance change in the planned release, it is recommended
that information describing the alternatives considered and the selected
solution be added to the maintenance request tracking documentation.
This information serves as documentation of the proposed change to the
system and provides traceability for changes to software and
documentation that result from the maintenance request.

### 2.3.2 Identify all Affected Elements of the Software Design 

Identify all software elements, including COTS/GOTS components and
documentation that are affected by the maintenance change(s). If a
Requirements Traceability Matrix exists, use it as a guide to help
identify the components that implement the impacted requirements.

### 2.3.3 Modify the Design 

Using the design approach and techniques specified in the SMP, design
modifications to existing components and design new components. The PMT
shall document the design modifications. Where possible, use mark-ups of
existing design material to highlight changes. If the changes are too
extensive to be shown as markups, generate replacement diagrams and
other design material, and make both the new material and the old
material available for review.

### 2.3.4 Identify Tailoring/Configuration Changes (where applicable)

If the maintenance change calls for changes to the tailoring or
configuration of a COTS/GOTS component, identify and document those
changes. If the maintenance request calls for changes to existing system
configuration parameters, identify and document those changes.

### 2.3.5 Procure and Install Replacement COTS/GOTS Components in the Development Environment (where applicable*)*

If approved by the customer, initiate procurement activities to obtain
the new version(s) of COTS/GOTS components and install the product for
testing when it arrives. (Note that in most cases, open competitive
procurement of COTS products is required. See GPG 5100.1 for additional
information on procurements.)

### 2.3.6 Verify Design Modifications

It is recommended that a PMT-level review of the resulting modified
design be conducted. Follow-up actions shall be documented and tracked
to resolution.

### 

### 2.3.7 Conduct a Maintenance Design Review

For each maintenance release, the PMT shall hold a Maintenance Design
Review that includes customers and users. If the release includes
changes to a critical system, if the changes are complex, or if there
are extensive changes to the system design (e.g., architectural changes
or replacement, not upgrade, of COTS/GOTS components), then it is
recommended that the review be formal. Meeting minutes shall be taken
and RFA's tracked to closure.

**Design Review Topics:**

-   Summary of maintenance requests in the release, including priority
    and resource estimates

-   Summary of requirements changes

-   Interface changes

-   Summary of operational impacts

-   Identification of system architectural changes (if any)

-   Description of significant design changes

-   Changes to COTS/GOTS

-   Release test plan overview

-   Updated release schedule

## 2.4 Implementation Activities

### 2.4.1 Code New Modules and Modify Existing Modules

PMT members shall code new modules and modify existing modules according
to the design, using the coding standards or conventions specified in
the Software Development Plan. This activity includes implementing
changes to COTS/GOTS tailoring and configuration parameters.

### 2.4.2 Verify New and Revised Modules 

The PMT shall verify new and revised modules. Code reading,
walkthroughs, inspections and unit tests are appropriate verification
activities. Records appropriate to the verification techniques shall be
kept. Errors are corrected and the module is certified as having been
verified. The PMT shall document the procedures it uses for verification
and certification.

### 2.4.3 Integrate the Release/Patch 

The PMT plans the integration activities, integrates the modules
(COTS/GOTS, glue-ware, and/or new modules), and tests the integrated
modules to ensure that they function together correctly. The PMT
documents the procedures it uses for integration. Problems are
identified and corrected in preparation for release/patch testing.

### 2.4.4 Update System and User Documentation if Required

The PMT produces draft updates to the appropriate documentation (e.g.,
system description, operations manuals, user's guide and training
material). At a minimum, the draft updates are mark-ups to the existing
documents.

## 2.5 Test Activities

### 

### 2.5.1 Release Testing

#### **2.5.1.1 Develop a Release/Patch Test Plan Based on the Existing Test Plans and Regression Tests**

The purpose of release/patch testing is to verify that the software
provides the functionality required of the release and is a correct
implementation of the design; the emphasis is on functional and
performance requirements. A test team, consisting of PMT members or a
team independent of the PMT but supported by the PDT, shall develop a
release/patch test plan using existing system test documentation as the
basis for the new and modified tests. Maintenance release tests are at
the level of build and functional scenario tests. They demonstrate that
the system functions as designed and that changed and new requirements
are satisfied. The plan documents the procedures to be followed in
testing the release to ensure that it satisfies the new or changed
requirements. When testing fixes for defects it is often useful to
follow the scenario that was used to recreate the problem as one of the
test cases. For critical systems the release test plan shall include
regression tests to ensure that unmodified functions are not adversely
affected. The plan defines build test completion criteria. The completed
test plan shall be placed under configuration management.

#### **2.5.1.2 Review the Test Plan**

It is recommended that the test plan be reviewed at the PMT level. If
the change is critical or affects interfacing systems a more formal
review that includes customer or user representation is recommended**.**
Review meeting notes shall be documented.

#### **2.5.1.3 Prepare Detailed Test Procedures for New Test Cases**

Detailed step-by-step test procedures shall be prepared. The test
procedures identify each test step, the data to be used and results
expected from each step. The detailed test procedures reference and
build on existing system test procedures.

#### 

#### **2.5.1.4 Modify Test Data and Software as Needed**

### Prepare changes to test data. If necessary, modify existing test drivers and/or test data generation software and verify the changes. {#prepare-changes-to-test-data.-if-necessary-modify-existing-test-drivers-andor-test-data-generation-software-and-verify-the-changes. .unnumbered}

#### **2.5.1.5 Conduct Patch/Release Tests**

The test team executes the tests specified in the plan. The results of
patch/release tests shall be recorded. The PMT analyzes and corrects
discrepancies found in build testing. It is recommended that the test
team produce a test report after the completion of testing for the
patch/release.

#### **2.5.1.6 Prepare Final Updates to System and User Documentation**

The SMP identifies user and system documentation that is to be
maintained. At the conclusion of release testing these documents are
updated according to project standards and placed under the specified
level of configuration control.

### 2.5.2 Acceptance Testing 

#### **2.5.2.1 Conduct a Review of the Test Results** 

It is recommended that an informal ATRR be held for each release. The
test team produces a test report after the completion of testing for the
release and reviews the results with customer and user representatives.
In most cases a PMT-level review of the release/patch test results is
either informal or optional. For large PMT\'s or critical changes a
formal review is recommended. Minutes of the meeting shall be taken and
RFA's tracked to closure.

If any external individuals (not Civil Servants) participate in the
ATRR, appropriate non-disclosure agreements must be signed in advance of
the review.

**ATRR:**

-   List of maintenance requests in the release, including priority and
    test status (pass/fail)

-   Summary of tests conducted and tests passed or failed.

-   List of nonconformances remaining in the release and operational
    impact of the nonconformances on operations.

#### **2.5.2.2 Support Acceptance Testing Performed by the Customer or by an Independent Organization**

It is recommended that members of the PMT be available to support
Acceptance Testing as conducted by the customer or an independent test
organization. This support consists of consulting activities and
analysis of test results as requested.

## 2.6 Delivery and Support Activities 

### 

### 2.6.1 Delivery

For external releases or deliveries, additional requirements may apply.
See NPD/NPG 2210.1, "External Release of NASA Software," for further
information on the domestic or foreign release of software created by or
for NASA.

#### **2.6.1.1 Prepare the Release Letter**

The PML shall document the delivery of the accepted system release in a
release letter to the customer that details the release identification,
release contents, release capabilities, and any remaining
nonconformances. The release letter identifies any changes that affect
operations or require user training. All software and documentation that
are part of the delivery are promoted to the configuration-controlled
delivery library at the time of delivery and constitute a system
baseline.

#### **2.6.1.2 Prepare Shipping Records for Delivery**

The PMT shall prepare shipping records that accompany any items
transported. The records identify each item being delivered, its
delivery medium, and its destination.

#### **2.6.1.3 Deliver the Completed System or Release**

The PMT shall deliver the software and all documentation using the media
and methods identified in the

Software Management Plan. Release documentation includes installation
and back-out instructions. (See NPD/NPG 2210.1 regarding external
releases or deliveries.)

### 

### 2.6.2 Operations Support 

#### **2.6.2.1 Support Review of the Acceptance Test Results** 

It is recommended that the PMT support the ORR if one is held. The ORR
serves as an opportunity to review the ability of the system to support
operations. The status of remaining nonconformances is presented.

**ORR Topics:**

-   Summary of Acceptance Testing including nonconformances and their
    impact on operations

-   Summary of tests conducted, tests passed and tests failed

-   Identification of nonconformances and suitability of workarounds in
    the operational environment

#### **2.6.2.2 Provide User Training**

To the extent specified in the SMP, the PMT shall provide user training
and installation instructions, and user documentation specific to new or
modified user and operations capabilities.

#### **2.6.2.3 Provide Technical Assistance** 

The PMT shall provide technical support to the extent specified in the
SMP. The PMT responds to questions raised by operations staff and other
users and assists in identifying and analyzing defects or anomalies in
the behavior or performance of the system.

####  

# Appendix C: Product Plan Instructions {#appendix-c-product-plan-instructions .PDHHeading1}

Product Plan---A description of the work to be performed and the
resources needed to accomplish the goals and objectives established in
the customer agreement. In QMS terminology, the Product Plan is the
*QUALITY PLANNING DOCUMENT* for producing the product. The Product plan
includes the *design planning information* and the *process management
information*.

Figure 1 is the table of contents to be followed by all Teams in
generating their Product Plan. All components shall be addressed, but
the level of detail is left to the Team based on product complexity and
customer needs/expectations.

ISO 9001 standards require certain quality control processes to be
documented. Those processes that are required and the criteria that they
must meet are described in Appendix F. Samples of these required
processes that meet these criteria are in the Library of Approved Team
Processes (see
[http://isc.gsfc.nasa.gov/ISO9k/ISO9001.htm](http://isc.gsfc.nasa.gov/Iso9k/ISO9001.htm)).
Each team in their product plan may either refer to one or more specific
approved processes, or develop and document their own processes. Any new
processes must meet the criteria specified in Appendix F and be approved
by the ISD QMS representative. The effective date of the Product
Development Handbook being used is to be placed directly under the
Product Plan "Table of Contents" label on the "Table of Contents" page
and should be labeled as "Version Date xx/xx/xx". This effectively
freezes the version of the Product Plan Table of Contents for the
duration of the Product Plan.

The following disclaimer should be placed in the footnote section of
every page:

"Printed copies of this document are for **REFERENCE PURPOSES ONLY**!
The only controlled copy of this document is located on-line at
[http://xxxxxxxx](http://xxxxxxxx/)."

Optionally for Product Plans originally signed prior to ***8/1/99***,
this disclaimer may appear directly after the signature page:

\"References to documents and data (hard copy or electronic) in the
Product Plan ***not*** directly under the Team's control shall contain
the version identification in the Product Plan.\"

The Work Order Authorization (WOA) equivalent for software development
referred to in the GSFC QMS is defined by the contents of Sections 3.1
and 4.2.1 of the ISD Product Plan, including all associated
documentation and references.

Section 1 of the Product Plan (Customer Agreement) may be under either
Project or ISD control. It follows the configuration management process
outlined in Section 1.15. A customer signature for the Section 1 of the
Product Plan is highly recommended (see Figure 2).

The remaining sections of the Product Plan are under ISD Configuration
Management and follow the configuration management process defined in
Section 4.2.1. At a minimum, these portions of the Product Plan require
a Team Lead signature and an appropriate ISD Management Team signature
for approval (see Figure 3).

The remainder of this appendix lists the product plan subsections with a
description of the contents following each subsection heading. Quality
records and controlled documents are identified in tables in the
appropriate subsections. Tables labeled as objective evidence indicate
other records that should be maintained by the Team, but are not
included on either the quality records list or the controlled documents
list.

Certain subsections may be included by reference if documented
elsewhere. These subsections are identified with an asterisk (\*). It is
recommended that subsections containing frequently changing information
be included by reference.

+-----------------------------------------------------------------------+
| Table of Contents for Product Plan                                    |
+-----------------------------------------------------------------------+
| Table of Contents---Product Development Handbook -Version Date:       |
| xx/xx/xx                                                              |
|                                                                       |
| Document Change History---Include version identifier and description  |
| of change                                                             |
|                                                                       |
| Customer Agreement Signature Page                                     |
|                                                                       |
| Product Development Signature Page                                    |
|                                                                       |
| 1.  Customer Agreement                                                |
|                                                                       |
| > 2\. Management Approach                                             |
| >                                                                     |
| > 3\. Technical Approach                                              |
|                                                                       |
| 4\. Product Assurance                                                 |
|                                                                       |
| 5\. Plan Update History                                               |
|                                                                       |
| Appendix A---Acronyms and Abbreviations                               |
|                                                                       |
| Appendix B---References                                               |
+-----------------------------------------------------------------------+
| *Figure 1. Template for Table of Contents*                            |
|                                                                       |
| **NOTE**: See following pages for detailed information on contents of |
| each section.                                                         |
+-----------------------------------------------------------------------+

Customer Agreement

for the

(project name)

(system type^1^) Development

Release Date

*Month/Year*

Prepared by:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

XXXXXXX

Team Lead

Approved by:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

XXXXXXX

Customer/Designee

Approved by:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

XXXXXXX

Information Systems Division Management Representative ^2^

The Team Lead, the Customer/Designee, and the Information Systems
Division Management Representative constitute the Configuration Control
Board for the Customer Agreement portion (Section 1) of this document.

Disclaimer

Printed copies of this document are FOR REFERENCE PURPOSES ONLY. It is
the user's responsibility to verify that the version of any printed
documentation matches the online version.

***FIGURE 2. Template for Customer Agreement***

**Notes: (1) System types---This would be ground data system, flight
software, command and data handling system, etc.**

(2) **Information Systems Division Management Representative---Title
    listed here should be the specific title of the ISD manager
    responsible for the development, for example, "Code 5xx Branch
    Head"**

Product Development

for the

(project name)

(system type^1^) Development

Release Date

*Month/Year*

Prepared by:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

XXXXXXX

Team Lead

Approved by:
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

XXXXXXX

Information Systems Division Management Representative ^2^

The Team Lead and the Information Systems Division Management
Representative constitute the Configuration Control Board for this
document, with the exclusion of the Customer Agreement (section 1).

Disclaimer

Printed copies of this document are FOR REFERENCE PURPOSES ONLY. It is
the user's responsibility to verify that the version of any printed
documentation matches the online version.

***FIGURE 3. Template for Product Development***

**Notes: (1) System for types---This would be ground data system, flight
software, command and data Handling system, etc.**

> **(2) Information Systems Division Management Representative---Title
> listed here should be the specific title of the ISD manager
> responsible for the development, for example, "Code 5xx Branch Head"**

**1.0 Customer Agreement (GPG 1310.1)**

1.1 Background

A brief (maximum of one paragraph) description of what larger
effort/activity this Team is supporting and how this product fits into
the larger picture.

1.2 Team Charter

A brief one-paragraph description of what this Team is being asked to
accomplish, including any time constraints or interface boundaries
within which this Team is expected to operate.

1.3 Customer(s) Identification

The customer is usually a Flight Project or the person who pays the
bill. Otherwise, it should be the person who will define the
requirements and accept the products.

1.4 Customer Goals and Objectives

Any special things that the customer wants to accomplish (e.g., rapid
turn around, new architecture, special COTS requirements, special
experiments, etc.) through this Team's activities.

1.5 Requirements\*

This section should list or reference (preferred) any
functional/operational requirements as specified by the customer.
Include any specific standards to be met and list the interface control
documents needed. Include any policies and practices of Electrical
Engineering Division (EED) or specific Projects for devices connecting
to flight hardware. References must include revision date/number for
documents not under direct control of Team Lead. Do not include
technical interface documents or databases here. Reference them in
Section 2.4

  ------------------------ ------------------------- ---------------------
  Controlled Document      Comment                   Record Held By

  Functional Requirements  Signed and dated by       Project or Team Lead
                           Customer                  
  ------------------------ ------------------------- ---------------------

1.6 Deliverables

List products to be delivered for each phase, including software,
hardware, licenses, documentation, etc., as directed by customer.

1.7 Schedules\*

List *customer---specified* schedule requirements, including such items
as documentation, releases and reviews.

1.8 Necessary Customer Training

Specify who is to be trained how many are to be trained, location and
nature of training.

1.9 Medium/Method for Product Delivery (GPG 6400.1)

List any required delivery medium and method of delivery for all
products listed in Section 1.6

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Shipping Records                                   Team Lead
  ------------------------ ------------------------- ---------------------

1.10 Product Destination

List product delivery destination for all products listed in
Section 1.6.

1.11 Post Delivery

Describe who will do maintenance after and how it will be
requested/approved. Describe process that will be used for maintenance
activities for all products in Section 1.6

1.12 Customer-supplied elements, both technical, and resources
(schedule, medium, and interfaces)

List any technical elements supplied by the customer that will be used
in the production, testing or packaging/delivery of the product. Do not
include funding. Include delivery schedule and medium of supplied items.

1.13 Customer involvement (roles, responsibilities, authority,
accountability)

Provide details on the extent of direct customer involvement with the
Team (Attends Team meetings? Reviews results? Provides direction? Etc.)

1.14 Acceptance Criteria\*

Describe the customer's criteria for determining when the product is
completed, (i.e., when will the customer accept the product?) This is
usually demonstrated by having a satisfactorily completed test
matrix/set of test plans. Customer verbal acceptance is not sufficient.

1.15 Customer Agreement Review and Update Process

Describe the process used to evaluate and approve changes to the
customer agreement. Be sure to note that the Team will be evaluating the
changes to assure that they have the capability of providing the
requested changes. Approval authorities (those listed on the signature
page) must be listed specifically by name and title. It must be stated
that they consist of the Change Control Board (CCB) or the CCB
process/membership must be described or referenced. The original
approval authority must approve changes.

**2.0 Design Planning and Interface Management (GPG 8700.1)**

+-----------------------+------------------------+---------------------+
| Controlled Document   | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Product Plan          | Signed and dated by    | Team Lead           |
|                       | the CCB which is       |                     |
|                       | listed in              |                     |
|                       | Plan---usually Team    |                     |
|                       | Lead, Branch Head,     |                     |
|                       | Customer.              |                     |
+-----------------------+------------------------+---------------------+
| Design Planning       | May be in product      | Team Lead           |
| Materials             | plan.                  |                     |
|                       |                        |                     |
| -   Review Plan       |                        |                     |
|                       |                        |                     |
| -   Development       |                        |                     |
|     Phases            |                        |                     |
+-----------------------+------------------------+---------------------+

+-----------------------+------------------------+---------------------+
| Objective Evidence    | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Design Planning       | May be included in     | Team Lead           |
| Materials             | product plan by        |                     |
|                       | reference.             |                     |
| -   Team Work         |                        |                     |
|     Assignments       |                        |                     |
|                       |                        |                     |
| -   Team Organization |                        |                     |
|                       |                        |                     |
| -   Schedules         |                        |                     |
|                       |                        |                     |
| -   Budgets           |                        |                     |
+-----------------------+------------------------+---------------------+

2.1 General development approach

Describe in a sentence or two the general philosophy that will be used
to build the product, discussing such aspects as use of
commercial-off-the-shelf (COTS), contractor involvement, schedule
constraint, use of a particular development methodology or new
technology, etc.

2.2 Resources needed (budget, people/skills, and facilities)\*

Indicate where the *official* budget is kept. In most cases, the budget
will probably reside with the Project. Budget information should be kept
by fiscal year, and include both civil servant manpower and any
contractor support. Address any specific facilities or any facility
modifications required for use in development or testing and their
expected required dates.

2.3 Team Organization

2.3.1 Team Organization

Include an explanation or diagram illustrating the organization of the
Team personnel and its activities. Note: Any Team organization chart not
included in the product plan must be signed and dated. Include the
relationship of the Team Lead to the higher level Project organization,
if applicable.

2.3.2 Roles, Responsibilities, Authority, Accountability of Team
Members\*

Describe the method used to assign work to Team members and document the
work assignments. Assignments can be made by subsystem (e.g. Command &
Data Handling, Planning & Scheduling) or by work function (e.g. testing)

2.3.3 Decision making and conflict resolution process

Describe the method used to resolve conflicts within the Team. If group
decisions are used, identify the tiebreaker or ultimate decision
authority.

2.4 Team interfaces to other teams, organizations, or groups

Describe any interfaces to other organizations, teams, or groups
necessary in developing the product, and a *brief* description of the
purpose of each interface. This may include things such as the interface
of the ISD flight software team to the flight hardware group for working
compatibility issues, or the interface of the Ground Data System to the
Flight Operations Team for acceptance of the system.

2.5 Procurement\* (GPG 5100.1)

Describe all hardware and software purchase requirements in detail
(i.e., What are you going to buy?) Include any purchases necessary for
facility modification. If you are using contractor support, list the
contractor name and contract number. If special or usual contracting
arrangements are required, describe them. Reference the procurement
process used to make purchases. Be sure to use RITS for all items within
scope.

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Purchase Requests                                  Team Lead or Team
                                                     procurement person
  ------------------------ ------------------------- ---------------------

2.6 Team training plan\* (GPG 3410.2)

Identify any QMS Required Task Specific training need for each Team
member. When training is complete, document it by keeping a list of
name, course, and date completed. (QMS Required Task Specific training
is defined as training that must be taken to acquire new skills or
enhance current skills required to perform tasks of that position that
affect quality. Examples are Hand Soldering Certification, Electrostatic
Discharge Awareness Training, Laser Safety, Cleanroom Procedures, Range
or Launch Safety Training, Flight Operations Team Certification, or any
required Project-specific training)

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Records of Required                                Team Lead or Project
  Training Needed                                    

  Records of Required                                Team Lead or Project
  Training Completed                                 
  ------------------------ ------------------------- ---------------------

2.7 Risk mitigation

Describe any areas where there is a special risk to the delivery of the
product (if any) and describe how it will be addressed. If there is
none, state that.

2.8 Security

Describe the plans for addressing security considerations, both
physically for the facilities involved and electronically for any
computer systems being used either for development and testing or as a
part of the final product.

2.9 Detailed Schedules\*

This should be the detailed schedule used to manage the Team's
activities. It should contain the Team life cycle schedule including
facility preparations, procurements, system development by phase and
release, product delivery, and maintenance (if applicable). Make sure to
include review dates, documentation, interface control document\'s
(ICD\'s) delivery dates; test dates, software release dates,
procurements, and external deliveries to the customer. It should include
and be consistent with customer schedules defined in Section 1.7.

2.10 Technology and commercialization plan

This should describe any technology advancement, technology infusion,
and commercialization initiatives that are drivers for any aspect of
this product plan. Especially describe any initiative for which the
corresponding product plan activities would not be performed if the
technology or commercialization initiative was not considered. If there
is none, state that.

**3.0 Technical Approach**

3.1 Design Development (GPG 8700.2)

3.1.1 Product Requirements\*

Describe (or reference) the derived requirements/specifications
developed by the Team and approved by the Customer. These should include
assumptions, interfaces, and performance information. Ensure that
requirements are testable. These requirements can be the customer's
original requirements (if so, just reference Section 2.3) or can be
those derived by the Team itself.

  ------------------------ ------------------------- ---------------------
  Controlled Document      Comment                   Record Held By

  Derived Requirements     Signed and dated by       Project or Team Lead
                           Customer and Team Lead    

  Interface Control        Signed and dated by       Project or Team Lead
  Documents                Representatives of        
                           Interfacing Organizations 
  ------------------------ ------------------------- ---------------------

3.1.2 Product Design\*

Describe the design of the product that the Team is planning to produce.
Describe how changes in design are updated and traced to changes in the
requirements.

+-----------------------+------------------------+---------------------+
| Quality Record        | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Completed Design      | May include:           | Team Lead           |
| Documentation         |                        |                     |
|                       | -   High-level         |                     |
|                       |     architecture       |                     |
|                       |     description        |                     |
|                       |                        |                     |
|                       | -   Design Review      |                     |
|                       |     Materials          |                     |
|                       |                        |                     |
|                       | -   Design Documents   |                     |
+-----------------------+------------------------+---------------------+

3.1.3 Development Strategy

Describe at a HIGH LEVEL the software components you will build, the
commercial off-the-shelf/Government off-the-shelf (COTS/GOTS) or
customer supplied items you will use, the prototyping plans, and the
integration requirements.

3.1.3.1 Buy Approach\* (GPG 5100.1)

Describe any special purchasing strategies for items specified in
Section 2.5.

3.1.3.2 Build Approach\*

Include the development phases, the sequence of builds, the high level
inputs and outputs per build, vendor/customer/prototype elements to be
integrated, and the high level functional requirements satisfied in each
build.

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Completed Build Plan     Part of software work     Team Lead
                           order authorization (WOA) 
  ------------------------ ------------------------- ---------------------

3.1.3.3 Prototyping Approach\*

Describe any prototyping activities required to develop the product and
the purpose for the prototype (i.e., what specific questions are to be
answered by the prototype?)

3.1.3.4 Customer Supplied Products Approach

List and briefly describe any software that will be received from the
customer for integration into the final product, and any assumptions
concerning them.

3.1.4 Product Testing

3.1.4.1 Product inspection and test \* (GPG 5330.1)

Describe the testing approach from unit through product delivery
(including in-process and final inspection). Reference your test plans
and discuss your testing approach for unit, build, and acceptance
testing, test team (developers, independent, customer), test data
(simulator, supplied data, flight hardware, real data), and any
acceptance criteria (particularly any from the customer for final
acceptance---Section 2.11). Describe how changes in design are mapped to
changes in test plans.

+-----------------------+------------------------+---------------------+
| Controlled Document   | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Documentation of      | Include one or more:   | Team Lead           |
| Verification          |                        |                     |
| Activities            | -   Unit Test Plans    |                     |
|                       |                        |                     |
|                       | -   Code Reading       |                     |
|                       |     Checklist          |                     |
|                       |                        |                     |
|                       | -   Design Baseline    |                     |
|                       |                        |                     |
|                       | -   Integration Test   |                     |
|                       |     Plans              |                     |
|                       |                        |                     |
|                       | -   Build Test Plans   |                     |
+-----------------------+------------------------+---------------------+

+-----------------------+------------------------+---------------------+
| Quality Record        | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Documentation of      | Include one or more:   | Team Lead           |
| Verification          |                        |                     |
| Activities            | -   Unit Test Results  |                     |
|                       |                        |                     |
|                       | -   Code Reading       |                     |
|                       |     Signoffs           |                     |
|                       |                        |                     |
|                       | -   Design Walkthrough |                     |
|                       |     Documentation      |                     |
|                       |     (including issues  |                     |
|                       |     and resolutions)   |                     |
|                       |                        |                     |
|                       | -   Integration Test   |                     |
|                       |     Results            |                     |
|                       |                        |                     |
|                       | -   Build Test Results |                     |
+-----------------------+------------------------+---------------------+

+-----------------------+------------------------+---------------------+
| Controlled Document   | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Documentation of      | **Include one or       | **Team Lead**       |
| (End-to-End) System   | more:**                |                     |
| Validation Activities |                        |                     |
|                       | -   High Level         |                     |
|                       |     Description of     |                     |
|                       |     Test To Be Run     |                     |
|                       |     (Can be in Test    |                     |
|                       |     Procedures)        |                     |
|                       |                        |                     |
|                       | -   Test Validation    |                     |
|                       |     Matrices           |                     |
|                       |                        |                     |
|                       | -   Detailed           |                     |
|                       |     Description of     |                     |
|                       |     Tests with Inputs, |                     |
|                       |     Expected Outputs,  |                     |
|                       |     and Step by Step   |                     |
|                       |     Procedures for     |                     |
|                       |     Running the Tests  |                     |
|                       |                        |                     |
|                       | -   Acceptance         |                     |
|                       |     Criteria           |                     |
+-----------------------+------------------------+---------------------+

+-----------------------+------------------------+---------------------+
| Quality Record        | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Documentation of      | Include one or more:   | Team Lead           |
| (End-to-End) System   |                        |                     |
| Validation Activities | -   Validation         |                     |
|                       |     Matrices           |                     |
|                       |                        |                     |
|                       | -   Validation Test    |                     |
|                       |     Results            |                     |
+-----------------------+------------------------+---------------------+

3.1.4.2 Incoming inspection and test (GPG 4520.2)

For purchased items, including hardware, document the Receiving
Inspection Instructions to describe special receiving instructions and
tests if other than kind, count and condition. Be sure that all in-scope
products received after May 1, 1999, are identified and entered into the
RITS system.

  ------------------------ ------------------------- ----------------------
  Quality Record           Comment                   Record Held By

  Receiving Inspection     RITS entry made by Team   Team Lead or Team
  Instruction (RITS entry) Lead or Team purchase     purchase person
                           person                    

  RITS Work Order                                    Team Lead
  Authorization (WOA)                                

  Incoming Inspection      See GPG 5340.2            Center Nonconformance
  Nonconformance Report                              Reporting/Corrective
                                                     Action (NCR/CA) System
  ------------------------ ------------------------- ----------------------

3.1.4.3 Statistical Techniques\* (GPG 8070.2)

Unless the Team determines a need for statistical testing of the product
or other statistical methods, include the following paragraph in this
section of your Product Plan.

"The Team has evaluated the need for statistical testing of the products
developed under this Product Plan and has determined that statistical
techniques are not required."

Examples of statistical techniques being used are (1) techniques to
obtain reliability of hardware systems and (2) comparisons of output
results after a platform language conversion. If statistical techniques
are being used, then the procedure for their use must be documented.

3.1.5 Development Status\*

3.1.5.1 Design/Implementation Status

Describe the method(s) that will be used to track the status through
this phase of the product.

+-----------------------+------------------------+---------------------+
| Objective Evidence    | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Status Information    | May include:           | Team Lead or        |
|                       |                        | designee            |
|                       | -   Schedule Charts    |                     |
|                       |     with Status        |                     |
|                       |     Indicated          |                     |
|                       |                        |                     |
|                       | -   Module-by-Module   |                     |
|                       |     Checklist          |                     |
|                       |                        |                     |
|                       | -   Configuration      |                     |
|                       |     Management Records |                     |
|                       |                        |                     |
|                       | -   Documentation of   |                     |
|                       |     Weekly Status      |                     |
|                       |     Meetings           |                     |
+-----------------------+------------------------+---------------------+

3.1.5.2 Testing Status

Describe the method used to track testing status of the product
throughout its life cycle.

+-----------------------+------------------------+---------------------+
| Objective Evidence    | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Test Status           | May include:           | Team Lead or        |
| Information           |                        | designee            |
|                       | -   Test Status Chart  |                     |
|                       |                        |                     |
|                       | -   Weekly Test        |                     |
|                       |     Meeting Status     |                     |
|                       |                        |                     |
|                       | -   Signoffs of        |                     |
|                       |     Completed Tests in |                     |
|                       |     Test Plan or       |                     |
|                       |     Procedures         |                     |
+-----------------------+------------------------+---------------------+

3.1.6 Development Environment

Describe the development and test hardware and locations, and all Team
development standards, as appropriate.

3.1.7 Technical Review Program (GPG 8700.4)

Describe the types of reviews you plan to have and the membership of the
review boards. This should include a discussion of any code or design
walkthroughs you plan to use as verification of the design. You must
have at least a requirements review, a design review, and a product
acceptance review with the customer where requests for action (RFAs) and
responses are kept. Peer reviews as defined by GPG 8700.6 may include
team reviews where one team member reviews another team member's work
and should be included on the Team review plan. Project related reviews
and other higher level reviews may be supported as requested and they
should be listed in the Project Plan.

+-----------------------+------------------------+---------------------+
| Quality Record        | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| -   Requests for      | Include One            | Team Lead or CCB    |
|     Action (RFAs) and |                        | Chair               |
|     Responses from    |                        |                     |
|     Review Meetings   |                        |                     |
|                       |                        |                     |
| -   Review Meeting    |                        |                     |
|     Notes with Action |                        |                     |
|     Item List and     |                        |                     |
|     Resolutions       |                        |                     |
+-----------------------+------------------------+---------------------+

3.2 Process for handling, storage, packing, marking, preservation and
transportation (GPG 6400.1)

List the medium for the various products to be delivered if different
from Section 1.6 and state how they will be delivered to the customer.
Describe any plans (such as back-ups) to prevent loss or damage to the
product in all phases of development, including software, documentation
and hardware.

3.3 Servicing

Describe the process for post delivery product maintenance (i.e., How do
you plan to meet the requirements specified in Section 2.8?) Address
responsibility for the maintenance, request process, process for doing
work, and product redelivery for custom, government off-the-shelf (GOTS)
and COTS software and hardware.

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Maintenance requests                               Team Lead

  Redelivery letters                                 Team Lead
  ------------------------ ------------------------- ---------------------

**4.0 Product Assurance**

4.1 Product Quality Assurance

4.1.1 Control of Nonconforming Products and Corrective Action\* (GPG
5340.2/\
GPG 1710.1)

Describe the process for recording and correcting problems in a "minor"
Nonconformance Reporting (NCR)/Corrective Action system. Include a
description of the process used to evaluate the cause of the problem and
to assess whether any changes need to be implemented to prevent future
recurrences. The minor NCR system should include the version or release
number where the problem was found and ideally, the version number that
includes the corrections. Nonconforming products are both identified by
their associated NCRs and the associated release numbers. Any products
released to the customer will include a ***release letter*** listing the
release number, the included capabilities of the release, and a
description of any remaining nonconformance in the release. Products
with remaining nonconformance may only be released to the customer with
proper approval. (See Library of Approved Team Processes or Criteria 7
in Appendix F.) The Center nonconformance reporting/corrective action
(NCR/CA) system shall be used if no minor nonconformance system exists
or if the nonconformance meets the Center wide criteria listed in the
GPG.)

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Nonconformance records                             Team Lead or
  from minor NCR system                              Nonconformance Lead

  Nonconformance records                             Center NCR system
  from Center NCR system                             

  Corrective Action Plans  May be in NCR system      Team Lead or
                                                     Nonconformance Lead

  Product Release Letters                            Team Lead
  ------------------------ ------------------------- ---------------------

4.2 Configuration Management (GPG 1410.2)

4.2.1 Control of Team software, hardware, documentation, and data\*

Describe how your Team does configuration management for your software,
hardware and documentation and who has the change authority for each. If
you use the Project's process for any of those, reference where their
procedures can be found. Describe the signature and change authority for
the development sections of the Product plan. Describe the method used
to uniquely identify versions of the software and the elements from
which it is built. The use of a commercial configuration management tool
is strongly recommended for environments where one is available. If
on-line copies of documentation or software are considered the
controlled copy, then the approval authority must control on-line
access. A list of documents and data under configuration management by
the team is to be referenced in the Product Plan in this section. The
list is to include the document or database name, the date or version
identification of the current version, the location of the documents or
database, and the person responsible for the item. Note: Any data bases
or web sites containing controlled information directly under the Team's
control shall contain a header identifying what is being viewed, as well
as the date of the last change and person responsible for its control.
(See Library of Approved Team Processes or Criteria 2B and 2B in
Appendix F.)

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Software CM records                                Team Lead (or
                                                     Configuration
                                                     Manager)

  List of items under                                Team Lead (or
  configuration management                           Configuration
                                                     Manager)

  Copy of signature page                             Team Lead (or
  of configuration                                   Configuration
  management item                                    Manager)

  Records of CCB approval                            Team Lead (or
                                                     Configuration
                                                     Manager)
  ------------------------ ------------------------- ---------------------

4.2.2 Control of test software and hardware (GPG 8730.1)

Describe anything used to test the product, which may be both hardware
and software. Describe how this software will be validated (i.e., how do
you convince yourself that the simulator is working properly?) If the
software used for testing is not the final validation, but is only used
as part of a self-check, where neither the test software or the product
being tested is considered correct until the final results are correct,
then describe that test scenario. Describe or reference the
configuration management process used to ensure the appropriate version
of the simulator is used. See Library of Approved Team Processes or
Criteria 4 in Appendix F. Also, discuss any inspection, measuring and
test equipment (IMTE) being used and any calibration requirements.

+-----------------------+------------------------+---------------------+
| Controlled Document   | Comment                | Record Held By      |
+-----------------------+------------------------+---------------------+
| Documentation of Test | Include one or more:   | Team Lead           |
| Software Verification |                        |                     |
| Activities            | -   Test Plans         |                     |
|                       |                        |                     |
|                       | -   Acceptance         |                     |
|                       |     Criteria           |                     |
+-----------------------+------------------------+---------------------+

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Calibration records and                            Team Lead
  calibration due dates                              
  for IMTE Team is using                             

  Test software test                                 Team Lead
  results                                            

  Records of Verification                            Team Lead
  from Contractor                                    
  ------------------------ ------------------------- ---------------------

4.2.3 Quality Records\* (GPG 1440.7)

Identify the Team's Quality Records Coordinator (person who keeps the
Quality Records List) and the location of the Quality Records List.
Describe the process by which an element is added to the Quality Record
List by the Coordinator and filed with the Custodian.

  ------------------------ ------------------------- ---------------------
  Controlled Document      Comment                   Record Held By

  Quality Records List                               Team Lead
  ------------------------ ------------------------- ---------------------

4.2.4 Control of customer supplied products\* (GPG 5900.1)

Describe the method that will be used to check out or test the software
or hardware supplied to you by the customer for inclusion into the
product or for testing or packaging of the product. If a customer
provides items for use by the Team in the development/testing of the
product, describe the process used to report any problems with the
item(s) back to the customer. Describe the configuration management
process for customer supplied elements listed in Section 1.12 for
changes initiated by the Team or by the customer. Include any other
processes used to safeguard customer supplied products. (See Library of
Approved Team Processes or Criteria 3 in Appendix F.) This section
should address simulators, test data, software algorithms, software
and/or hardware received from the customer.

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Problem Reports on                                 Team Lead
  Customer-Supplied                                  
  Products                                           
  ------------------------ ------------------------- ---------------------

4.3 Process and product metric analysis\*

It is a requirement to collect the metrics described in Appendix G, at a
minimum. Additionally, product metrics, Team process metrics, and ISD
metrics may be collected. Describe how you will use these metrics for
process improvement (see Library of Approved Team Processes or
Criteria 10 in Appendix F).

  ------------------------ ------------------------- ---------------------
  Quality Record           Comment                   Record Held By

  Data and Completed Forms                           Team Lead
  Representing the                                   
  Required Metrics                                   
  ------------------------ ------------------------- ---------------------

**5.0 Product Development Journals**

5.1 Team Lessons Learned

Maintain a log of lessons learned throughout the life cycle of the team
activities. The final lessons learned report is intended to be a brief
(about one-half page) summary of the key recommendations for changes or
inclusion in future similar projects.

  ------------------------ ------------------------- ---------------------
  Objective Evidence       Comment                   Record Held By

  Lessons Learned Log or                             Team Lead
  document                                           
  ------------------------ ------------------------- ---------------------

5.2 Key Issues, Decisions, and Rationale

Maintain a log of key issues, decisions, and rationale through the life
cycle of the Team.

  ------------------------ ------------------------- ---------------------
  Objective Evidence       Comment                   Record Held By

  Log of Key Issues,                                 Team Lead
  Decisions, rationale                               
  ------------------------ ------------------------- ---------------------

\*Items may be included by reference

Appendix A: Acronyms and Abbreviations

Appendix B: References

Include a list of any references used in the Product Plan.

# Appendix D: List of Quality Record Types  {#appendix-d-list-of-quality-record-types .PDHHeading1}

This appendix contains examples of typical ISD ("Center") level and Team
level Quality Record types which must be maintained for ISO 9001
compliance.

General information on quality records:

-   Quality records demonstrate implementation or completion of the
    activity or function or the actual as-built or as-tested
    configuration. They are *not* revised once completed.

-   Each Branch and each Product Team must have a Quality Records List
    and a Quality Records Coordinator. For the Quality Records List use
    the template in GSFC form 2266 which can be found on the GSFC
    Quality Management System (ISO 9001) web site.

-   Unless otherwise documented in a QMS document, the Quality Records
    Coordinator is the Product Team lead for Teams and the Branch Head
    for Branches.

-   Product Team Quality Records List shall only contain those quality
    records controlled by the Team (Other quality records associated
    with the Team, but controlled by another group, shall be listed on
    the other group's quality record list.

-   In preparation for an ISO 9001 audit, copies of these types of
    records would need to be provided to the Lead Auditor for
    inspection, upon request within one hour.

-   All Team members must know the location of these Quality Records.

-   Quality records for internal (to ISD) teams shall be maintained in
    the appropriate Branch for one year following delivery of the
    product. For the flight projects, the quality records transition to
    the flight project at the time of the termination of the Team
    activities.

-   Those quality records listed under the Team need not be separate
    documents, but could be combined into the number of documents that
    the Team feels is appropriate.

Examples of ISD Quality Management System Documents

-   ISD Product Development Handbook

Examples of Typical ISD Quality Record Types

-   Team Lessons Learned

-   ISD Metrics Collected

-   Employee Training Records (formal, other) (see GPG 3410.2- Official
    records should be in OHR)

-   Required On the Job Training Records (Form GSFC 17-112, see GPG
    3410.2)

Team Quality Record Types are described in Appendix D.

# Appendix E: ISD Roles and Responsibilities  {#appendix-e-isd-roles-and-responsibilities .PDHHeading1}

Below are the aggregate list of the roles and responsibilities of ISD
members in the total Business Development Process components defined
throughout the Product Development Handbook. It is collected here for
convenience. Team and Team Lead roles have been elaborated on from an
ISO 9001 perspective.

Roles and Responsibilities of the Team Lead

From Section 2:

-   Meet with the customer to understand and document the product
    requested

-   Present to the ISD Management a estimate of the staffing and skill
    levels, and approximate time commitments required to meet the
    customer's request

-   Assess the skills and expertise needed. Work with the ISD Management
    Team to select an appropriate qualified Team to ensure that the
    individuals are qualified to perform their assigned
    responsibilities.

-   Maintain a list of any "required" training needed and received by
    Team members

-   Assign work to Team members

From Section 3:

-   Manage the entire customer order fulfillment process from receipt of
    the initial customer order to the final fielding of quality products
    to the customer

-   Document each Team member's roles and responsibilities for inclusion
    in the Product Plan.

-   Provide performance assessment on Team member as requested by the
    ISD Management Team

-   Report appropriate information to customers, and the designated ISD
    Management Team member in a timely and accessible manner

-   Determine and document any training required for Team members and
    work with the ISD Management Team to ensure that training needs are
    met and that only qualified people work in producing the product

-   Maintain a list of "required" training received by Team members

-   Work with the customer and the ISD Management Team to resolve issues
    and conflicts that are beyond control of the Team

-   Identify any internal or external consultants that may be needed

-   Identify and request resources that are not directly under the
    Team's control

-   Ensure that the Team maintains a customer focus throughout the
    process and aligns the Teams goals with the customer's objectives

-   Ensure the Team works with the customer to define project guidelines
    and reflects these guidelines in the Product Plan

-   Ensures the success of the Team in meeting customer requirements
    according to cost and schedule goals

-   Provide leadership in developing a cohesive, focused, motivated Team

-   Ensure Team personnel are implementing and following through on
    their responsibilities

-   Maintain documentation of key issues and decisions, and, optionally,
    supporting rationale or information

With respect to QMS (ISO 9001) compliance, the Team Lead specifically
needs to:

-   Produce and maintain a master list of documentation/information for
    the project and ensure that it remains under configuration control.
    This is to include a:

```{=html}
<!-- -->
```
-   **Product Plan** (see Appendix C)

-   **List of all processes used**

-   **List of all quality records** (on Form in GPG 1440.7) (see
    Appendix D)

```{=html}
<!-- -->
```
-   Identify a Quality Records Custodian whose name will be recorded in
    the Product Plan and who has responsibility for control of the
    quality records associated with the Product Plan

-   Ensure Team personnel are implementing and following all of their
    responsibilities

```{=html}
<!-- -->
```
-   In addition to the work processes and processes identified above,
    documents that appear on the master list must be controlled

```{=html}
<!-- -->
```
-   Have a design plan and process management plan for your product
    development. Include these plans in the Product Development Plan.

```{=html}
<!-- -->
```
-   Identify design verification and validation activities and assure
    that they are executed and documented

-   Establish a Team review plan and ensure that all planned reviews are
    conducted.

-   Document action items as RFAs and document responses of RFAs

```{=html}
<!-- -->
```
-   Ensure that the Team evaluates the need for statistical techniques
    required for developing or testing the product as well as the need
    for any statistical techniques required to analyze the product
    development process

```{=html}
<!-- -->
```
-   Document any processes used to implement or control the application
    of any statistical techniques identified above

```{=html}
<!-- -->
```
-   Identify and use a corrective action process

```{=html}
<!-- -->
```
-   The project corrective action process must address the following:
    customer complaints, actions resulting from audit findings, and
    major nonconformances (see GPG 1710.1)

-   Ensure that non-conformances meeting the criteria listed in GPG
    1710.1 are documented in the Center on-line NCR/CA database. This
    shall occur [after]{.underline} an initial acceptance tested version
    release of a system or subsystem to the customer or representative
    has occurred (accompanied by a release letter). For flight software,
    this is after initial acceptance testing completion on the flight
    software test bed.

-   The process must be written and controlled. It must be referenced or
    detailed in the Product Plan.

```{=html}
<!-- -->
```
-   Identify and use a preventive action process

```{=html}
<!-- -->
```
-   Maintain a Preventive Action Action List

```{=html}
<!-- -->
```
-   Define and use a Configuration Management process

```{=html}
<!-- -->
```
-   The project must have an explicit, written Configuration Management
    process. It may be by reference, but it must be explicit and define
    what items will be controlled, who has authority to make changes,
    how the status of changes will be handled, and what configuration
    checking (audit) processes will be used.

```{=html}
<!-- -->
```
-   Have a process for reviewing and approving documents

```{=html}
<!-- -->
```
-   All controlled documents and deliverables must be reviewed and
    approved. Projects must have a written process for completing this
    activity for documents controlled by the project.

-   All controlled documents and data must have evidence that they were
    reviewed and approved by authorized personnel. Further, changes to
    these must be reviewed and approved by the same
    functions/organizations that initially performed the activities.

```{=html}
<!-- -->
```
-   Have a written process for uniquely identifying and controlling all
    products

```{=html}
<!-- -->
```
-   Provide a system to identify the product and track status of work
    being performed on product and provide a method for software that
    tracks the status of tests planned/ test results/ and test status by
    unique software version number or build identification and date

-   All products must have a unique product identification name or
    number so that erroneous products or versions are not used by
    project personnel in work performance

-   Ensure that status of tests and inspection s are documented a form
    which tracks the status of planned tests/ test results and test
    status by unique software version or build identification and date

-   Ensure that test results are documented and that testing does not
    proceed past a planned test/inspection point until non-conformances
    have been documented

```{=html}
<!-- -->
```
-   Every formal release shall be documented in a written release notice
    containing the version/release changes, including database changes,
    the requirements and the nonconformance information (see GPG 5330.1
    Section 2.1.5)

-   Have a process for controlling incoming products

```{=html}
<!-- -->
```
-   Ensure that Receiving Inspection Instructions are prepared for
    incoming products where applicable and document on WOA (see GPG
    4520.2) using GSFC RITS

-   Ensure that Receiving Inspection Instructions are executed and that
    any non-conformances are documented on WOA and on the appropriate
    non-conformance form (see GPG 5100.1)

```{=html}
<!-- -->
```
-   Provide input for evaluation for Supplier Performance Records (see
    GPG 5100.1)

-   Ensure that test equipment and test software is working properly and
    maintain records of this verification

-   Have a process for controlling and safe-guarding any
    customer-supplied product

```{=html}
<!-- -->
```
-   Inform the customer and obtain authorization if repair or rework is
    required of the customer supplied product

-   Document any damage or malfunction of customer supplied product and
    report to the customer for disposition of the product

```{=html}
<!-- -->
```
-   Ensure that any products requiring handling, storage or shipping are
    processed according to the GPG 6400.1. Retain shipping records on
    Form 20-4.

Roles and Responsibilities of the Team

-   Defines their mission, vision, and objectives based on their
    customer's objectives and ISD strategic implementation goals

-   Identify customer expectations and develop metrics to ensure that
    those expectations are met quantitatively and qualitatively

-   Establish effective communications mechanisms to facilitate the
    change process and to avoid misunderstandings

-   Identify customer supplied resources and information that will be
    needed for the successful completion of the project

-   Record effort and progress in the form of project metrics. Specific
    metrics are listed in Appendix G.

With respect to QMS (ISO 9001) compliance, the Team specifically needs
to:

-   Know the GSFC Quality Policy and how it affects products and
    processes

-   Have an easy access to all processes required for the job

```{=html}
<!-- -->
```
-   All ISD projects must have written processes for performing their
    work functions that typically include development, testing,
    configuration Management, systems engineering, analysis, quality
    assurance, maintenance, and operations. The primary duties that you
    perform must have written processes describing how they are done.

-   Relevant processes for your job must be written, reviewed, approved
    (signed by an authorized person), available (you must have access),
    current (up to date), and followed

-   Know how processes you use are improved or changed. The use of
    defect analysis, customer feedback, internal reviews, and results
    from the overall measurement program help guide changes to all ISD
    and project processes

-   Know that a master document list identifying the acceptable
    processes for performing your job exists and must be used

```{=html}
<!-- -->
```
-   Discard or appropriately mark all outdated controlled documents

```{=html}
<!-- -->
```
-   Documents that fall into this category are those contained in
    Product Plan master list or on the ISD master list. Each project
    must establish a related process.

-   Old versions of controlled documents must be marked as obsolete or
    outdated. A simple handwritten identifier suffices.

```{=html}
<!-- -->
```
-   Know where project schedules for reviews and tests are identified

```{=html}
<!-- -->
```
-   Tests (such as system tests or regression tests) and reviews (such
    as design reviews) must be identified and scheduled in the Product
    Plan

```{=html}
<!-- -->
```
-   Know how defects found in project products are recorded, tracked,
    corrected

```{=html}
<!-- -->
```
-   As problems such as "bugs" are found in testing or in the inspection
    or review process, you must be aware of how such problems are
    handled for products you are working on. You must be able to refer
    to some written process that describes how that problem is handled.

```{=html}
<!-- -->
```
-   Be aware of and use inspection and test processes for products
    during development and before delivery

```{=html}
<!-- -->
```
-   Almost all individuals on the Team participate in the generation of
    some intermediate or end product such as software, included data
    bases, data products, documents, or full systems. Each individual
    must be aware of what inspections take place and where that
    inspection is defined in a written process.

```{=html}
<!-- -->
```
-   Be able to produce records of all inspections, tests and reviews

```{=html}
<!-- -->
```
-   There must be evidence that activities in your processes have been
    completed and recorded

-   Know where quality records are maintained

```{=html}
<!-- -->
```
-   Know how each unique version of a system or product is identified
    and controlled

```{=html}
<!-- -->
```
-   Different versions of products such as software systems and
    documents under development or data files to be delivered must have
    some unique identifier to ensure that erroneous or incorrect
    versions are not used

```{=html}
<!-- -->
```
-   Know how you are qualified to do your work, how training is offered,
    and how work assignments are made

```{=html}
<!-- -->
```
-   You are qualified to do your job because of your education and
    experience, which must meet GSFC job descriptions, and identifies
    any needed training to meet your work assignment. Each year the
    performance appraisal review reassesses your capabilities, including
    both formal and relevant OJT experiences.

-   A training handbook is provided each year listing available courses.
    Formal training records are kept in your personnel folder.

-   Be aware of how work is assigned to you and who assigns you work.
    Your work is assigned by your immediate supervisor.

```{=html}
<!-- -->
```
-   Follow the processes that pertain to your job and maintain
    appropriate records

```{=html}
<!-- -->
```
-   The most important responsibility to ensure QMS compliance is for
    you to adhere to the processes that pertain to your job. Those
    processes designated for your use must be applied, and appropriate
    records must be kept as defined by the processes.

ISD Chief

From Section 2:

-   Ensure a Team Lead is identified

-   Ensure the Team Lead and Team consist of qualified individuals

From Section 3:

-   Designates the ISD Management Team member responsible for the
    approval of any Product Plan covered by this handbook.

From Section 4:

-   Establish a continuous improvement process for improving the
    Business Development Process and the Product Development Handbook

ISD Management Team

From Section 2:

-   Develop individual skills, Team skills and centers of expertise

-   Select and provide qualified Team Leads and Team members

-   Define the Team and Team Leads responsibility, authority and
    accountability for delivering quality products

-   Document any "required" training needs or OJT of employees in
    accordance with GPG 3410.2

-   Ensure that training received using funds other than GSFC training
    funds is documented either in the employee's official or unofficial
    personnel folder

From Section 3:

-   Ensure that the approaches taken by the Team stay aligned with ISD
    Strategic Implementation Plan

-   Ensure that the Team obtains necessary resources in a timely and
    effective manner

-   Reviews and approves the Team's Product Plan

-   Effectively remove barriers which impede a Team's progress

-   Ensure and defend the Team's empowerment rights throughout the
    process

-   Document the process for reviewing and changing Product Plans
    (customer agreements)

-   Facilitate support that the Team cannot provide themselves

-   Ensure customer inputs for the Team Lead's performance plans and
    evaluations are made and maintained (appropriate supervisor)

-   Ensure Team Lead's inputs for the Team performance plans and
    evaluations are made and maintained (appropriate supervisor)

-   Provide training to Team members as requested by the Team Lead

-   Document any identified training needs of employees on Performance
    Plan

-   Ensures that training received using funds other than training funds
    is documented on the OHR Employee Training Records

-   Document any required OJT on GSFC form 17-112 (see GPG 3410.2)

-   Identify any specific processes requiring qualification (see GPG
    3410.2)

-   Allocate resources as requested by the Team and approved

From Section 4:

-   Ensure the ISD Quality Manual (i.e., the Product Development
    Handbook) remains aligned with the GSFC Quality Manual

-   Review GSFC Quality Management System as requested for recommended
    improvements

-   Ensure alignment of organization to the ISD Quality Manual

-   Define responsibilities and authority for delivering quality
    products

ISD/QMS Representative

From Section 5:

-   Represent ISD at the Directorate level in all matters concerning QMS
    (ISO 9001) certification implementation and maintenance

-   Direct all activities pertaining to QMS (ISO 9001) within ISD

-   Overall responsibility for maintaining the ISD Quality Manual
    (Product Development Handbook)

-   Review newly developed Team Processes specified in the Product Plan

-   Overall responsibility for adding/deleting/maintaining the ISD
    Library of Approved Team Processes

-   Review newly developed Team Processes specified in the Product Plan

-   Overall responsibility for adding/deleting/maintaining the ISD
    Library of Approved Team Processes

ISD Document Configuration Manager

From Section 5.1:

-   Assign identification numbers for quality records, documentation and
    data

-   Maintains a list of ISD level quality records and configuration
    controlled items, date or identification number/letter, their
    locations and their owners

-   Obtains approval for deletion of items from controlled list from the
    ISD Management Team

Available Contracts Data Base Administrator

From Section 3:

-   Maintain a data base of current contracting mechanisms (hardware,
    software, manpower) that have demonstrated acceptable performance

-   Details of the information and format for this database may be found
    in the Supplier Performance Records, GPG 5100.2

# Appendix F: Required Team Processes & Functions {#appendix-f-required-team-processes-functions .PDHHeading1}

Below are listed the criteria for Team processes required to be defined
in the Product Plan. This definition of processes may be done by
reference acceptable existing Team processes located at
[http://isc.gsfc.nasa.gov/ISO9k/ISO9001.htm](http://isc.gsfc.nasa.gov/Iso9k/ISO9001.htm)
or by direct inclusion in the Product Plan itself. Below each process
name in this section are listed the criteria required for QMS 9001
compliance that must be performed in the process. The Team may develop
their own process as long as the criteria listed here are satisfied. The
ISD/QMS Representative has the responsibility for independently
reviewing the Team Product Plans to ensure that the processes defined
are ISO 9001 compliant. New processes documented by Teams and found to
be QMS compliant, will be added to the acceptable Team Processes
database by the ISD/QMS Representative for use by reference by
subsequent Teams (see process in Section 5.2).

Criteria 1 - Development Methodology (GPG 8072.1)

1\. Are requirements documented and agreed upon between customer and
Team?

2\. Are changes to requirements and software documented according to
Team configuration management process?

3\. Is Team collecting and documenting an agreed upon set of metrics in
order to manage the development?

4\. Is Team developing a proposal for method of satisfying customer
requirements, determining use of Off-the-Shelf or new development, and
considering customer's cost, schedule and quality? Is proposal reviewed
with and approved by customer?

5\. Have organizational and technical interfaces between different
groups which input into the design process been defined?

6\. Does the design output identify those characteristics in the design
that are crucial to the safe and proper functioning of the product?

7\. Are records of design reviews maintained as quality records?

8\. Are design verification measures recorded? (quality records)

9\. Are design changes and modifications identified, documented,
reviewed and approved by authorized personnel prior to their
implementation?

10\. Is Team verifying design against customer requirements and
documenting the results?

The following are methods of performing such reviews: a) reviews such as
preliminary or critical design reviews, b) Team reviews, c) design
walkthroughs, d) prototyping

11\. Has Team selected a development methodology to control the quality
of the software being developed? A methodology addresses design,
development, testing and delivery of a system.

Methodology may also address the following:

-   Programming rules

-   Programming language

-   Naming conventions

-   Standards for commentary (program design language, comments).

-   Code reading

12\. Are software deliveries tested against customer requirements and
results documented?

13\. Are delivery schedules and contents of software deliveries agreed
upon with the customer and the Team and documented? Is test status of
the delivery discussed with the customer prior to delivery and
documented?

Criteria 2A - Control of Documents and Data (GPG 1410.1)

These questions shall be addressed for this process:

1\. Is there a documented process for control of all documents and data
that relate to the quality system?

2\. Does the document control process include documents of external
origin such as standards and customer requirements?

3\. Are documents and data reviewed and approved by authorized personnel
prior to use?

4\. Is there a master list or equivalent document control process that
identifies the current revision status of documents and is it readily
available?

5\. Do controls ensure that:

> a\) Pertinent issues of appropriate documents are available at all
> locations where operations essential to the effective functioning of
> the quality system are performed?
>
> b\) Invalid and/or obsolete documents are promptly removed from all
> points of issue or use, or otherwise assured against unintended use?
>
> c\) Any obsolete documents retained for legal and/or
> knowledge-preservation purposes are suitably identified?

6\. Are changes to documents and data reviewed and approved by the same
function/organization that performed the original review and approval?

7\. If specified otherwise, do the designated functions/organizations
have access to pertinent background information upon which to base their
review and approval?

8\. Is the nature of the change identified in the document or
appropriate attachment where practicable?

Criteria 2B - Control of Quality Records (GPG 1440.7)

1\. Has the supplier established documented processes for control of
quality records and does this process include: a) Identification? b)
Collection? c) Indexing? d) Access? e) Filing? f) Storage? g)
Maintenance? h) Disposition?

2\. Are quality records maintained to demonstrate conformance to
specified requirements and the effective operation of the quality
system?

3\. Are pertinent quality records from subcontractors an element of this
data?

4\. Are all quality records legible?

5\. Are all quality records stored and retained in such a way that they
are readily retrievable?

6\. Are all quality records stored in facilities that provide suitable
environment to prevent damage or deterioration and to prevent loss?

7\. Are quality records made available for evaluation by the customer or
customer's representative for an agreed period, where agreed
contractually?

Criteria 3 - Control of Customer Supplied Elements (GPG 5900.1)

1\. Have documented processes been established for control of
verification, identification, storage and maintenance of customer
supplied product?

2\. Do the processes for customer supplied product include provisions
for recording and reporting to the customer a product that is lost,
damaged or otherwise unsuitable for use? (For projects, use the NCR/CA
system)

Criteria 4 - Identification and Traceability of Products (GPG 5310.4)

Traceability should include how builds and releases are identified
including identification of components that comprise the build or
release.

1\. Are documented processes established for identified items and
products by suitable means from receipt and during all stages of
production, delivery and installation?

2\. Are documented processes available for traceability where
appropriate?

3\. When traceability is a specified requirement is the product thereof
uniquely identified?

4\. Is this identification recorded?

Criteria 5 - Product Inspection and Test Approach (GPG 5330.1)

1\. Has the supplier established documented processes for inspection and
testing activities in order to verify that the specified requirements
are met?

2\. Does the quality plan or documented process state the required
inspection or tests and the records to be established?

3\. Does the supplier ensure that incoming product is not used or
processed until it has been inspected and determined to be conforming?
(Except in circumstances as noted below in question 6)

4\. Is verification of conformance to the specified requirement in
accordance with the quality plan and/or documented processes? All
inspection of Commercial-Off-the-Shelf products should be based on kind,
count, and condition unless specified otherwise in the procurement. See
GPG 4520.2. All problems with incoming products should use processes
defined in Criteria 6. Results of all inspections are quality records.

5\. Is consideration given to the amount of control exercised at the
subcontractor's premises and recorded evidence of conformance when
determining the amount and nature of receiving inspection?

6\. If incoming product is released for urgent production is it
positively identified and recorded in order to permit recall and
replacement in the event it is later determined to be nonconforming?

7\. Is the product inspected and tested as required by the quality plan
or documented processes?

8\. Is product held until the required inspections and tests have been
completed or necessary reports have been received and verified? (Except
as noted in question 6 above)

9\. Are all final inspections and test carried out in accordance with
the quality plan and/or documented processes?

10\. Do quality plans or documented processes require that all specified
inspections and tests (including receiving and in-process inspections
and tests) have been carried out and the results meet specified
requirements?

11\. Do the suppliers processes ensure that no product is dispatched
until all activities specified in the quality plan or documented
processes have been satisfactorily completed and the associated data and
documentation is available and authorized?

12\. Are records established and maintained which provide evidence that
the product has been inspected and/or tested?

13\. Do the records clearly show whether the product has passed or
failed the inspection and/or tests according to defined acceptance
criteria?

14\. Do records identify the inspection authority responsible for
release of the product?

Criteria 6 - Control of Test Equipment and Software (GPG 8730.1)

1\. Are documented processes established to control, calibrate and
maintain inspection, measuring and test equipment (including test
software) that is used to demonstrate the conformance of the product to
the specified requirements?

2\. Is inspection, measuring and test equipment used in a manner which
ensures that the measurement uncertainty is known and is consistent with
the required capability?

3\. Is test software or comparative references such as test hardware
checked to prove that they are capable of verifying the acceptability of
product prior to release for production, installation or servicing?

4\. Is test software of comparative references such as test hardware
rechecked at prescribed intervals?

5\. Has the supplier established the extent of such checks and are
records of these checks maintained?

6\. Is technical data pertaining to inspection, measuring and test
equipment made available to the customer or customer's representative
when this is a specified requirement?

7\. Has the supplier determined the measurements to be made and the
accuracy required?

8\. Are the appropriate inspection, measuring, and test equipment
selected that are capable of the necessary accuracy and precision?

9\. Is all inspection, measuring, and test equipment that can affect
product quality identified, calibrated and adjusted at prescribed
intervals, or prior to use, against verified equipment having a known
valid relationship to internationally or nationally recognized
standards?

10\. Where no such standards exist is the basis for calibration
documented?

11\. Has the process to be employed for calibration of inspection,
measuring, and test equipment been defined? Including details of: a)
Equipment type? b) Unique identification? c) Location? d) Frequency of
checks? e) Check method? f) Acceptance criteria? g) Action to be taken
when results are unsatisfactory?

12\. Is inspection, measuring and test equipment identified with a
suitable indicator or approved identification record to show the
calibration status?

13\. Are calibration records maintained for inspection, measuring and
test equipment?

14\. Is the validity of previous inspections and test results assessed
and documented when inspection, measuring and test equipment is found to
be out of calibration?

15\. Does the supplier ensure that the environmental conditions are
suitable for the calibration, inspections, measurements and test to be
carried out?

16\. Does the supplier ensure that the handling, preservation and
storage of inspection, measuring and test equipment is such that the
accuracy and fitness for use is maintained?

17\. Are inspection, measuring and test facilities, including both test
hardware and test software, safeguarded from adjustments which would
invalidate the calibration testing?

Criteria 7 - Control of Nonconforming Product (GPG 5340.2)

1\. Has the supplier established a documented process to ensure that
product that does not conform to specific requirements is prevented from
unintended use or installation?

2\. Does the control provide for: a) Identification? b) Documentation?
c) Evaluation? d) Segregation (when practical)? e) Dispositioning?

3\. Does the control provide for notification of the functions
concerned?

4\. Is the responsibility for review and authority for the disposition
of nonconforming product defined?

5\. Is nonconformance documented in the on-line Nonconformance Form and
information on the use of the form is found in the GSFC GPG 5340.2
located at
[http://arioch.gsfc.nasa.gov/ISO9000](http://arioch.gsfc.nasa.gov/iso9000)?
This form should be used only for those nonconformities specified in the
GPG 5340.2. All nonconformities shall be retained in the Product Team's
database. Nonconformities at the Project level should begin when a new
release has been delivered to the customer. All nonconformances shall be
cross-referenced to the software product work authorization form.

6\. Is nonconforming product reviewed in accordance with documented
processes? Are the following options considered? a) Rework to meet the
specified requirements? b) Accept with or without repair by concession?
c) Regrade for alternative standards? d) Reject or scrap?

7\. When required by the contract, is the proposed use or repair of the
product, which does not conform to specified requirements, reported for
concession to the customer or the customer's representative?

8\. Is the description of the nonconformity that has been accepted, and
of repairs, recorded to denote the actual conditions?

9\. Is repaired and/or reworked product reinspected in accordance with
the quality plan and/or documented processes?

Criteria 8 - Preventive and Corrective Action (GPG 1710.1)

Product Teams are expected to have their own documented corrective and
preventive action process with attributes described below. However, any
nonconformance which meets the criteria specified in the Control of
Nonconforming Products (GPG1710.1) and has been entered into the Project
Nonconformance Report database will be tracked and closed out by the
Project, using the processes described in GPG 1710.1.

1\. Does the supplier have documented processes for implementing
corrective and preventive action?

2\. Is the magnitude of the problem and the risks encountered taken into
account when corrective and preventive action is taken to eliminate the
causes of actual or potential nonconformities?

3\. Are changes to the documented processes resulting form corrective
and preventive action implemented and recorded?

4\. Do the processes for corrective action include: a) The effective
handling of customer complaints and reports of product nonconformities?
b) Investigation of the cause of nonconformities relating to product,
process and quality system, and recording the results of the
investigation? c) Determination of the correction action needed to
eliminate the cause of nonconformities? d) Application of controls to
ensure that corrective action is taken and that it is effective?

5\. Do the processes for preventive action include: a) The use of
appropriate sources of information such as: i) Processes and work
operations which affect product quality, ii) Concessions, iii) Audit
results, iv) Quality records, v) Service records, vi) Customer
complaints to detect, analyze and eliminate potential causes of
nonconformities? b) Determination of the steps needed to deal with any
problems requiring preventive action? c) Initiation of preventive action
and application of controls to ensure that is effective? d) Ensuring
that relevant information on actions taken is submitted for management
review?

Criteria 9 -- Process for Product Maintenance

1\. Where servicing (maintenance) is a specified requirement, has the
Team established documented processes for: a) Performing, b) Verifying,
c) Reporting that servicing meets the specified requirements?

Criteria 10 - Process for Process and Product Metrics Analysis (GPG
1710.1)

1\. Has the Team identified the need for statistical techniques required
for a) Establishing b) Controlling c) Verifying process capability and
product characteristics?

2\. Has the Team established documented processes to implement and
control the application of the statistical techniques identified above?

3\. Has the Team identified the method to used to analyze the collected
metrics from the product production and the method by which such
analysis information will be fed back into the improvement of the
process for the next production process?

# Appendix G: Mandatory Team Metrics  {#appendix-g-mandatory-team-metrics .PDHHeading1}

Table 1 lists the required^(1)^ metrics which all Product Teams are
expected to collect, record, and periodically analyze for process
improvement potential. The intent is to reduce the number of parameters
to a set that is meaningful to the team in managing, assessing, and
improving their own internal performance. Since many teams have not
previously been required to collect metrics in any specific format or
units and are in mid-development, the Team Lead may specify in the
product plan the format and units of the metrics collected and the
frequency at which they are collected. Table 2 provides a recommendation
for the units of the metrics and the frequency of their collection. The
information will be analyzed across similar and dissimilar activities to
assess best practices and help identify potential for process
improvements.

+---------------+------------------------------------------------------+
| Metric Group  | Metric                                               |
+---------------+------------------------------------------------------+
| Project       | Name                                                 |
+---------------+------------------------------------------------------+
| Schedules     | Major milestones (including reviews, test dates,     |
|               | release and build delivery dates) (2)                |
|               |                                                      |
|               | Quantification of progress (3)                       |
+---------------+------------------------------------------------------+
| Cost          | Budget in dollars, under the responsibility of the   |
|               | Team                                                 |
|               |                                                      |
|               | Civil service effort (4)                             |
|               |                                                      |
|               | Contractor effort (4)                                |
+---------------+------------------------------------------------------+
| Quality       | Open problem reports (5)                             |
+---------------+------------------------------------------------------+

Table 1 - Required Metrics

Notes:

\(1\) If it is determined that this set of metrics is not meaningful an
exception for the collection of some or all of these metrics may be
granted by the appropriate Branch Head. Metrics are not meaningful if
the Team or ISD cannot use them for managing, assessing, or improving
the team's internal performance; or for comparison with similar Teams to
assess the effectiveness of ISD processes. This exception will be
specifically documented in Section 4.3 of the Team's Product Plan.

\(2\) Expected project milestones should be recorded as part of the
design planning activities.

\(3\) Quantification of progress may be kept in one of the following
ways:

a)  By using an earned value system, with each activity assigned a
    relative weight (or number of points)

b)  As a checklist of activities to be completed versus those completed

c)  As a collection of regular periodic progress reports

\(4\) Civil service (or contractor) effort may be recorded in staff
months or as a percentage of the total time a person is assigned to the
Team. Civil Service effort may be based on requested support instead of
actual accumulated manpower if not readily accessible, but should
reflect actual personnel assignments when the two significantly deviate.

\(5\) Open problem reports should be the number of unresolved problems
or discrepancies listed in the minor NCR/CA system or the Goddard NCR/CA
system.

+------------+-----------------------+-----------------+---------------+
| **Metric   | **Metric**            | **Units**       | **When        |
| Group**    |                       |                 | Recorded      |
|            |                       |                 | (1)**         |
+------------+-----------------------+-----------------+---------------+
| *          | Name                  | Names           | At project    |
| *Project** |                       |                 |               |
|            |                       |                 | start         |
+------------+-----------------------+-----------------+---------------+
| **S        | -Planned major        | Milestone Dates | -During       |
| chedules** | milestones. Should    |                 | project       |
|            | include the           | Start/Stop      | planning      |
|            | following:            | Dates           | activities    |
|            |                       |                 |               |
|            | Reviews               | Start/Stop      | -Schedules    |
|            |                       | Dates           | should be     |
|            | Documentation         |                 | updated for   |
|            | (including ICDs)      | Star            | changes in    |
|            |                       | t/Stop/Delivery | plan          |
|            | Tests                 | Dates           |               |
|            |                       |                 | -At           |
|            | Releases/builds       | Sta             | completion of |
|            |                       | rt/Key/Delivery | milestone     |
|            | Procurements          | Dates Milestone |               |
|            |                       | Dates           | -On a         |
|            | External Deliveries   |                 | continuing    |
|            | to Team               | Same as above   | basis         |
|            |                       |                 |               |
|            | -Actual major         | Earned value    |               |
|            | milestones. See above | point system    |               |
|            |                       |                 |               |
|            | -Quantification of    |                 |               |
|            | progress              |                 |               |
+------------+-----------------------+-----------------+---------------+
| **Cost**   | -Budget for:          | Dollars by      | -Initial      |
|            |                       | fiscal year     | planned       |
|            | COTS hardware         |                 | budget        |
|            |                       | Dollars by      | numbers       |
|            | COTS software         | fiscal year     | should be     |
|            |                       |                 | recorded      |
|            | COTS software         | Dollars by      | during the    |
|            | maintenance           | fiscal year     | project       |
|            |                       |                 | planning      |
|            | COTS hardware         | Dollars by      | activities.   |
|            | maintenance           | fiscal year     | Budget        |
|            |                       |                 | expenditures  |
|            | -Civil service        | Staff months    | should be     |
|            | manpower estimated(2) |                 | monitored on  |
|            |                       | Staff months    | a periodic    |
|            | -Civil service        |                 | basis and     |
|            | actual(3)             | Staff months    | compared with |
|            |                       |                 | the planned   |
|            | -Contractor manpower  | Staff months    | budget        |
|            | estimated             |                 |               |
|            |                       | Number of       |               |
|            | -Contractor manpower  | classes         |               |
|            | actual                |                 |               |
|            |                       | Number of trips |               |
|            | -Training (4)         |                 |               |
|            |                       |                 |               |
|            | -Travel               |                 |               |
+------------+-----------------------+-----------------+---------------+
| **Non      | -Open nonconformances | Numbers and     | -On a         |
| Conform-** |                       | severity        | continuing    |
|            | -Number of            |                 | basis         |
| **ances    | nonconformances due   | Numbers and     |               |
| (5)**      | to:                   | severity\       | -On a         |
|            |                       | Numbers and     | continuing    |
|            | \>requirements        | severity\       | basis         |
|            | misinterpretation\    | Numbers and     |               |
|            | \>missed              | severity\       |               |
|            | requirements\         | Numbers and     |               |
|            | \> improper           | severity        |               |
|            | implementation\       |                 |               |
|            | \> poor performance\  | Numbers and     |               |
|            | \>Cosmetic            | severity        |               |
+------------+-----------------------+-----------------+---------------+
| **System** | -Number of changes    | Number          | -On a         |
|            | due to:               |                 | continuing    |
| **Changes  |                       |                 | basis         |
| (6)**      | \>New requirements    |                 |               |
|            |                       |                 |               |
|            | \>Modified            |                 |               |
|            | requirements          |                 |               |
|            |                       |                 |               |
|            | \>Deleted             |                 |               |
|            | requirements          |                 |               |
+------------+-----------------------+-----------------+---------------+

Table 2: Recommended Metrics

Notes:

1\) Unless otherwise noted, metrics are collected at the project start,
at key reviews and releases. Maintain a log of reasons for changes.

2\) Estimated Civil Service effort may be based on requested support.

3\) Actual Civil Service manpower should reflect actual personnel
assignments if actual accumulated manpower is not readily available.

4\) Training should represent the number of classes that Team members
plan to and/or have taken (if two people take the same class, that
counts as 2 classes).

Travel should represent the number trips that Team members plan to
and/or have taken (if two people go on the same trip, that counts as two
trips).

5\) This is to include errors only, not Customer requested changes to
the system (see System Changes). For each type of error, include the
cumulative number that have been identified and the cumulative number
that have been recorded in the GSFC NCR system. Also for each
non-conformance, record the level of severity, where "High" implies that
the software problem(s) are serious and cannot/should not be used, where
"Medium" implies that the problem is serious but that the software is
usable (possibly with work-around), and where "Low" implies the problems
are not serious.

6\) These are Customer directed changes that may result in a change to
cost, schedule and/or size, and should be logged as required. Only the
number of times this has occurred is to be recorded, NOT actual number
of requirement changes. There is no attempt to normalize changes, but to
just record the number of times redirection has occurred, even if
proposed by the Team and accepted by the Customer.

# Appendix H: References {#appendix-h-references .PDHHeading1}

1\. Product Development Handbook

[http://isc.gsfc.nasa.gov/ISO9k/pdh/PDH.pdf](http://isc.gsfc.nasa.gov/Iso9k/iso9001.htm)

2\. Strategic Implementation Planning Process

TBD

3\. Yearly Action Planning Process

TBD

4\. Library of Approved Team Processes

[http://isc.gsfc.nasa.gov/ISO9k/ISO9001.htm](http://isc.gsfc.nasa.gov/Iso9k/ISO9001.htm)

5\. GSFC Quality Management System Manual (ISO 9001)

[http://arioch.gsfc.nasa.gov/ISO9000/index.htm](http://arioch.gsfc.nasa.gov/iso9000/index.htm)

# Appendix I: Acronym List {#appendix-i-acronym-list .PDHHeading1}

AETD Applied Engineering and Technology Directorate

CA Corrective Action

COTS commercial off-the-shelf

EPG Engineering Process Group

GOTS government off-the-shelf

GPG Goddard Procedure and Guidelines

GSFC Goddard Space Flight Center

ICD interface control document

ISD Information Systems Division

ISO International Organization of Standardization

NASA National Aeronautics and Space Administration

NCR Non-Conformance Report

OBE on-board equivalent

OHR Office of Human Resources

OJT on the job training

PD position description

PG Procedures and Guidelines

QM Quality Manual

QMS Quality Management System

RFA Requests for Action

RITS Receiving Inspection and Test System

S/W software

TBD to be determined

URL uniform resource locator

WOA Work Order Authorization